Using the Wordpress RSS Feed Object and handling Timeouts

Handling timeouts with the Wordpress RSS Feed Object and SimplePie

In later versions of WordPress they have implemented their own RSS feed object which is basically just a wrapper for the well known RSS reader plugin SimpliePie.

On some of my own WordPress websites I need to make use of some RSS feeds to show results from various places such as horse racing results.

I first try and obtain an RSS results feed from Betfair and if that fails I then resort to a backup from the BHA.

However sometimes the first feed loads rather slowly and I end up with the lower quality feed due to the WordPress object timing out after 10000 milliseconds.

Even though I cache the results I require the first load of the feed to take however long it needs to - up to 30 seconds if it needs the time but due to the 10 second timeout I often don't get the first feed loaded into my cache.

Even after looking at the WordPress RSS feed object there seemed to be no easy way to pass in a timeout parameter into an overloaded constructor therefore the feed was always timing out after 10 seconds.

For all I know there is a way to set the HTTP load timeout with the WordPress RSS object but as of yet I haven't found it plus I haven't had the time or inclination to scour the WordPress codebase looking for it.

The code I was using for my first attempt is below.

$rss = fetch_feed('http://rss.betfair.com/RSS.aspx?format=rss&sportID=7');

// Checks that the object is created correctly 
if (!is_wp_error( $rss ) ) : 
 
 // Figure out how many total items there are, but limit it to 10. 
 $maxitems = $rss->get_item_quantity(10); 

 // Build an array of all the items, starting with element 0 (first element).
 $rss_items = $rss->get_items(0, $maxitems); 
endif;

However as you can see, although the RSS call is wrapped in an error object and I can pass in the maximum number of items I want to get. However I cannot pass in a timeout parameter.

Therefore instead of waiting for ages for a fix from anyone from WordPress I decided to jump straight over the WordPress wrapper object and go straight to the underlying SimpliePie object to get my data.

Not only does this mean no calls to wrapper objects that only ultimately point to SimplePie anyway but I have much more control over the parameters that are used for my feed collection.

I still use the 2 feeds and resort to my backup feed if I cannot get any items from my Betfair call but I now use the following code for my first attempt.

$feed = new SimplePie();
$feed->set_feed_url('http://rss.betfair.com/RSS.aspx?format=rss&sportID=7');

// set timeout to 30000 milliseconds (30 seconds)
$feed->set_timeout(30);

// set number of items to return to 10
$feed->set_item_limit(10);
$feed->set_stupidly_fast(true);

// enable caching and set the duration to cache for in seconds (200 seconds)
$feed->enable_cache(true);  
$feed->set_cache_duration(200);
$feed->init();
$feed->handle_content_type();

if (!$feed->error()):

 // get up to 10 items
 $rss_items = $feed->get_items(0, 10); 

 // if there are less than 10 items get the actual number in case I need to know
 $maxitems = count($rss_items);
else:
 // error output it
 echo "Error is " . $feed->error();
endif;

As you can see the ELSE branch just outputs an error message and this is just an example of how you can access the error if you need to. I don't use it on my site but it is useful to know if you are not using the WordPress error object.

The HTTP load timeout is easily set with the following line of code:

$feed->set_timeout(30);

Since I have changed my first attempt at loading the feed to skip the WordPress wrapper object with a call to SimplePie I haven't had any problems at all.

You might find this helpful if you are doing your own work with RSS feeds and WordPress.

Wordpress, WP-O-Matic and custom HTML tags

Automating Wordpress posts with WP-O-Matic

If you use wordpress you should really look into the great WP-O-Matic plugin that allows you to automate postings by importing content from RSS or XML feeds. You can set up schedules to import at regular times or import on demand from the admin area.

One issue however which I have just spent ages getting to the bottom of is the use of HTML such as OBJECT and EMBED tags. As a lot of content feeds contain multimedia files nowadays and you want this content to be imported directly into your site. The problem with WP-O-Matic and Wordpress in their default mode is that you will only get this content imported when you run the import from the admin menu or the page that the CRONJOB calls directly whilst logged in as an admin or publisher.

If you try to run the page the cronjob calls e.g /wp-content/plugins/wp-o-matic/cron.php?code=XXXXX whilst logged out or allow the job to run by itself you will find that certain HTML tags and attributes are removed including OBJECT and EMBED tags.

The reason is for security to prevent XSS hacks and its possible to get round this if you require to. This took me quite a long time to get to the bottom of as I am very new to Wordpress but I managed it in the end.

1. WP-O-Matic makes use of another object called SimplePie which is a tool for extracting content from XML and RSS. This object has a number of settings for stripping out HTML and the behaviour depends on how the feed import is called.

When running the import from the admin menu a setting called set_stupidly_fast is set to true which bypasses all the normal formatting and HTML parsing. When the CRONJOB runs this is set to false so the reformatting is carried out. In reality you want to run the reformatting as it does much more than just parse the HTML such as remove excess DIV's and comment tags and ordering the results by date.

If you don't care about this formatting you need to find the fetchFeed method in the \wp-content\plugins\wp-o-matic\wpomatic.php file and force it to be false all of the time:

$feed->set_stupidly_fast(false);

If you do want to keep the benefits of the stupidly_fast function but allow OBJECT and EMBED tags then you can override the strip_htmltags property in Simplepie that defines the tags to remove. You can do this in the same fetchFeed method in the wpomatic.php file just before the init method is called by passing in an array of tags that you do want Simplepie to remove from the extracted content.

// Remove these tags from the list
$feed->strip_htmltags(array('base', 'blink', 'body', 'doctype', 'font', 'form', 'frame', 'frameset', 'html', 'iframe', 'input', 'marquee', 'meta', 'noscript', 'script', 'style'));
$feed->init();

So that takes care of the WP-O-Matic class but unfortunatley we are not done yet as Wordpress runs its own sanitisation on posts in a file called kses.php found in the wp-includes folder. If you are logged in as admin or a publisher you won't get this problem but your CRONJOB will run into it so you have two choices.

1. Comment out the hook that runs all the kses sanitisation which isn't recommended for security reasons but if you wanted to do it the following line should be commented out in the kses_init_filters function e.g

function kses_remove_filters() {
// Normal filtering.
remove_filter('pre_comment_content', 'wp_filter_kses');
remove_filter('title_save_pre', 'wp_filter_kses');

// Post filtering
// comment out the hook that sanitises the post content
//remove_filter('content_save_pre', 'wp_filter_post_kses');
remove_filter('excerpt_save_pre', 'wp_filter_post_kses');
remove_filter('content_filtered_save_pre', 'wp_filter_post_kses');
}

Commenting out this line will ensure no sanitisation is carried out on your posts whoever or whatever does the posting. Obviously this is bad for security as if you are importing a feed that one day contained an inline script or an OBJECT that loaded a virus you could be infecting all your visitors.

2. The other safer way is to add the tags and attributes that you want to allow into the list of acceptable HTML content that the kses.php file uses when sanitising input. At the top of the kses file is an array called $allowedposttags which contains a list of HTML elements and their allowed attributes.

If you wanted to allow the playing of videos and audio through OBJECT and EMBED tags then the following section of code can just be inserted into the array.

'object' => array(
'id'=>array(),
'classid'=>array(),
'data'=>array(),
'type'=>array(),
'codebase'=>array(),
'align'=>array(),
'width'=>array(),
'height'=>array()),
'param' => array(
'name'=>array(),
'value'=>array()),
'embed' => array(
'id'=>array(),
'type'=>array(),
'width'=>array(),
'height'=>array(),
'src'=>array(),
'bgcolor'=>array(),
'wmode'=>array(),
'quality'=>array(),
'allowscriptaccess'=>array(),
'allowfullscreen'=>array(),
'allownetworking'=>array(),
'flashvars'=>array()
),

Obviously you can add whichever tags and attributes you like and this is the preferred way in my opinion of getting round this problem as you are still whitelisting content rather than allowing anything.

It took me quite a while to get to the bottom of this problem but I now have all my automated feeds running correctly importing media content into my blog. Hopefully this article will help some people out.