Problems with CloudFlare

Problems with CloudFlare

By Strictly-Software

Recently I moved a couple of my sites behind the free Cloudflare proxy option and set the DNS so it pointed to their server rather than the 123reg.co.uk ones I had been using.

However before I did this I tested out whether there was much difference between the sites.

1. I set to use Cloudflare and WP Super Cache.
2. I used WP Super Cache, Widget Cache and WP Minify.

I actually found that that the 2nd set-up gave me better results. Why I don't know.

However in the end due to all the spam and apparent blocking Cloudflare claimed to be able to do I set the other site up behind it.

However after a while I noticed a few things you might want to be aware of

1. I had a number of email scripts that sent out thousands of emails and I used a

set_time_limit(5000);

at the top of the file to ensure it didn't time out by the Virtual Servers standard 30 seconds.

Also and in-between each email (which I appended to a log file) I did a wait command with:

sleep(2);

So that I didn't overkill the server.

However you should be aware that when you use set_time_limit and then use functions like sleep or file_put_contents or file_get_contents the time it takes to wait, access files and retrieve data is not included in the time limit.

Also as I was using Cloudflare, my PHP script which is web based so I can easily call it by hand if I need to, was using the standard domain e.g http://www.mysite.com/mysendemailjob.php.

However Cloudflare it seems has it's own timeout limit which overrides anything you set in Apache or in your PHP file of about 60 seconds.

I noticed this because my script kept bombing out after around 60 seconds and returned a Cloudflare 524 error which you can read about here.




So to get round this problem I used the "direct" domain method they set as default to bypass CloudFlare.

I have obviously changed mine as you should to for security sake but once changed to something else when I ran the url: http://www.mysite.com/mysendemailjob.php it didn't bomb out any-more and carried on until the end.

Another thing you have to be careful about CloudFlare is if you have spent ages filling your IP Table up with IP addresses that you want banned due to spam, hack attempts or just over use.

Because all IP addresses in the Apache Access Log are now CloudFlare addresses these won't be used and you are now relying on ClouldFlare's own security measures to block dangerous IP's.

The same goes for your .htaccess file. If you have banned a whole countries range say China or Russia (biggest hackers on the earth - apart from the NSA of course) then these ranges won't mean jack as the user from China will be going through a CloudFlare proxy IP address to your site so any IP you had banned him from will now be useless.

The only thing left to do is ban by user-agent, blank agents is a good one and so are short ones (less than 10 characters as they are usually jibberish).

I ban most of the standard HTTP libraries like CURL, WGET, WIN HTTP, Snoopy and so on as most script kiddies download a library, and don't even bother changing the user-agent before crawling and spamming. Therefore if someone isn't going to tell me who they really are then they can get a 403!

So they are a few things to watch out for with Cloudflare.

I know you can get modules that replace the Cloudflare IP with the original users IP but if you are on an old Debian Lenny box then they don't have support for that.

They must be supplying x-forwarded-for or other headers as when I did a scan using the bypass URL I got back my original IP but with a www.mysite.com scan I got back CloudFlare IPS e.g 104.28.25.11 etc.

The only thing you can do if you cannot take a modern module and reverse engineer it to older code is use the WordPress CloudFlare plugin that lets you get real Akismet IP addresses so you can still ban them.

It is a pain and one I am debating on whether to return to the days of before Cloudflare where my own security measures meant I banned over 50% of traffic and my server bills didn't go up and up.

"CloudFlare is supposed to save me bandwidth but ever since I have installed it although it claims it has saved me loads my Rackspace bill for bandwidth use has gone up and up!"

So just be careful when using Cloudflare it may seem like a magic tool but all those rocketscripts they add to your code are just "async" attributes and you can get many a plugin to minify your source and compress it on the server without using PHP to do so.

The choice is yours but be warned!

Handling Blocking in SQL 2005 with LOCK_TIMEOUT and TRY CATCH statements

Handing Blocking and Deadlocks in SQL Stored Procedures

We recently had an issue at night during the period in which daily banner hit/view data is transferred from the daily table to the historical table.

During this time our large website was being hammered by BOTs and users and we were getting lots of timeout errors reported due the the tables we wanted to insert our hit records into being DELETED and UPDATED causing locks.

The default lock time is -1 (unlimited) but we had set it to our default command timeout of 30 seconds.

However if the DELETE or UPDATE in the data transfer job took over 30 seconds then the competing INSERT (to insert a banner hit or view) would time out and error with a database timeout due to the Blocking process not allowing our INSERT to do its job.

We tried a number of things including:

• Ensuring all tables were covered by indexes to speed up any record retrieval

• Reducing the DELETE into small batches of 1000 or 100 at a time in a WHILE loop to reduce the length of time the LOCK was held each time.

• Ensuring any unimportant SELECT statements from these tables were using WITH (NOLOCK) to get round any locking issues.

However none of these actually helped solve the problem so in the end we rewrote our stored procedure (SQL 2005 - 2008) so that it handled the LOCK TIMEOUT error and didn't return an error.

In SQL 2005 you can make use of TRY CATCH statements which meant that we could try a certain number of times to insert our data and if it failed we could just return quickly without an error as we also used a TRANSACTION to enable us to ROLLBACK or COMMIT the transaction.

We also set the LOCK_TIMEOUT to 500 milliseconds (so x 3 = 1.5 seconds) as if the insert couldn't be done in that time frame then there was no point logging it. We could have inserted it into another table to be added to our statistics later on but that is another point.

The code is below and shows you how to trap BLOCKING errors including DEADLOCKS and handle them.

Obviously this doesn't fix anything it just "masks" the problem from the end user and reduces the number of errors due to database timeouts due to long waiting blocked processes.

CREATE PROCEDURE [dbo].[usp_net_update_banner_hit]

@BannerIds varchar(200), -- CSV of banner IDs e.g 100,101,102

@HitType char(1) = 'V', -- V = banner viewed, H = banner hit

AS

SET NOCOUNT ON

SET LOCK_TIMEOUT 500 -- set to half a second


DECLARE @Tries tinyint

-- start at 1
SELECT @Tries = 1


-- loop for 3 attempts

WHILE @Tries <= 3

  BEGIN

 BEGIN TRANSACTION

 BEGIN TRY

  -- insert our banner hits we are only going to wait half a second

  INSERT INTO tbl_BANNER_DATA
  (BannerFK, HitType, Stamp)
  SELECT  [Value], @HitType, getdate()
  FROM dbo.udf_SPLIT(@BannerIds,',') -- UDF that splits a CSV into a table variable
  WHERE [Value] > 0

  --if we are here its been successful ie no deadlock or blocking going on
  COMMIT    

  -- therefore we can leave our loop
  BREAK

 END TRY

 -- otherwise we have caught an error!
 BEGIN CATCH

  --always rollback   
  ROLLBACK

  -- Now check for Blocking errors 1222 or Deadlocks 1205 and if its a deadlock wait for a while to see if that helps

  IF ERROR_NUMBER() = 1205 OR ERROR_NUMBER() = 1222

    BEGIN

   -- if its a deadlock wait 2 seconds then try again
   IF ERROR_NUMBER() = 1205
     BEGIN

    -- wait 2 seconds to see if that helps the deadlock
    WAITFOR DELAY '00:00:02'

     END   

       -- no need to wait for anything for BLOCKING ERRORS as our LOCK_TIMEOUT is going to wait for half a second anyway
       -- and if it hasn't finished by then (500ms x 3 attempts = 1.5 seconds) there is no point waiting any longer

    END      

  -- increment and try again for 3 goes
  SELECT @Tries = @Tries + 1

  -- we carry on until we reach our limit i.e 3 attempts
  CONTINUE    

   END CATCH

  END

Using the Wordpress RSS Feed Object and handling Timeouts

Handling timeouts with the Wordpress RSS Feed Object and SimplePie

In later versions of WordPress they have implemented their own RSS feed object which is basically just a wrapper for the well known RSS reader plugin SimpliePie.

On some of my own WordPress websites I need to make use of some RSS feeds to show results from various places such as horse racing results.

I first try and obtain an RSS results feed from Betfair and if that fails I then resort to a backup from the BHA.

However sometimes the first feed loads rather slowly and I end up with the lower quality feed due to the WordPress object timing out after 10000 milliseconds.

Even though I cache the results I require the first load of the feed to take however long it needs to - up to 30 seconds if it needs the time but due to the 10 second timeout I often don't get the first feed loaded into my cache.

Even after looking at the WordPress RSS feed object there seemed to be no easy way to pass in a timeout parameter into an overloaded constructor therefore the feed was always timing out after 10 seconds.

For all I know there is a way to set the HTTP load timeout with the WordPress RSS object but as of yet I haven't found it plus I haven't had the time or inclination to scour the WordPress codebase looking for it.

The code I was using for my first attempt is below.

$rss = fetch_feed('http://rss.betfair.com/RSS.aspx?format=rss&sportID=7');

// Checks that the object is created correctly 
if (!is_wp_error( $rss ) ) : 
 
 // Figure out how many total items there are, but limit it to 10. 
 $maxitems = $rss->get_item_quantity(10); 

 // Build an array of all the items, starting with element 0 (first element).
 $rss_items = $rss->get_items(0, $maxitems); 
endif;

However as you can see, although the RSS call is wrapped in an error object and I can pass in the maximum number of items I want to get. However I cannot pass in a timeout parameter.

Therefore instead of waiting for ages for a fix from anyone from WordPress I decided to jump straight over the WordPress wrapper object and go straight to the underlying SimpliePie object to get my data.

Not only does this mean no calls to wrapper objects that only ultimately point to SimplePie anyway but I have much more control over the parameters that are used for my feed collection.

I still use the 2 feeds and resort to my backup feed if I cannot get any items from my Betfair call but I now use the following code for my first attempt.

$feed = new SimplePie();
$feed->set_feed_url('http://rss.betfair.com/RSS.aspx?format=rss&sportID=7');

// set timeout to 30000 milliseconds (30 seconds)
$feed->set_timeout(30);

// set number of items to return to 10
$feed->set_item_limit(10);
$feed->set_stupidly_fast(true);

// enable caching and set the duration to cache for in seconds (200 seconds)
$feed->enable_cache(true);  
$feed->set_cache_duration(200);
$feed->init();
$feed->handle_content_type();

if (!$feed->error()):

 // get up to 10 items
 $rss_items = $feed->get_items(0, 10); 

 // if there are less than 10 items get the actual number in case I need to know
 $maxitems = count($rss_items);
else:
 // error output it
 echo "Error is " . $feed->error();
endif;

As you can see the ELSE branch just outputs an error message and this is just an example of how you can access the error if you need to. I don't use it on my site but it is useful to know if you are not using the WordPress error object.

The HTTP load timeout is easily set with the following line of code:

$feed->set_timeout(30);

Since I have changed my first attempt at loading the feed to skip the WordPress wrapper object with a call to SimplePie I haven't had any problems at all.

You might find this helpful if you are doing your own work with RSS feeds and WordPress.

Problem with SFTP after new router installation

Problem with SFTP / SSH after new router installation

This may seem like an obvious one but it can catch you out if you are not aware of the implications of having a new IP address assigned to your house's broadband. You may have moved your laptop to a new house or using a new wifi system to connect to the Internet.

More commonly you may have been given a new or upgraded router by your ISP provider.

Even if you are told the IP address has not changed by BT, Virgin, Sky, Verizon or whoever is giving you the new router you should do a check on any of the many IP checking pages out there on the web.

E.G this script shows you your current IP and ISP details.


Why is this important?

Well if you have your own server being hosted by a company e.g a cloud server somewhere and you have installed DENY HOSTS to block hacking attacks then you might find that you cannot SFTP (Secure FTP) into your server anymore or that using Putty and SSH to access your remote server suddenly stops working for no apparent reason. Obviously you want to access your server so the problem needs fixing.

Symptoms of an IP change causing problems include:

• Your server reporting error messages such as "server unexpectedly closed the connection."
• When you change the file transfer settings to simple FTP from SFTP you can access the server but then experience timeout errors or when the list directory command is run nothing happens.
• Not being able to use Putty or another SSH tool to connect to your server.
• Not having changed any settings on your computer but not being able to connect to your server anymore.

Solution to IP change:

• Check and write down your new IP address.
• Log into your server through WebMin or a web based system or from another computer that hasn't had an IP change.
• Check your DENY HOSTS list to see if your IP address is listed and if so delete the record.
• Add your new IP address in the ALLOW HOSTS list.
• Re-start your server. 

If you don't know how to do this read the 3rd part of my Wordpress Survival Guide about security.