Showing posts with label CPU. Show all posts
Showing posts with label CPU. Show all posts

Sunday, 29 December 2019

Removing Chrome Has Sped Up My PC

How Removing Google Products Can Speed Up Your Laptop


By Strictly-Software

If you read my other recent pieces on speeding up my PC you might be interested in the fact that the other day I removed Google Chrome and now my PC is fast as road runner.

If you ever look at the task manager when you have 5+ tabs open in Chrome then you will see 5+ Google Chrome processes running. Their idea was that each tab of Google Chrome would run in it's own process so as not to interfere with the running of other tabs that maybe open.

Not only does that not seem to be the case in my situation, a slow tab will cause high memory, high CPU and DISK usage which then effects the other tabs, therefore I removed Google Chrome altogether and now just use Firefox, Opera and TOR Browser.

Just look at my task manager processes now that I have removed Chrome. Whilst I was suffering 100% DISK Usage, and high CPU and memory I am now showing only low numbers.

Currently as I write this article with 7 tabs open in Firefox including Facebook a heavy hitting site due to all the Ajax it uses and Betfair which updates constantly I am still only using 4% CPU, 54% Memory, 9% Disk Usage.


Not only has removing Google Chrome from my laptop has sped it up but if anyone has been paying attention, Google is an evil company (they had to remove that it didn't do evil from their motto), that is hooked into nearly every smart phone measuring when you sleep, walk, drive, the current temperature as well as listening to everything you say just in case it wants to be helpful.

I cannot count the number of times that I have been watching a football match streamed from my phone to my TV and then Google pops up to ask "What would you like to know about Liverpool" or some other nonsensical question I didn't want it to help me with in the first place.

This obviously means it is listening to you ALL THE TIME, otherwise it would not know when to offer help.

If you read the Terms and Conditions on your smart phone then you will see under privacy, a mile down the long winded notes, a section about how they are allowed to listen to you and send off to "unspecified 3rd parties" content that "may" help improve their service.

These unspecified parties maybe Google AdSense to show more relevant adverts by listening to the surrounding sound around the phone and then translating that to adverts you will see online. Or it maybe listening for more illicit speech that it can send local police enforcement or even the numerous 3 letter agencies in the UK and USA.

Will Binney from the NSA who was arrested for going public about the Prism program he had written and Edward Snowden had stolen to expose, told us about how he had designed it for spying on the Soviets in communist USSR. However once the cold war was over they flipped the script and used it to spy on US citizens illegally instead. 

If you read up on the web tech companies that were first involved in the unknowing spying on users, then they were Microsoft, Yahoo, AOL, Facebook, Google, Apple, PalTalk, YouTube, and Skype.

The UK was used as a test bed due to our lack of a real bill of rights or constitution and therefore we here in UK are rats roaming a maze designed by the NSA all so they can catch the right words and tell the correct authorities if need be.

The main problem is that you cannot remove Google products from Android phones and trying to do so will probably break your phone. Google is ingrained into most smart phones and almost impossible to remove totally. If you want an app you go to Google Play Store and trying to remove that from your phone would be a huge hassle.

Therefore I have done the logical thing due to the slowness of my PC and removed Google Chrome from it. Not only has it sped up my laptop but it means Google searches are no longer logged and kept for years as I now use DuckDuckGo or TOR Browser to surf the net. 

Even the browser Opera has a mini "VPN" built into it which goes through a proxy computer before the site you wanted to visit, hiding your real IP address, and making you look like you are in Germany or Bulgaria. 

Just try Opera and then go to this site https://manytools.org/http-html-text/http-request-headers/ and see what headers and IP address you are showing to websites and servers.



In another tab either type in the search box "Where am I now geo-IP" or just go to this link and you will see information like the results below when I just tested it with a new instance of Opera. My real IP is 86.164.1.XXX by the way.
However viewing my IP address on Opera with that search term or a header search reveals my location to be elsewhere from South East England. In fact they show me to be in the Scandinavian region of Europe with an IP address of 77.111.247.188.


Their Geo-IP location data is complied from 3 sources, IP2Location, ipinfo.io and www.db-ip.com they all use my Opera proxy IP of 77.111.247.188, and put my location in either Norway or Sweden, in either Oslo or Östergötland.

If you want more details on why Google is evil and Youtube is broken then read and watch the video on this blog. 

However if you don't care about Google working with the NSA and GCHQ and listening to you all the time through your phone and just want a faster PC, try removing Google Chrome from your machine and using another browser instead.

Remember to keep your task manager open to see if the DISK, CPU and Memory levels drop after a restart and a clearing of your registry and unused objects with CCleaner.


By Strictly-Software

© 2019 Strictly-Software

Thursday, 28 May 2015

Twitter Rush - The Rush just gets bigger and bigger!

Twitter Rush - The Rush Just Gets Bigger And Bigger!

By Strictly-Software

The amount of BOTs, social media sites and scrapers that hit your site after you post a Tweet with a link in it to a site to Twitter just gets bigger and bigger. When I first started recording the BOTS that hit my site after a post to Twitter it was about 15 now it has grown to over 100+!

You can read about my previous analysis of Twitter Rushes here and here however today I am posting the findings of a recent blog posting using my Strictly TweetBOT WordPress plugin to Twitter and the 108 HTTP Requests that followed in the following minutes after posting.

If you are not careful these Twitter Rushes could consume your web servers CPU and Memory as well as making a daisy chain of processes waiting to be completed that could cause high server loads and long connection / wait times for the pages to load.

You will notice that the first item in the list is a POST to the article.

That is because in the PRO version of my Stricty TweetBOT I have an option to send an HTTP request to the page before Tweeting. Then you can wait a few seconds (a setting you control), before any Tweets are sent out to ensure the plugin has enough time to cache the page.

This is so that if you have a Caching Plugin installed (e.g on WordPress WP Super Cache) or another system, the page is hopefully cached into memory or hand written as an HTML file to prevent any overload when the Twitter Rush comes.

It is always quicker to deliver a static HTML file to users than a dynamic PHP/.NET file that needs DB access etc.

So here are the results of today's test.

Notice how I return 403 status codes to many of the requests. 

This is because I block any bandwidth wasters that bring no benefit at all to my site.

The latest batch of these bandwidth wasters seem to be social media and brand awareness BOTS that want to see if their brand or site is mentioned in the article.

They are of no benefit to you at all and you should either block them using your firewall or with a 403 status code in your .htacces file.

Please also note the amount of duplicate requests from either the same IP address or the same company e.g TwitterBOT or Facebook that are made to the page. Why they do this I do not know!

The Recent Twitter Rush Test - 28-MAY-2015

XXX.XXX.XXX.XXX - - [28/May/2015:17:08:17 +0100] "POST /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/?r=12 HTTP/1.1" 200 22265 "-" "Mozilla/5.0 (http://www.strictly-software.com) Strictly TweetBot/1.1.2" 1/1582929
184.173.106.130 - - [28/May/2015:17:08:22 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 403 252 "-" "ShowyouBot (http://showyou.com/crawler)" 0/3372
199.16.156.124 - - [28/May/2015:17:08:21 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22377 "-" "Twitterbot/1.0" 1/1301263
199.16.156.125 - - [28/May/2015:17:08:21 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22375 "-" "Twitterbot/1.0" 1/1441183
185.20.4.220 - - [28/May/2015:17:08:21 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22377 "-" "Mozilla/5.0 (TweetmemeBot/4.0; +http://datasift.com/bot.html) Gecko/20100101 Firefox/31.0" 1/1224266
17.142.151.49 - - [28/May/2015:17:08:21 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22375 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1; +http://www.apple.com/go/applebot)" 1/1250324
151.252.28.203 - - [28/May/2015:17:08:22 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22374 "http://bit.ly/1eA4GYZ" "Go 1.1 package http" 1/1118106
46.236.26.102 - - [28/May/2015:17:08:23 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22376 "-" "Mozilla/5.0 (TweetmemeBot/4.0; +http://datasift.com/bot.html) Gecko/20100101 Firefox/31.0" 0/833367
199.16.156.124 - - [28/May/2015:17:08:23 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22376 "-" "Twitterbot/1.0" 0/935200
142.4.216.19 - - [28/May/2015:17:08:24 +0100] "HEAD /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 403 - "-" "Mozilla/5.0 (compatible; OpenHoseBot/2.1; +http://www.openhose.org/bot.html)" 0/1964
17.142.152.131 - - [28/May/2015:17:08:24 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22375 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1; +http://www.apple.com/go/applebot)" 0/875740
52.5.154.238 - - [28/May/2015:17:08:25 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22376 "-" "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/26.0.1410.64 Safari/537.31" 1/1029660
4.71.170.35 - - [28/May/2015:17:08:26 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate HTTP/1.1" 403 251 "-" "grokkit-crawler (pdsupport@purediscovery.com)" 0/1883
192.99.19.38 - - [28/May/2015:17:08:26 +0100] "HEAD /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 403 - "-" "Mozilla/5.0 (compatible; OpenHoseBot/2.1; +http://www.openhose.org/bot.html)" 0/1927
141.223.91.115 - - [28/May/2015:17:08:28 +0100] "HEAD /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 - "_bit=55673d4a-0024b-030a1-261cf10a;domain=.bit.ly;expires=Tue Nov 24 16:07:38 2015;path=/; HttpOnly" "-" 1/1592735
17.142.151.101 - - [28/May/2015:17:08:32 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22260 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Applebot/0.1; +http://www.apple.com/go/applebot)" 17/17210294
184.173.106.130 - - [28/May/2015:17:08:49 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 403 252 "-" "ShowyouBot (http://showyou.com/crawler)" 0/1870
142.4.216.19 - - [28/May/2015:17:08:49 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 403 252 "-" "Mozilla/5.0 (compatible; OpenHoseBot/2.1; +http://www.openhose.org/bot.html)" 0/1601
52.6.187.68 - - [28/May/2015:17:08:28 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22262 "-" "Typhoeus - https://github.com/typhoeus/typhoeus" 20/20260090
45.33.89.102 - - [28/May/2015:17:08:28 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22262 "-" "Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36" 20/20370939
134.225.2.7 - - [28/May/2015:17:08:26 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22260 "-" "Mozilla/5.0 (TweetmemeBot/4.0; +http://datasift.com/bot.html) Gecko/20100101 Firefox/31.0" 22/22337338
2a03:2880:1010:3ff4:face:b00c:0:8000 - - [28/May/2015:17:08:25 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22261 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)" 23/23973749
134.225.2.7 - - [28/May/2015:17:08:27 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22259 "-" "Mozilla/5.0 (TweetmemeBot/4.0; +http://datasift.com/bot.html) Gecko/20100101 Firefox/31.0" 21/21602431
54.167.214.223 - - [28/May/2015:17:08:25 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22259 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36" 24/24164062
4.71.170.35 - - [28/May/2015:17:08:51 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 403 252 "-" "grokkit-crawler (pdsupport@purediscovery.com)" 0/1688
192.99.19.38 - - [28/May/2015:17:08:51 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 403 252 "-" "Mozilla/5.0 (compatible; OpenHoseBot/2.1; +http://www.openhose.org/bot.html)" 0/1594
54.246.137.243 - - [28/May/2015:17:08:51 +0100] "HEAD /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 403 - "-" "python-requests/1.2.3 CPython/2.7.6 Linux/3.13.0-44-generic" 0/1736
92.246.16.201 - - [28/May/2015:17:08:51 +0100] "HEAD /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 - "-" "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:30.0) Gecko/20100101 Firefox/30.0" 0/725424
4.71.170.35 - - [28/May/2015:17:08:55 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 403 252 "-" "grokkit-crawler (pdsupport@purediscovery.com)" 0/1808
2a03:2880:2130:9ff3:face:b00c:0:1 - - [28/May/2015:17:08:57 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate HTTP/1.1" 301 144 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)" 0/657830
54.198.122.232 - - [28/May/2015:17:08:51 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22258 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_6) AppleWebKit/534.24 (KHTML, like Gecko) (Contact: backend@getprismatic.com)" 7/7227418
2a03:2880:1010:3ff7:face:b00c:0:8000 - - [28/May/2015:17:08:51 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22255 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)" 7/7169003
54.198.122.232 - - [28/May/2015:17:08:51 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22257 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_6) AppleWebKit/534.24 (KHTML, like Gecko) (Contact: backend@getprismatic.com)" 7/7185701
2607:5300:60:3b37:: - - [28/May/2015:17:08:53 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22258 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:34.0) Gecko/20100101 Firefox/34.0" 5/5298648
2a03:2880:2130:9ff7:face:b00c:0:1 - - [28/May/2015:17:08:56 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22267 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)" 1/1999466
178.32.216.193 - - [28/May/2015:17:08:49 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22258 "http://bit.ly/1eA4GYZ" "LivelapBot/0.2 (http://site.livelap.com/crawler)" 9/9518327
199.59.148.209 - - [28/May/2015:17:08:58 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22260 "-" "Twitterbot/1.0" 1/1680322
54.178.210.226 - - [28/May/2015:17:08:58 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22257 "-" "Crowsnest/0.5 (+http://www.crowsnest.tv/)" 1/1842148
54.198.122.232 - - [28/May/2015:17:08:58 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22258 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_6) AppleWebKit/534.24 (KHTML, like Gecko) (Contact: backend@getprismatic.com)" 1/1903731
54.198.122.232 - - [28/May/2015:17:09:00 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22259 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_6) AppleWebKit/534.24 (KHTML, like Gecko) (Contact: backend@getprismatic.com)" 1/1131792
2607:5300:60:3b37:: - - [28/May/2015:17:09:00 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22260 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:34.0) Gecko/20100101 Firefox/34.0" 1/1048667
199.59.148.209 - - [28/May/2015:17:09:02 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22258 "-" "Twitterbot/1.0" 1/1024583
54.178.210.226 - - [28/May/2015:17:09:02 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22260 "-" "Crowsnest/0.5 (+http://www.crowsnest.tv/)" 1/1251088
65.52.240.20 - - [28/May/2015:17:09:03 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22259 "-" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0)" 0/814087
54.92.69.38 - - [28/May/2015:17:09:04 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22258 "-" "Crowsnest/0.5 (+http://www.crowsnest.tv/)" 0/925457
54.92.69.38 - - [28/May/2015:17:09:05 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22266 "-" "Crowsnest/0.5 (+http://www.crowsnest.tv/)" 0/932984
54.178.210.226 - - [28/May/2015:17:09:06 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22259 "-" "Crowsnest/0.5 (+http://www.crowsnest.tv/)" 0/927202
54.178.210.226 - - [28/May/2015:17:09:08 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22260 "-" "Crowsnest/0.5 (+http://www.crowsnest.tv/)" 0/717344
54.167.123.237 - - [28/May/2015:17:09:09 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 403 252 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:28.0) Gecko/20100101 Firefox/28.0 (FlipboardProxy/1.1; +http://flipboard.com/browserproxy)" 0/2286
54.178.210.226 - - [28/May/2015:17:09:12 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22254 "-" "Crowsnest/0.5 (+http://www.crowsnest.tv/)" 0/971022
37.187.165.195 - - [28/May/2015:17:09:52 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22258 "-" "Mozilla/5.0 (compatible; PaperLiBot/2.1; http://support.paper.li/entries/20023257-what-is-paper-li)" 0/688208
74.112.131.244 - - [28/May/2015:17:10:24 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22257 "-" "Mozilla/5.0 ()" 3/3572262
52.68.118.157 - - [28/May/2015:17:11:35 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22259 "-" "Crowsnest/0.5 (+http://www.crowsnest.tv/)" 0/688056
52.68.118.157 - - [28/May/2015:17:11:35 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22259 "-" "Crowsnest/0.5 (+http://www.crowsnest.tv/)" 0/719851
52.68.118.157 - - [28/May/2015:17:11:37 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22256 "-" "Crowsnest/0.5 (+http://www.crowsnest.tv/)" 0/739706
52.68.118.157 - - [28/May/2015:17:11:38 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22258 "-" "Crowsnest/0.5 (+http://www.crowsnest.tv/)" 0/760912
74.6.254.121 - - [28/May/2015:17:12:05 +0100] "HEAD /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 - "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)" 0/248578
66.249.67.148 - - [28/May/2015:17:12:38 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22259 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 0/493468
54.145.93.204 - - [28/May/2015:17:13:25 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 403 252 "-" "jack" 0/1495
54.145.93.204 - - [28/May/2015:17:13:26 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22257 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.8) Gecko/2008091620 Firefox/3.0.2" 0/597310
178.33.236.214 - - [28/May/2015:17:13:41 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 403 252 "-" "Mozilla/5.0 (compatible; Kraken/0.1; http://linkfluence.net/; bot@linkfluence.net)" 0/2065
173.203.107.206 - - [28/May/2015:17:13:50 +0100] "POST /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/?r=12 HTTP/1.1" 200 22194 "-" "Mozilla/5.0 (http://www.strictly-software.com) Strictly TweetBot/1.1.2" 4/4801717
184.173.106.130 - - [28/May/2015:17:13:58 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 403 252 "-" "ShowyouBot (http://showyou.com/crawler)" 0/96829
178.32.216.193 - - [28/May/2015:17:13:57 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22303 "http://bit.ly/1eA4GYZ" "LivelapBot/0.2 (http://site.livelap.com/crawler)" 1/1032211
192.99.1.145 - - [28/May/2015:17:13:59 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22303 "http://bit.ly/1eA4GYZ" "LivelapBot/0.2 (http://site.livelap.com/crawler)" 1/1535270
184.173.106.130 - - [28/May/2015:17:14:02 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 403 252 "-" "ShowyouBot (http://showyou.com/crawler)" 0/1764
52.6.187.68 - - [28/May/2015:17:14:01 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22293 "-" "Typhoeus - https://github.com/typhoeus/typhoeus" 66/66512611
146.148.22.255 - - [28/May/2015:17:15:10 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22292 "-" "Mozilla/5.0 (compatible; Climatebot/1.0; +http://climate.k39.us/bot.html)" 0/885387
74.6.254.121 - - [28/May/2015:17:15:11 +0100] "HEAD /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 - "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)" 1/1789256
146.148.22.255 - - [28/May/2015:17:15:17 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22290 "-" "Mozilla/5.0 (compatible; Climatebot/1.0; +http://climate.k39.us/bot.html)" 1/1275245
54.162.7.197 - - [28/May/2015:17:15:18 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22291 "http://bit.ly/1eA4GYZ" "Mozilla/5.0 (X11; U; Linux x86_64; en-US) AppleWebKit/534.13 (KHTML, like Gecko) Chrome/9.0.597.107 Safari/534.13 v1432829642.1352" 0/711142
146.148.22.255 - - [28/May/2015:17:15:24 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22293 "-" "Mozilla/5.0 (compatible; Climatebot/1.0; +http://climate.k39.us/bot.html)" 0/742404
54.162.7.197 - - [28/May/2015:17:15:32 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22289 "-" "msnbot/2.0b v1432829684.8617" 0/717679
23.96.208.137 - - [28/May/2015:17:16:05 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22294 "-" "Mozilla/5.0 (compatible; MSIE 10.0; Windows NT 6.1; Trident/6.0)" 0/560954
69.164.211.40 - - [28/May/2015:17:17:38 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22293 "-" "Mozilla/5.0 (compatible; EveryoneSocialBot/1.0; support@everyonesocial.com http://everyonesocial.com/)" 0/516967
96.126.110.221 - - [28/May/2015:17:18:24 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22300 "-" "Mozilla/5.0 (compatible; EveryoneSocialBot/1.0; support@everyonesocial.com http://everyonesocial.com/)" 0/464585
69.164.217.210 - - [28/May/2015:17:18:42 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22288 "-" "Mozilla/5.0 (compatible; EveryoneSocialBot/1.0; support@everyonesocial.com http://everyonesocial.com/)" 0/482230
173.255.232.252 - - [28/May/2015:17:19:03 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22293 "-" "Mozilla/5.0 (compatible; EveryoneSocialBot/1.0; support@everyonesocial.com http://everyonesocial.com/)" 0/514587
173.255.232.252 - - [28/May/2015:17:19:12 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22288 "-" "Mozilla/5.0 (compatible; EveryoneSocialBot/1.0; support@everyonesocial.com http://everyonesocial.com/)" 0/858459
96.126.110.222 - - [28/May/2015:17:19:26 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22288 "-" "Mozilla/5.0 (compatible; EveryoneSocialBot/1.0; support@everyonesocial.com http://everyonesocial.com/)" 0/469048
92.222.100.96 - - [28/May/2015:17:19:28 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22291 "-" "sfFeedReader/0.9" 0/574409
54.176.17.88 - - [28/May/2015:17:20:20 +0100] "HEAD /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+DarkPolitricks+%28Dark+Politricks%29 HTTP/1.1" 200 - "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36" 1/1112283
184.106.123.180 - - [28/May/2015:17:20:56 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22285 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; es-ES; rv:1.8.1.14) Gecko/20080404 Firefox/2.0.0.14" 0/522039
74.6.254.121 - - [28/May/2015:17:22:35 +0100] "HEAD /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 - "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)" 0/260972
54.163.57.132 - - [28/May/2015:17:23:05 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 403 252 "-" "Ruby" 0/2749
54.163.57.132 - - [28/May/2015:17:23:07 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 403 252 "-" "Ruby" 0/1647
54.163.57.132 - - [28/May/2015:17:23:09 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 403 252 "-" "Ruby" 0/1487
178.33.236.214 - - [28/May/2015:17:23:14 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+DarkPolitricks+%28Dark+Politricks%29 HTTP/1.1" 403 252 "-" "Mozilla/5.0 (compatible; Kraken/0.1; http://linkfluence.net/; bot@linkfluence.net)" 0/1996
168.63.10.14 - - [28/May/2015:17:23:23 +0100] "HEAD /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 403 - "-" "Apache-HttpClient/4.1.2 (java 1.5)" 0/1602
168.63.10.14 - - [28/May/2015:17:23:23 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 403 252 "-" "Apache-HttpClient/4.1.2 (java 1.5)" 0/1486
74.6.254.121 - - [28/May/2015:17:24:05 +0100] "HEAD /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 - "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)" 0/260635
45.33.35.236 - - [28/May/2015:17:24:59 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22284 "-" "Mozilla/5.0 ( compatible ; Veooz/1.0 ; +http://www.veooz.com/veoozbot.html )" 0/618370
74.6.254.121 - - [28/May/2015:17:25:35 +0100] "HEAD /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 - "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)" 0/255700
70.39.246.37 - - [28/May/2015:17:26:10 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22283 "-" "Mozilla/5.0 Moreover/5.1 (+http://www.moreover.com; webmaster@moreover.com)" 0/469127
82.25.13.46 - - [28/May/2015:17:28:55 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22285 "https://www.facebook.com/" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36" 0/568199
157.55.39.84 - - [28/May/2015:17:29:17 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22284 "-" "Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)" 0/478186
188.138.124.201 - - [28/May/2015:17:30:10 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22284 "-" "ADmantX Platform Semantic Analyzer - ADmantX Inc. - www.admantx.com - support@admantx.com" 2/2500606
54.204.149.66 - - [28/May/2015:17:30:30 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22285 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0 (NetShelter ContentScan, contact abuse@inpwrd.com for information)" 0/680643
54.198.122.232 - - [28/May/2015:17:30:31 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+DarkPolitricks+%28Dark+Politricks%29 HTTP/1.1" 200 22220 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_6) AppleWebKit/534.24 (KHTML, like Gecko) (Contact: backend@getprismatic.com)" 0/650482
64.49.241.208 - - [28/May/2015:17:30:32 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22284 "-" "ScooperBot www.customscoop.com" 0/658243
50.16.81.18 - - [28/May/2015:17:30:46 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22283 "-" "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:36.0) Gecko/20100101 Firefox/36.0 (NetShelter ContentScan, contact abuse@inpwrd.com for information)" 0/673211
54.166.112.98 - - [28/May/2015:17:30:47 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A%20DarkPolitricks%20%28Dark%20Politricks%29 HTTP/1.1" 403 252 "-" "Recorded Future" 0/1645
54.80.130.191 - - [28/May/2015:17:30:59 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 403 252 "-" "Recorded Future" 0/2777
74.6.254.121 - - [28/May/2015:17:31:35 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22282 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)" 0/606530
74.6.254.121 - - [28/May/2015:17:33:05 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22283 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)" 0/490218
54.208.89.59 - - [28/May/2015:17:34:31 +0100] "HEAD /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 - "-" "-" 0/309191
54.208.89.59 - - [28/May/2015:17:34:32 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22281 "-" "-" 0/647914
74.6.254.121 - - [28/May/2015:17:34:35 +0100] "GET /2015/05/study-finds-severe-cold-snap-during-the-geological-age-known-for-its-extreme-greenhouse-climate/ HTTP/1.1" 200 22284 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)" 0/555798

So the moral of the story is this:

  • Be careful when you post to Twitter as you will get a rush of traffic to your site in the following minutes which could cause your site problems.
  • Try and block social media / brand awareness / spam BOTS if you can so they don't consume your bandwidth and CPU/Memory/
  • Use either your servers firewall, or .htaccess file to block BOTS you consider a waste of your money. Remember any HTTP request to your site when you are using a VPS will cost you money. Why waste it on BOTS that provide no benefit to you.
  • Try and mitigate the rush by using the Crawl-Delay Robots.txt command to stop the big SERP BOTS from hammering you straight away.

I am sure I will post another Twitter Rush analysis in the coming months and the number of BOTS will have grown from the initial 15+ or so when I first tested it to 200+!

Monday, 9 February 2015

Speeding up Chrome can KILL IT!

Speeding up Chrome can kill it!

By Strictly-Software

Lately I have been really disappointed with the performance of my preferred browser Chrome.

I moved from FireFox to Chrome when the amount of plugins on FireFox made it too slow to work with however this was when Chrome was a clean, fast browser. Now it has just as many plugins available to install as FireFox and the performance has deteriorated constantly over the past few versions.

I am a developer so having 20+ tabs open in my browser is not unusual however when they are all hanging for no reason with "resolving host" messages in the status bar something is wrong.

I have even removed all my plugins that I had installed and decided to leave that to FireFox if I need to test different agents, hacking and so on. However even with a simple install the performance has been crap lately!

Therefore looked up on the web for tips on speeding up Chrome and found this article:

http://digiwonk.wonderhowto.com/how-to/10-speed-hacks-thatll-make-google-chrome-blazing-fast-your-computer-0155989/

It basically tells you some tricks to speed up Chrome by modifying some settings by going to chrome://flags/ in your address bar.

There is a warning at the top of the page that says:

WARNING These experimental features may change, break or disappear at any time. We make absolutely no guarantees about what may happen if you turn one of these experiments on, and your browser may even spontaneously combust. Jokes aside, your browser may delete all your data or your security and privacy could be compromised in unexpected ways. Any experiments that you enable will be enabled for all users of this browser. Please proceed with caution. Interested in cool new Chrome features? Try our beta channel at chrome.com/beta.

So these are all "experimental" features and from the sounds of it they could even make it's security and performance worse not better. Even a few of the tweaks suggested by the article had already disappeared from the settings page.

I did what was still available and a few more tweaks after careful consideration and what happened?

Well at first some pages seemed to load quicker but then I found that:

  • Some sites without a www. sub domain wouldn't load.
  • Some pages wouldn't load at all.
  • When I came into work today even though a Chrome process was running with 0 CPU usage nothing was displayed.
I had to re-boot, and try 3 times to open Chrome before getting back to the chrome://flags/  page and restoring all the defaults. Since then everything has been okay.

So if your going to tweak be careful - it could take down your whole browser!

The best way to speed it up is to remove all the plugins and add-ons and leave that to FireFox. Turn off 3rd party cookies and any 3rd party services that involve constant lookups and try to keep it clean and simple.

It seems that with the over usage of AJAX and sites like Facebook/LinkedIn/Google+ where as you type it constantly looks up the word to see if it matches a name or contact that this "API JIZZ" as I call it has really slowed down the web.

Just by having Google+, Facebook and LinkedIn open at the same time can eat up your memory and I'm on a quad core 64 bit machine.

In my opinion there should be settings to enable you to turn off the API JIZZ and flashy features that rely on lots of JavaScript and AJAX.

It slows down your computer and is not needed most of the time. Having big database lookups on each keystroke is obviously going to use up lots of memory so it should be an option you can disable.

Anyway that's down to the developers of all these social sites who seem to love AJAX for everything. A simple submit button would do in a lot of cases!


Tuesday, 11 November 2014

Turn off WordPress HeartBeat to reduce bandwidth and CPU

Turn off WordPress HeartBeat to reduce bandwidth and CPU

By Strictly-Software

I recently noticed a spike in bandwidth and costs on my Rackspace server. The cost had jumped up a good $30 from normal months.

Now I am still in the process of finding out why this has happened but one thing I did come across was a lot of calls to a script called /wp-admin/admin-ajax.php which was happening every 15 seconds.

Now this is the sign of WordPress's HeartBeat functionality which allows the server and browser to communicate and I quote from the inmotionhosting.com website, HeartBeat
allows WordPress to communicate between the web-browser and the server. It allows for improved user session management, revision tracking, and auto saving.
The WordPress Heartbeat API uses /wp-admin/admin-ajax.php to run AJAX calls from the web-browser. Which in theory sounds awesome, as WordPress can keep track of what's going on in the dashboard.
However this can also start sending excessive requests to admin-ajax.php which can lead to high CPU usage. Anytime a web-browser is left open on a page using the Heartbeat API, this could potentially be an issue.
Therefore I scanned my log files and found that my own server IP was making calls to a page every 15 seconds e.g

62.21.14.247 - - [11/Nov/2014:15:00:20 +0000] "POST /wp-admin/admin-ajax.php HTTP/1.1" 200 98 "http://www.mysite.com/wp-admin/post.php?post=28968&action=edit&message=1" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.124 Safari/537.36" 0/799585
62.21.14.247 - - [11/Nov/2014:15:00:35 +0000] "POST /wp-admin/admin-ajax.php HTTP/1.1" 200 98 "http://www.mysite.com/wp-admin/post.php?post=28968&action=edit&message=1" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.124 Safari/537.36" 25/25540888

I checked my browser (Chrome) which I do leave lots of windows open for ages, multi tasking :) , and I found that I had left open a post edit window in WordPress. This was causing the HeartBeat to call the script every 15 seconds.

Now I don't know that this is the ONLY reason for my increase in Bandwidth and obviously CPU due to all the HTTP requests but I am guessing it made up a big part of it.

Therefore I decided to turn off the HeartBeat functionality.

I have already disabled auto saving and revisions as I don't need that functionality so I am going to see what happens - hopefully my costs will go down!

Turning Off WordPress HeartBeat

To turn off the HeartBeat functionality go to your themes functions.php file and put the following code at the top of it.


// stop heartbeat code
add_action( 'init', 'stop_heartbeat', 1 );

function stop_heartbeat() {
        wp_deregister_script('heartbeat');

}


So I will see what happens with this turned off. So far not a lot.

However if you do notice a spike in your Bandwidth or CPU and you use WordPress check you haven't left a page open in your browser that would be causing the HeartBeat function to call the /wp-admin/admin-ajax.php every 15 seconds!

Monday, 13 October 2014

Debugging Issues with Strictly AutoTags or Strictly TweetBOT

Debugging Issues with Strictly AutoTags or Strictly TweetBOT

By Strictly-Software

You can purchase the pro versions (and download the free versions with less features) for these plugins here.

http://www.strictly-software.com/plugins/strictly-tweetbot
http://www.strictly-software.com/plugins/strictly-auto-tags

This article is about debugging issues you may occasionally get with these two plugins.

I usually find (as my code hasn't changed) that if the plugins stop working that the issue is down to one of these scenarios.
  • Another plugin causing issues.
  • WordPress code / update (bugs) - I usually wait until the first subversion is released before upgrading due to having to many problems upgrading to 3.7, 3.8 and 3.9 so I am waiting until 4.1 is released before upgrading this time.
  • Not enough memory on your server. Lots of tags equals lots of regular expressions to run which means memory usage.
  • Too many tags for the post save job to loop through on save. Trim them down regularly.
  • If you are on a shared server / VPS another user maybe taking up all your memory and preventing your site from running correctly - contact your hosting provider.
  • A hacker has caused issues on your server. Check your SSH file to see if it has been compromised with this console line /etc/init.d/ssh and view the contents.

Strictly AutoTags

Specific things to try with this plugin include:

  • De-Activating any auto-feed importer tools that create articles from RSS feeds and then saving a draft / published article from the test article in the Readme.txt file.
  • Ensuring you keep your tag list trim. If you have 25,000+ articles you shouldn't have many tags related to a single article or even three. So use the delete feature to remove tags with few articles related to them. The less articles, the more relevant the tags and the less memory is used.
  • If Auto Discovery is enabled and the test article in Readme.txt file works and tags are added the plugin is working. This means another cause for the issue.
  • Disable Jetpack, I have found this plugin to cause many issues.
  • Disable Post Revisions and set the Auto Save time so far in the future it won't cause problems. Depending on your settings you could be re-tagging your article every X seconds due to Auto Save and Post Revisions.
  • The code for disabling the WordPress features in the wp-config.php file are below.

/* turn off post revisions */
define ('WP_POST_REVISIONS', false);

/* set auto saves so far in future they shouldn't get saved on auto posts */
define( 'AUTOSAVE_INTERVAL', 10000 );

// increase memory limit
define( 'WP_MEMORY_LIMIT', '120M' );


Buy Now


Strictly TweetBot

Specific things to try with this plugin include:
  • Disable Jetpack Sharing or Jetpack altogether. Having multiple tweet systems can cause issues such as duplicate tweets and Twitter thinking you are sending automated messages. Jetpack has also been found to cause issues with many plugins.
  • Ensure you don't have too many Twitter BOT accounts setup. If you do increase your memory in wp-config.php (see the above code for Strictly AutoTags)
  • Constantly check your Admin panel to see if you are getting messages back from Twitter such as "duplicate tweet" - "this tweet looks like an automated message" - "this account needs re-authenticating". If you get these errors then Twitter is blocking your tweets. Change the formats of your messages and prevent duplicates from being sent.
  • If you need to re-authenticate you can try to delete the account, then re-add it and see if it works straight away without requiring a new PIN code.
  • If this doesn't work then you may need to re-install the plugin and all the accounts one by one and re-authenticate the OAuth with new PIN codes.
  • If you are linked automatically with Strictly AutoTags then ensure that the AutoTag plugin is working correctly. Make sure the tag list is not too large, there is enough memory to run the tagging and that when a post is made the POST META value strictlytweetbot_posted_tweet has been added to the post. If it hasn't and the post has tags but no tweets went out then you need to find out why. Check the Strictly AutoTags help list for information on fixing this. Memory and the size of the tag list are two major issues.

Important!

Always remember that if the code in the plugin hasn't changed but then the plugin suddenly stops working LOGICALLY it is likely NOT to be the plugins fault. Therefore:
  • Have you just done something on the server or website? If so roll it back to see if that effected the plugin.
  • Have you just upgraded or installed a new plugin? If so rollback or remove it.
  • Check other plugins by disabling or removing them.
  • Ensure your host has not had issues e.g memory / CPU is too high.
  • Ensure any new plugins have not caused issues.
  • If you have just upgraded WordPress try rolling back to the last working version.
  • Try re-starting Apache, then if that fails try a reboot (soft).

Buy Now

Saturday, 31 August 2013

Displaying Apache Server Status HTML

Display Your Apache Server Status As HTML Webpage

Diagnosing and fixing Apache issues can be a fucking nightmare (excuse the french). However one thing you might want to do is allow yourself access to view key Apache stats on a webpage rather than have to SSH in and use the console.

If you read some articles on the web to set this up it's as simple as adding the following lines to your main apache2.conf or httpd.conf file.

Depending on your server you will need to go into /etc/apache2/apache2.conf or etc/httpd/conf/httpd.conf and then adding the following lines.

# Allow server status reports, with the URL of http://dv-example.com/server-status
# Change the ".dv-example.com" to match your domain to enable.
ExtendedStatus on

   SetHandler server-status
   Order Deny,Allow
   Deny from all
   Allow from mywebsite.com

Or if you wanted to only allow your own IP address to access the page you could do something like this.

# Allow server status reports, with the IP addresss you want to allow access
ExtendedStatus on

   SetHandler server-status
   Order Deny,Allow
   Deny from all
   Allow from 86.42.219.12

However when I tried either of these approaches I was just met with an error message:

Forbidden

You don't have permission to access /server-status on this server.




The Fix
At the bottom of my main apache2.conf file I noticed the lines
# Include generic snippets of statements
Include /etc/apache2/conf.d/

Therefore I went into that folder and was met with 3 other files:

apache2-doc
charset
security

The top of the security file had these lines.


# Disable access to the entire file system except for the directories that
# are explicitly allowed later.
#
# This currently breaks the configurations that come with some web application
# Debian packages. It will be made the default for the release after lenny.
#
#
# AllowOverride None
# Order Deny,Allow
# Deny from all
#

So it seems that the main file was loading in these other files and the security file was blocking access to all other directories. By adding my rule to the top conf file the security file just over-ruled it again which caused the 403 error. Therefore I added the rules to the bottom of the security file and hey presto it worked.

Now I can access one of my domains to find my apache info. However I found that due to most of my sites on this virtual server using WordPress their ISAPI rules prevent a simple mysite.com/server-status from working.

WordPress is obviously trying to resolve the URL to a page, post or category list.

Therefore if you have got this problem you might need to use a domain not full of rules or create a rule that bypasses the WordPress guff.

Once you get it working you will get Apache info such as the following.
Current Time: Saturday, 31-Aug-2013 08:28:06 BST
Restart Time: Saturday, 31-Aug-2013 08:15:13 BST
Parent Server Generation: 0
Server uptime: 12 minutes 53 seconds
Total accesses: 622 - Total Traffic: 4.2 MB
CPU Usage: u26.65 s3.24 cu0 cs0 - 3.87% CPU load
.805 requests/sec - 5.5 kB/second - 6.9 kB/request
3 requests currently being processed, 4 idle workers

Plus lots of info about current HTTP requests, CPU usage and session info.

It is well worth doing if you need info about your server and you are away from your SSH console.

Saturday, 18 May 2013

Why I hate the new Google+ API

I absolutely hate the new Google+ API

Yes Google+ have had a revamp and if you are not on it then you won't know what the old version was like if you now join.

To me it's as if someone has read too many books on the jQuery effects library and basically orgasmed code across the API.

If you go to type a new status message into a box the whole page shifts round so that your box moves to the centre of the screen and the rest of the messages and segments of the page do a little jig around it so that you are supposed to go "wow".

Not me. Too much API Jizz is something I hate. 

Not only does it repeatedly turn my PC into a helicopter as the CPU rises and falls like a coke head on the lash but it just is too much for my ageing eyes.

It really seems to me as if someone is showing off by writing their "funky" API code. Hey boss look what I can do with a shit load of JavaScript that takes ages for all the page segments to load but makes non techies go "oooh" as they see it in action.

Whilst an API should be friendly and easy to use there is nothing "useful" about the whole screen moving around just so your current type box is in the middle of the screen.

Why not just put the "new message" box in the middle to start with?

Not only that but the amount of times I go to reply to a conversation down the right hand side and someone I have never seen before pops up in a box on top of the place I am trying to write is beyond annoying.

It means not only can I hit the send button but sometimes if I can find a way to get rid of the annoying box (and that's not 100% of the time) the message I was writing disappears!

I know writing the whole page in JavaScript stops (or limits script kiddy's) from scraping easily but there really is a limit. Personally I just think Google+ have crossed it and that there was nothing too wrong with their old API.

What do you think?


Wednesday, 29 September 2010

Analysing Bot Traffic from a Twitter Rush

Twitter Rush - Bot Traffic from Twitter

I blogged the other day about a link I found that listed the traffic that visits a site whenever a link to that site is posted upon twitter.

It seems that if you post a Tweet that contains a link a Twitter Rush is caused due to numerous BOTS, Social Media sites, SERPS and other sites noticing the new link and then all visiting your site at the same time.

This is one reason I created Strictly TweetBOT PRO which differs from my free version of Strictly TweetBOT as it allows you to do the following:


  • Make an HTTP request to the new post before Tweeting anything. If you have a caching plugin on your site then this should put the new post into the cache so that when the Twitter Rush comes they all hit a cached page and not a dynamically created one.
  • Add a query-string to the URL of the new post when making an HTTP request to aid caching. Some plugins like WP Super Cache allow you to force an un-cached page to be loaded with a query-string. So this will enable the new page to be loaded and re-cached.
  • Delay tweeting for N seconds after making the HTTP request to cache your post. This will help you ensure that the post is in the cache before the Twitter Rush.
  • Add a delay between each Tweet that is sent out. If you are tweeting to multiple accounts you will cause multiple Twitter Rushes. Therefore staggering the hits aids performance.

Buy Now
I have been carrying out performance tests on one of my LAMP sites and have been analysing this sort of data in some depth. I thought I would post an update with the actual traffic my own site receives when a link is Tweeted which is below.

A few interesting points:

1. This traffic is instantaneous so that the first item in the log file visiting the site has exactly the same time stamp as the WordPress URL that submitted the tweets to my Twitter account.

2. Yahoo seems to duplicate requests. This one tweet to a single post resulted in 3 requests for Yahoo's Slurp BOT but they originated from two different IP addresses.

3. These bots are not very clever and don't seem to log the URL's they visit to prevent duplicate requests. Not only does Yahoo have issues with the same account but if you post the same link to multiple Twitter accounts you will get all this traffic for each account.

For example when I posted the same link to 3 different Twitter accounts I received 57 requests (19 * 3). You would think maybe these Bots would be clever enough to realise that they only need to visit a link once every so often no matter which account posted it.

It just serves to prove my theory that most Twitter traffic is BOT related. 

BOTS following BOTS and Re-Tweeting and following traffic generated by other BOTS.

  • 128.242.241.133 - - [29/Sep/2010:21:06:45 +0000] "HEAD /2010/09/capital-punishment-and-law-and-order/ HTTP/1.1" 200 - "-" "Twitterbot/0.1"
  • 216.24.142.47 - - [29/Sep/2010:21:06:47 +0000] "GET /2010/09/capital-punishment-and-law-and-order/ HTTP/1.0" 200 26644 "-" "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.1.7) Gecko/20091221 Firefox/3.5.7 OneRiot/1.0 (http://www.oneriot.com)"
  • 204.236.254.109 - - [29/Sep/2010:21:06:48 +0000] "HEAD /2010/09/capital-punishment-and-law-and-order/ HTTP/1.1" 200 - "-" "PostRank/2.0 (postrank.com)"
  • 67.195.112.56 - - [29/Sep/2010:21:06:46 +0000] "GET /2010/09/capital-punishment-and-law-and-order/ HTTP/1.0" 200 100253 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
  • 72.30.142.220 - - [29/Sep/2010:21:06:47 +0000] "GET /2010/09/capital-punishment-and-law-and-order/ HTTP/1.0" 200 100253 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
  • 65.52.2.10 - - [29/Sep/2010:21:06:49 +0000] "GET /2010/09/capital-punishment-and-law-and-order/ HTTP/1.1" 200 26643 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0)"
  • 85.114.136.243 - - [29/Sep/2010:21:06:49 +0000] "GET /2010/09/capital-punishment-and-law-and-order/ HTTP/1.1" 200 26634 "-" "Mozilla/5.0 (compatible; Windows NT 6.0) Gecko/20090624 Firefox/3.5 NjuiceBot"
  • 72.30.142.220 - - [29/Sep/2010:21:06:49 +0000] "GET /2010/09/capital-punishment-and-law-and-order/ HTTP/1.0" 200 100253 "-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"
  • 89.151.113.134 - - [29/Sep/2010:21:06:49 +0000] "GET /2010/09/capital-punishment-and-law-and-order/ HTTP/1.1" 200 100253 "-" "Mozilla/5.0 (compatible; MSIE 6.0b; Windows NT 5.0) Gecko/2009011913 Firefox/3.0.6 TweetmemeBot"
  • 67.202.63.158 - - [29/Sep/2010:21:06:54 +0000] "GET /2010/09/capital-punishment-and-law-and-order/ HTTP/1.1" 200 26634 "-" "kame-rt (support@backtype.com)"
  • 38.113.234.180 - - [29/Sep/2010:21:06:57 +0000] "GET /2010/09/capital-punishment-and-law-and-order/ HTTP/1.1" 200 100253 "-" "Voyager/1.0"
  • 74.112.128.61 - - [29/Sep/2010:21:07:03 +0000] "GET /2010/09/capital-punishment-and-law-and-order/ HTTP/1.0" 200 100253 "-" "Mozilla/5.0 (compatible; Butterfly/1.0; +http://labs.topsy.com/butterfly/) Gecko/2009032608 Firefox/3.0.8"
  • 64.233.172.20 - - [29/Sep/2010:21:07:10 +0000] "GET /2010/09/capital-punishment-and-law-and-order/ HTTP/1.1" 200 26640 "-" "AppEngine-Google; (+http://code.google.com/appengine; appid: mapthislink)"
  • 208.94.147.190 - - [29/Sep/2010:21:07:17 +0000] "HEAD /2010/09/capital-punishment-and-law-and-order/ HTTP/1.1" 200 - "http://longurl.org" "LongURL API"
  • 208.94.147.190 - - [29/Sep/2010:21:07:17 +0000] "GET /2010/09/capital-punishment-and-law-and-order/ HTTP/1.1" 200 100253 "http://longurl.org" "LongURL API"
  • 66.249.65.166 - - [29/Sep/2010:21:07:25 +0000] "GET /2010/09/capital-punishment-and-law-and-order/ HTTP/1.1" 200 26653 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
  • 64.12.237.17 - - [29/Sep/2010:21:07:32 +0000] "GET /2010/09/capital-punishment-and-law-and-order/ HTTP/1.1" 403 455 "-" "Jakarta Commons-HttpClient/3.1"
  • 204.236.205.4 - - [29/Sep/2010:21:08:55 +0000] "HEAD /2010/09/capital-punishment-and-law-and-order/ HTTP/1.1" 403 - "-" "Firefox"
  • 67.207.201.163 - - [29/Sep/2010:17:01:06 +0000] "GET /2010/09/capital-punishment-and-law-and-order/ HTTP/1.1" 403 473 "-" "Mozilla/5.0 (compatible; mxbot/1.0; +http://www.chainn.com/mxbot.html)"

Buy Now

Friday, 28 August 2009

Firebug So So Slow

Developer Toolbars Slowing Sites Down

Following on from my whinge the other day about how IE 8.0's developer toolbar is causing pages to hang, CPU to max out (50% on dual core). I have noticed on a few sites now since upgrading to Firefox 3.5.2 and Firebug 1.4.2 that I have similar problems.

I have just been to my football site www.hattrickheaven.com and was clicking some of the leagues down the sidebars and wondering why the pages were taking so long to load. I then went to Chrome and tried the same links and the pages loaded as fast as lightening. This made me wonder about Firebug and low and behold disabling Firebug stopped the delay and the pages loaded very quickly. I would like some other people to try the same thing if they have Firebug and let me know the results e.g try these links with Firebug enabled and with it disabled:



With Firebug disabled they should load pretty quickly however with Firebug enabled its taking anything up to 15 seconds! Also check the Task Manager and see what CPU level Firefox shows as mine shows 50%.

I don't know if this is something to do with something within my page that Firebug cannot handle but the only validation errors I have are missing ALT attributes.

So as a tip for slow loading sites I would suggest in both IE and Firefox disabling the developer toolbar and Firebug and seeing if that helps speed up the load times. It seems to have worked for a number of sites now which is a shame as both toolbars should be great add-ons if only they could work without slowing everything down.

Tuesday, 21 October 2008

Dangers of Pattern Matching

Regular Expressions and Pattern Matching

Most modern systems use some form of pattern matching in there day to day running whether its client side Javascript replace or match statements, server side regular expression objects or database residing LIKE and PATINDEX statements and since SQL 2005 the ability to run CLR based regular expressions. Now although these are very useful tools to use you must be aware that its possible to max out your web or database servers CPU by running certain types of patterns.

I will tell you a short story about a very long weekend spent in the office at the beginning of 2007. We were migrating a big system to another server and we had copied all the code and database settings up and were watching the performance of the systems. Something strange was happening every minute or so. The CPU would jump from hardly anything up to 25%. After 30 seconds it would go back down again. Sometimes it would jump to 25% and then a few seconds later to 50%. In the daytime with all the sites running full throttle at intermittent stages throughout the day the CPU would jump in blocks of 25% up to 100% and all the sites on the server would become unusable causing lots of error messages and customer phone calls. We were at a loss to what was going on. It took quite a while before someone discovered that clicking on a link on a certain company profile on a certain site would always cause the CPU to spike. The site was a jobboard and the link was to display all the job results for that company. I copied down that sites job data to our dev box and ran the same test. Instead of causing the CPU to jump to 25% it maxed straight out to 100%. This made sense as our live server had a quad core. I debugged the code and found the cause was a regular expression that was in a function that re-formatted the snippet of job description shown on the results page. Although this function worked on 99% of the data in the system this company had a couple of jobs that contained certain Unicode characters and it was obviously causing the regular expression engine to freak out. We put it down to a problem inherit in Microsoft's Regular Expression engine and rewrote the code to break the one pattern down into 3 smaller steps which worked fine. However I then read an article about catastrophic backtracking and how certain patterns that can be matched in multiple ways can cause these sorts of CPU issues as the complexity of the pattern grows exponentially. If you have a very small pattern you may not notice it but if you have long strings then you can run into these CPU nightmares.

As well as problems that maybe built into your application due to patterns that could cause these high CPU issues when a certain replace or match is carried out in a specific instance you may be opening your site up to deliberate attacks by users that take advantage of an online searching tool that uses SQL and LIKE to search records in your database. If you are not careful and screen out wildcards and other symbols used within LIKEs pattern matching or regular expression symbols (+*^?$) if you are using a CLR build proc or function in SQL 2005 then you could be vulnerable to what is known as SQL Denial of Service attacks. Again its possible to max out your database servers CPU whilst a complex LIKE statement is run against your database.


Ways to avoid complex pattern matching affecting your system

  • Avoid backtracking.
  • If you have overly complex expressions consider breaking them down into multiple smaller expressions. Expressions that include lots of nested quantifiers ?*+.
  • Avoid expressions that can be matched by multiple string patterns.
  • If you are re-using an expression then compile it if possible.
  • If you are using an inherently useless regular expression engine (VB) then make sure you re-use your objects. Create a global regular expression object variable that's instantiated on first use and then re-used on subsequent calls and then destroyed at the end of the page.
  • If you have a page that is hit frequently that uses lots of expressions for formatting such as the job results example I mentioned then consider rewriting your code so that any complex formatting is done on input rather than output. Consider creating a separate column in your table that stores the formatted version which can be accessed quickly whenever it needs to be viewed. The performance gained by not having to run complex pattern matching constantly on each page load will be considerable.
  • If you are taking input from a user and using it in pattern matching in SQL LIKE or regular expression database searches then sanitise all input to make sure you do not fall victim to an SQL denial of service attack.
  • Also consider rewriting any database searching to use a full text index instead of LIKE searches. You will be able to offer your users a more feature rich and comprehensive search facility.

So now you know to be on the lookout for spikes in CPU. If you see the CPU in performance monitor jump up to 25% on a quad, 50% on dual or 100% in one leap then it could well be down to a regular expression pattern somewhere in your application that needs investigating.


Links

SQL Server Denial of Service Attack (LIKE Attack)

Detailed description of how backtracking can cause CPU issues

Catastrophic Backtracking

Tuesday, 9 September 2008

SQL Performance Tuning Queries

SQL 2005 Performance Tuning Queries

Here are some of my favourite queries that help with SQL performance tuning.
I have gathered most of these from the web and 3rd party sources over the years with the odd tweak here and there if needed and they form the bulk of my performance tuning toolkit. You should have a file containing a number of similar queries always close at hand as they are invaluable in hunting down problematic queries or even SQL injections or DOS attacks.

Update: 07-Dec-08
In response to the comment about filtering by db_id() I have removed that filter from a few of the examples as he is correct in stating that filtering by db_id() will not return all the possible data and will miss out queries run from the client or from within stored procs that use dynamic sql. Also if you are in a hurry and want to skip to a very useful procedure that will output nearly 20 different performance related reports for one or more databases then go straight to this download of mine: Download SQL 2005 Performance Reports. 


Top 50 worst queries for I/O

There are 2 different kinds of I/Os that are tracked within SQL Server: Logical and Physical IOs. Logical I/O account for data that is processed from the buffer pool which resides in memory, hence the phrase Logical I/O. Physical I/Os are I/Os that are associated with accessing data directly from the physical disks that SQL Server uses to store databases. Physical I/O’s are more expensive I/O’s, meaning they take longer to process. I/O is general the single most expensive operation that impacts the overall performance of a TSQL statement. So when you are tuning your queries you want to minimize the number of logical and physical I/O’s operation performed to produce a result set.

If you query shows physical I/O on the first run but none on the second its due to SQL holding the data in memory so you need to clear your buffer cache using DBCC DROPCLEANBUFFER

SELECT TOP 50
(qs.total_logical_reads + qs.total_logical_writes) /qs.execution_count as [Avg IO],
substring (qt.text,qs.statement_start_offset/2,
(case when qs.statement_end_offset = -1
then len(convert(nvarchar(max), qt.text)) * 2
else qs.statement_end_offset end - qs.statement_start_offset)/2)
as query_text,
qt.dbid,
qt.objectid
FROM sys.dm_exec_query_stats qs
CROSS APPLY sys.dm_exec_sql_text (qs.sql_handle) as qt
ORDER BY [Avg IO] DESC


Top 50 worst queries by average running time

The following query will list the worst performing queries based on their average elapsed running time. It will give you the times, logical & physical reads and the TSQL of the statement in question.

SELECT TOP 50
creation_time,last_execution_time,total_physical_reads
,total_logical_reads,total_logical_writes, execution_count
,total_worker_time, total_elapsed_time, total_elapsed_time / execution_count
avg_elapsed_time, SUBSTRING(st.text, (qs.statement_start_offset/2) + 1,
((CASE statement_end_offset WHEN -1 THEN DATALENGTH(st.text) ELSE
qs.statement_end_offset END - qs.statement_start_offset)/2) + 1) AS statement_text
FROM sys.dm_exec_query_stats AS qs
CROSS APPLY sys.dm_exec_sql_text(qs.sql_handle) st
ORDER BY total_elapsed_time / execution_count DESC;


This query is similar to the previous one in the DMVs (dynamic management views) it uses but it provides the most CPU intensive queries in a system. If you are seeing high CPU spikes or top end flatlining on your DB server then this query will help find out whether you are being attacked with SQL Denial of Service attacks or whether you have some queries that are using pattern matching techniques (LIKE, PATINDEX, CLR Regular Expressions) that have been given a complex string to search for that has caused the server to max out on CPU.

SELECT TOP 50 (a.total_worker_time/a.execution_count) AS [Avg_CPU_Time],
Convert(Varchar,Last_Execution_Time) AS 'Last_execution_Time',
Total_Physical_Reads,
SUBSTRING(b.text,a.statement_start_offset/2,
(case when a.statement_end_offset = -1 then len(convert(nvarchar(max), b.text)) * 2
else
a.statement_end_offset end - a.statement_start_offset)/2) AS Query_Text,
db_name(b.dbid)as DatabaseName,
b.objectid AS 'Object_ID'
FROM sys.dm_exec_query_stats a
CROSS APPLY sys.dm_exec_sql_text(a.sql_handle) AS b
ORDER BY [Avg_CPU_Time] DESC


Suggested index columns and usage

SELECT d.*
,s.avg_total_user_cost
,s.avg_user_impact
,s.last_user_seek
,s.unique_compiles
FROM sys.dm_db_missing_index_group_stats s
,sys.dm_db_missing_index_groups g
,sys.dm_db_missing_index_details d
WHERE s.group_handle = g.index_group_handle
AND d.index_handle = g.index_handle
AND database_id = db_id()
ORDER BY s.avg_user_impact DESC
View index usage by table.

Ideally you want high figures in the user_seeks column.
If you have no seeks and lots of lookups then you should investigate. Also if you have indexes with no seeks, scans or lookups then they should be dropped as the index is not being used.

SELECT i.Name, i.Type_Desc, i.Is_unique, d.*
FROM sys.dm_db_index_usage_stats as d
JOIN sys.indexes as i
ON i.object_id = d.Object_id and
i.index_id = d.Index_id
WHERE database_id=db_id()
AND d.object_Id=object_id('JOBS')
View the fragmentation levels of your In Row (non blob/text) indexes


SELECT object_name(d.object_id), d.object_id, d.index_id, i.name as IndexName,
avg_fragmentation_in_percent, alloc_unit_type_desc
FROM sys.dm_db_index_physical_stats (db_id(), NULL, NULL, NULL, NULL) as d
JOIN sys.indexes as i
ON i.object_id = d.object_id and
d.index_id = i.index_id
WHERE d.Index_Type_Desc <> 'HEAP' AND
i.Name is not null AND
avg_fragmentation_in_percent>= 30 --change this figure to reflect your own requirements
ORDER BY object_name(d.object_id)


Finds the degree of selectivity for a specific column in a row

The higher the percentage the more selective it is and an index should be considered. There is no point having indexes on columns that have low selectivity as it would not help differentiate between rows when 50% of rows are one value and 50% another. This is the reason why its not a good idea to place indexes on bit columns as there are only two possible values. The only exception to this would be when you have a very high percentage that have one value (99% true) and you want to find the 1% that have the opposite value (false) however even then I would advise against it.


Declare @total_unique float
Declare @total_rows float
Declare @selectivity_ratio float

SELECT @total_unique = 0
SELECT @total_rows = 0
SELECT @selectivity_ratio = 0

/*
Finds the Total Number of Unique Rows in a Table
Be sure to replace OrderID below with the name of your column
Be sure to replace [Order Details] below with your table name
*/
SELECT @total_unique = (SELECT COUNT(DISTINCT ClientFK) FROM JOBS)

/*
Calculates Total Number of Rows in Table
Be sure to replace [Order Details] below with your table name
*/
SELECT @total_rows = (SELECT COUNT(*) FROM JOBS)

--Calculates Selectivity Ratio for a Specific Column
SELECT @selectivity_ratio = ROUND((SELECT @total_unique/@total_rows),2,2)
SELECT @selectivity_ratio as 'Selectivity Ratio'



So those are a few of my favourite queries that I use very frequently to help hunt down and resolve back end performance bottlenecks. Since SQL 2005 came out with all these really cool DMVs a lot of people have spent time putting these sorts of queries together into one report.

If you want to get your hands on one of these all encompassing queries then check out the following stored procedure which uses a query I came across and extended to make it more customisable:


Its a beast of a report but gives you everything you need to know in one report. Read the comments before each section carefully so you understand what the data is showing you. Also if you run the report as it is on a server with lots of databases, tables, indexes etc it will take some considerable time to execute so I suggest using one of the modes that enables you to filter only on those databases you require statistics for.  So call it like so:

EXEC dbo.usp_sql_rpt_database_performance_stats 3, 'strictlysoftware%'

will only report on databases on that server that have a name (sys.databases) that start with strictlysoftware. You will most likely also want to exclude the system tables (master, msdb, model) although you may want to report on tempdb so call it like so:

EXEC dbo.usp_sql_rpt_database_performance_stats 1, NULL

Another script that contains some useful code related to blocking and wait times can be found at Microsoft's site. The code is near the bottom of the page.



So there you go a one stop shop of goodies that should help you on your way. Please let me know if you have other useful DMV related queries or anything specific to SQL 2008 that is coming our way soon.