Showing posts with label Proxies. Show all posts
Showing posts with label Proxies. Show all posts

Friday, 29 April 2016

Chome and FireFox really getting on my tits....

Chome and FireFox really getting on my tits....

By Strictly-Software.com

Chome and FireFox really getting on my tits....

Chrome was by browser of choice, due to being light weight and fast.

FireFox was in 2nd place due to the range of plugins available.

I had relegated IE into usage only to test code for cross browser compatibility issues.

However I am finding that I am actually using Internet Explorer more and more due to constant issues with both of the latest versions of these browsers.

I am running Chrome: 50.0.2661.75 (64-bit) And FireFox 46.0 buld no: 20160421124000 (64 bit) on all 3 of my machines (Win7 & Win 8.1)

There was a stage when both these honey's were humming like a bees. I even put up some articles on how to improve the speed on both browsers:

Speeding Up Chrome Can Kill It
Speeding up Google Chrome with DNS Pre-Fetching
Performance Tuning FireFox

I also put up a general PC and Browser tune up article with free tools, command line prompts and some basic things to try if you had a slow computer: Speeding up your PC and Internet connection.

However I have even found myself using IE 11 more and more due to constant hanging, pages not loading at all with the "processing request" message in the footer, or waiting for some 3rd party non asynchronous loaded in script, to download and run that blocks the site or page from running.

I think there is a far too much "API JIZZ" in the community at the moment.

What this means is that developers, due to their nature to impress and gold plate code, even when the spec doesn't call for it, are now using so many 3rd party and remotely hosted plugins like jQuery, Google Graphs, tracker code, plus loads of funky looking CPU consuming widgets to make their pages look good.

You only have go into Facebook or G+ and try and write a message. Not only will Google Plus's new post box move around the page before you can start writing, but both websites are constantly analysing your keystrokes to find out if the previous string matches a contact, community or page, in your contact book for them to link to.

The more people and pages you have stored the slower this process becomes. Yes is might be handy but why not just require a symbol like + in Google+ to be put before the person name so that the code only checks that word for a relation.

Imagine having a list of thousands of pages, liked communities/pages and contacts to be constantly checked on every keydown press with AJAX requests. That is overkill. It slows down systems .

I still have two windows from Chrome spinning away for (Google Blogger blogs) at the moment. There is not much 3rd party code on these pages but they are having trouble and showing common "Waiting for Cache" and "Processing Request" messages in the status bar.

This is the same sort of thing I get in FireFox. Although in this browser, what kills me is just the slowness of getting from page to page. On many sites I have to refresh it multiple times before the code all loads and this goes for online banking to online betting sites. Just trying to watch a race on their Flash screens is a nightmare.

I had a bet on a horse the other day on Bet365.com just so I could watch the big race with an unbeaten in 11 straight wins, Douvan, running. However Bet365.com video didn't start and in SkyBet it was stuttery and kept losing picture and sound. I missed the end of one race where a horse I had backed jumped the last fence into the lead but when the picture came back it had finished 3rd!

They keep telling me to clear the cache, reboot the router and do speed tests. Things I have done many times. I have 54Mbps download speed at work and 28Mbps at home. I can stream 4k UHD TV to multiple screens so download speed is not the issue something else is.

Speedof.me is the best online speed testing site I have found as it as it uses no extra files and is ran in pure HTML5 with no Flash, Java or ActiveX type objects requiring to be loaded for it to run.

What is causing the problem I have no idea as my broadband speed seems okay. I suspect it's the large number of reverse proxies being used and the download of shared 3rd party scripts and widgets that can hang due to a large number of HTTP requests.

I tried deleting my userdata file for Google by searching for it in the address bar of Windows Explore with this line: %USERPROFILE%\AppData\Local\Google\Chrome\User Data 

I have also tried disabling Flash as so many times I see the "An object has crashed" bar in the header that is related to the Flash Container object failing. Sometimes a reload works other times it doesn't.

However so many sites STILL use Flash it is hard to live without it really. For example the WHOLE of Bet365.com is made in Flash which makes it very user unfriendly and hard to use with sticky scrollbars and issues with selection of items.

If anyone has similar issues or ideas on resolving them let me know, as I never thought I would be going back to IE to use as my main browser!

By Strictly-Software.com

©2016 Strictly-Software.com

Thursday, 9 January 2014

Download and Decompress a GZIP file with C Sharp

Using C# to download a remote GZIP file and then decompress it

By Strictly-Software

Recently I had the task of writing a program in C# that would obtain a list of proxies from various sources and then run code to ensure that they were working so I could then use the useful ones.

In this project I had a window that showed the proxy IP address, Country it came from and whether it was Anonymous, High Anonymous, Transparent etc.

I also had a check button which once the proxy list had loaded would run a test against each IP address and Port No to see the time that it took to do the following:
  • Ping the proxy if possible e.g 408 ms
  • HTTP ping the proxy by using the details to request one of a randomly selected number of fast loading pages e.g www.google.com, www.bing.com etc.
Personally I think the HTTP ping is more important when dealing with proxies than a normal PING.

A simple ping to an IP address could respond very quickly or not at all but when you are using Proxies in computing to request HTML pages you want to know how fast it takes to return such a page.

Anyway the whole point of the exercise was that I needed to have a list of countries that I could check the IP addresses against.

Luckily the great site http://geolite.maxmind.com have a free GeoIP.dat.gz file that you can download and use that is pretty accurate (but not as accurate as the paid for version). However the free version was good enough for what I needed.

The issue was that the .dat file came as a GZipped file and once I had downloaded it I needed to decompress it. This wasn't the normal .zip decompress but in .NET 4.5 it is pretty easy to accomplish.

I have shown you a basic example of the class at the bottom of the page but the most important function is the method which does the Gzip decompression.


/// 
/// Decompress a gzipped file to compress we can just use the CompressionMode.Compress parameter instead
/// 
/// 
public static void Decompress(FileInfo fileToDecompress)
{
    using (FileStream originalFileStream = fileToDecompress.OpenRead())
    {
 string currentFileName = fileToDecompress.FullName;
 string newFileName = currentFileName.Remove(currentFileName.Length - fileToDecompress.Extension.Length);

 using (FileStream decompressedFileStream = File.Create(newFileName))
 {
     using (GZipStream decompressionStream = new GZipStream(originalFileStream, CompressionMode.Decompress))
     {
  decompressionStream.CopyTo(decompressedFileStream);                        
     }
 }
    }
}
The three libraries you will need to accomplish all this apart from anything else you intend to do will be:

using System.Net
using System.IO;
using System.IO.Compression;

System.Net is required for the WebClient class to do it's work downloading the remote file to our computer and the System.IO one is required for checking that files and folders exist.

The last one is the most important, System.IO.Compression as it's the library that lets us decompress the file.

You might have to add this in as a reference in Visual Studio. Just go to: Project > Add Reference > Framework > and tick the box next to System.IO.Compression.

Also note that I am using .NET 4.5 on a Windows 7 64 bit machine. In Windows7 for security sake (I presume) most applications that need to write and read to a file, or download and hold data of some sort is done in the new C:\ProgramData folder.

You will notice that this directory is full of well known names like Microsoft, Skype, Sun, Apple and any many other software producers that needs somewhere to log data in a safe place.

In the old days people could just write programs that saved files all over the place which obviously wasn't safe. Especially if you were the admin of the computer and hit a button on a program that you thought was going to do one thing but was actually adding or deleting files all over your computer's hard drive.

Anyway the whole code is below. Make of it what you will but it's pretty simple and I found it very useful.


using System;
using System.Linq;
using System.Text;
// we need this to download the file from the web
using System.Net
// these are the two we need to do our decompression job
using System.IO;
using System.IO.Compression;

// this will hold any error message incase we get one and need to return it to the calling program
private string ErrorMessage = "";

/// 
/// Ensure our special folder in C:\ProgramData\ exists e.g C:\ProgramData\MyProgram
/// Then check the file we need to get countries related to IP's exists from http://geolite.maxmind.com and if it doesnt' download it
/// and copy it to this folder. Then we need to decompress it as its a gzip file e.g GeoIP.dat.gz so we need the uncompressed GeoIP.dat file to work with
/// 
public SetUp()
{
    // ensure a folder we can write to exists - named after my program
    dataFolder = Environment.GetFolderPath(Environment.SpecialFolder.CommonApplicationData) + @"\MyProgramName";

    if (!Directory.Exists(dataFolder))
    {
 try
 {
     // The folder doesn't exist so try and create it now
     Directory.CreateDirectory(dataFolder);

 }
 catch (Exception ex)
 {
     // set a global error message we can return to the calling object
     this.ErrorMessage = "The data folder could not be created: " + ex.Message.ToString();

     return;
 }
    }

    // we have a folder but do we have an uncompressed .dat file?
   
    // set up the paths
    // first the path of the uncompressed .dat file in case we already have it
    string geoCityDataPath = dataFolder + @"\" + this.GeoLiteCityDataFile;

    // then the path of the .dat.gz compressed file in case we need to uncompress
    string zipFilePath = dataFolder + @"\" + this.ZippedGeoLiteCityDataFile;

    // check for an uncompressed data file
    if (!File.Exists(geoCityDataPath))
    {
 try
 {

     // we don't have a file so download it from the website and copy it to our folder
     // we could schedule this behaviour to get the latest file by checking the dates or just doing a download once a week/month
     WebClient webClient = new WebClient();
     webClient.DownloadFile("http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz", zipFilePath);

     // now we have our file create a FileInfo object from it to pass to our gzip decompress method
     FileInfo gzFileInfo = new FileInfo(zipFilePath);

     // Call the method to decompress the gzip file
     this.Decompress(gzFileInfo);
 }
 catch (Exception ex)
 {
     // set a global error message we can return to the calling object
     this.ErrorMessage = "The GeoIP.dat.gz file could not be downloaded or decompressed: " + ex.Message.ToString();

     return;
 }
    }
}


/// 
/// Decompress a gzipped file to compress we can just use the CompressionMode.Compress parameter instead
/// 
/// 
public static void Decompress(FileInfo fileToDecompress)
{
    using (FileStream originalFileStream = fileToDecompress.OpenRead())
    {
 string currentFileName = fileToDecompress.FullName;
 string newFileName = currentFileName.Remove(currentFileName.Length - fileToDecompress.Extension.Length);

 using (FileStream decompressedFileStream = File.Create(newFileName))
 {
     using (GZipStream decompressionStream = new GZipStream(originalFileStream, CompressionMode.Decompress))
     {
  decompressionStream.CopyTo(decompressedFileStream);                        
     }
 }
    }
}

Wednesday, 6 March 2013

Internet Censorship and Privacy - How they track you

Internet Censorship and Privacy - How they track you

This was taken from the www.darkpolitricks.com site.

Internet Censorship and Surfing Anonymously

By Dark Politricks

Sometimes it feels like the good old days of the Internet and being anonymous have passed - and you would be right.

With restrictive and snooping laws being passed all over the world, firewall filters wrapped around whole countriesTwitter users sued for Re-Tweeting libellous claims and big tech companies working hand in hand with the biggest security forces on the planet there really is no way to escape.

However there are ways to minimise your "footprint" and if you are not a serious criminal or terrorist then you shouldn't have anything to fear.

However if you are then you're probably being watched through your webcam right now whilst your iPhone's microphone is being channelled into GCHQ or Langley for analysing - tough luck!

From a users perspective the Internet contains a myriad of security and privacy issues which if the user is not aware of could cause potential problems on all manner of levels.

For the privacy conscious person who wants to be able to surf the net without worrying about someone looking at the content they have visited in real time or at a future date e.g your work, government or Police then there are a number of issues they need to be concerned about.

With the recent dismissal of the head of America's most powerful spy, David Petraeus, and knowledge about the way he was caught it is good to know the way's you are tracked so you can choose whether you want to take that risk.

As with most web content if you wanted to be 100% anonymous on the web it will be pretty hard to do.

If you want to stay totally anonymous you should probably move underground somewhere as there is always a satellite up there somewhere and with Google Earth you cannot even escape commercial companies anymore. So moving to the woods to live in a hut without electricity or broadband is not even an option anymore!

However there are various forms of tracking that you should be aware of so that you can limit the risks to you whilst surfing or using the Internet or phone. These don't have to be to hide from the Government but could just be to prevent your personal details from being sold to advertisers or having horrible popup boxes show when you close a window.

This is not a comprehensive list but it is a start and it is also one that is constantly changing as technology changes. When I wrote this originally tablets (iPad etc) were not that common. Now they are just another place to accumulate your browsing history and a tool to be used against you if you are ever in that unfortunate position.

Emails

This is is how the head of the CIA was caught out. He wasn't sending the emails but he was saving them as a draft under an anonymous Google account and then letting his mistress login and read them.

This way there was no Internet trail as when you send an email the mail is routed from server to server and the IP addresses of the mail servers it travelled through are recorded in the headers of your mail. 

Check it yourself. If you use a mail client find the option to "View All Mail headers" and then view an email that has been sent to you. At the top you should see the details of the route that the email took which should show the originating mail server, the receiving mail server and the IP addresses of any it traveled through in-between.

When people use this draft save only technique they are trying to avoid this trail. However you are defeating yourself in the first instance by:

a) Using an online mail server such as Gmail or Hotmail. All the data including drafts are saved on THEIR servers i.e "in the cloud" so if they disable your account, or are served with a warrant there is a good chance your draft emails will be accessed and read along with your sent, read and junk emails.

b) If you don't hide your origin when signing up for a throwaway Gmail or Hotmail account (you have to fill in a form to get one in the first place) then they will still get your IP address unless you have gone through known secure Proxy servers or used someone else's computer without their knowledge (e.g an open Wifi router). Do a scan on your PC / Phone now and see if there are any around you. Open ones won't have a lock symbol next to them.

c) Remember that Microsoft computers store all deleted emails, web history and other files even when you think they have been deleted on your computer. Here is a very old article from 2000 which shows even back then Microsoft was hiding emails, web searches and other files from users. The scripts and batch files you might find if you search for "Microsofts Really Hidden Files" probably won't work anymore but they probably have no need for such old methods anymore especially when people are buying computers and defaulting them to backup everything to the cloud.

How to bypass

Don't use "cloud based" systems that are well known to have links with the US security services (Microsoft and Google have - as I have shown and both these mega companies are actively helping the US spy agencies with their own huge database). Read this article on why we are sleep walking into a surveillance society by consent. Then ask yourself whether using any of these big Internet companies software is safe, especially as they seem to gobble up smaller companies by the minute. Why create back-doors when you can walk in the front I asked myself when Microsoft bought SKYPE.

It may pain you but by using any Internet based email service it means recording the data as it leaves the device you write your message on, storing it on a computer system you have no control over and by signing the Terms and Conditions you have allowed them to "own" your data and use it for advertising and God knows what else.

Plus nothing leaves the Internet, you can view cached versions of Google (or any site) all the way back to the 90's on this site: http://archive.org/web/web.php

If you don't want your own website to appear on this search engine that archives everything forever then they are pretty good about obeying the robots.txt directives so you can put this in your robots.txt file (read about it here) to prevent that site indexing your site.

# alexa archiver
User-agent: ia_archiver
Disallow: /

To be really sure you can block the IP 207.241.224.41 in your .htaccess file or at your firewall if you wanted to stop them crawling your robots.txt file at all.

Also use throwaway email addresses if you can or even create your own with some basic scripting (not hard if you can be bothered) and put it on a server in another "more freedom friendly country" and use proper proxies to access the webpage front end to send your emails (the part about proxies come later on).

The need for these disposable email systems sprung out of the need for a quick email address to sign up to a site or set up an anonymous blog or anything else that you don't want all the spam emails that follows. I also have found with some basic hacking you actually use them to send AND receive email - it all depends on how good the programmer is.

Either do a search for "Disposable Email Addresses" to find the latest ones or check out guerrillamail.com or Mailinator  however a word of warning - there are lots of disposable email accounts out there who knows who really owns them? If you do use them make sure their URL starts with https:// (this means data is encrypted from your PC to their server).

As for Internet files you can use "cleaner" tools like CCleaner to remove cookies, old registry files, old programs, start up applications and Internet history easily. Plus you should never use Internet Explorer anyway, as there are a myriad of more security conscious browsers such as FireFox out there which are much better on the privacy front as they are not tied into the operating system of your computer.

Using the Cloud to backup your "Secure" data

Storing anything on anyone else's computer means you don't have control over it. Therefore cloud based storage systems should be avoided for anything personal or secure. Even phones nowadays have settings to automatically back up your numbers, texts, photos and videos.

It might be good if you ever lose your computer or phone but remember that if the cloud based backup server is in the USA then they are probably sniffing everything inputted into it anyway.

This is  the same for Facebook, Google, Tumblr and any other social media site. As soon as you put anything on that site THEY own it and if they are served a subpoena or warrant to hand the data over there is nothing you can do.

How to bypass

It may pain you but just don't use Facebook, Google, Twitter or any other social media system if you are going to put anything dodgy on it.

The same goes for dropbox and any other web based storage centre that may have to hand over your data to the authorities one day.

If you must keep files secret then keep a portable external hard drive at home and backup all your files to that device before hiding it. At least that way you have control and ownership over the backed up data and you are the only person who knows where it is located.

Using your computer hardware to spy on you

There have been many cases lately where computers and phones have been used against their owners to spy on them.

Last year there was a big outcry about iPhones "secret" database that logged all the GPS positions you had been to with your phone. Even without GPS they can use phone mast triangulation to find a near enough point that your phone pinged the mast.

Also we had the case of cops using tools to download this data as they pulled motorists over or arrested people and then illegally accessed this database of locations to find out where you had been.

As for computers we had a school in Lower Merion school district in Philadelphia that was accused of spying on students in their bedrooms via school issued laptops and the webcams built into them. Would you want a headmaster in his office alone at night watching your kids in their bedroom?

How to bypass

Take the battery out of your phone whenever you don’t want to be tracked.

As the earlier report shows cellphone triangulation tracking takes less power than GPS tracking and even when your phone is turned off a tiny amount of battery charge is available to the phone which is enough to log your presence at a nearby tower and then log your presence down to the nearest 100 metres or so. 

Either that or use pay phones or pay passers by to use their phone when you need to make a call when your phone is unavailable.

On your computer turn off the microphone and webcam with your settings e.g on Windows it's in Control Panel. On Windows 7 it will be under Speech Recognition and Audio Devices.  To be extra safe wrap masking tape over the webcam when you don't want to use it as well as the speaker (blue-tack or something else that would muffle the sound). Anything that can be useful to you can be useful to someone with control of your computer.

To test if your microphone is working either go into your computers settings e.g control panel or go to the old Google Search Engine (if it's still available at http://www.google.com/webhp?hl=all ) hit the microphone symbol in the input box and talk.

If you see the blocks under the microphone move up and down and then a result similar to what you said appear in the box - the microphone is still on. If it's off it will say so.

Javascript urchins

These are little bits of script that are added to the source code of the HTML page you are visiting. They use JavaScript to record identifying features about the user and their browser such as the user-agent, system details and location by calling a script on another server that then logs these details to a central database. A good example is Google Analytics which most sites including this one use to tell the owner about the amount of traffic they receive and where it is coming from.

How to bypass

Turning off Javascript will prevent this logging from occurring. You can do this in most browsers through their Tools > Options settings but you can get toolbars and add-ons like the FireFox Web Developer Toolbar or the NoScript add-on that do this for you.

Webbugs

Similar to urchins these are little images, usually so small they cannot be seen, that point back to a web server and run some code whenever the image is loaded by a client. They tend to be used by email marketing tools and are embedded within HTML emails so that they can record who has actually opened the email and track the email if its forwarded it on. 

They can also exist on web pages or within desktop applications and as the image is hosted remotely whenever it is loaded it records the location of the application or user who is loading it.

How to bypass

Many email clients if they don't do it automatically have the option to display emails as plain text which would prevent these webbugs from working. I use Thunderbird which is free and you can set to ask you first whether to load any remote content at all whether they are images, scripts or anything not already embedded within the email.

 In Browsers you can disable images easily with the Web Developer toolbar, Google Chromes privacy settings or by using a text browser like Lynx.

Server Side logging by the page

Most pages on the web nowadays are more than pure HTML/CSS and contain code that runs server side e.g .asp, .php, .jsp, .aspx etc.

When the page is requested the web server parses the page and runs any code before returning the generated HTML to the client. This code has access to a lot of information about the client requesting the page such as IP address which can be used for GEO tagging, User-agent details, accepted file types and other information contained within the headers. They could choose to log this information to a database or file if they wanted to even if the IIS or Apache web server had its own logging disabled.

For example if you got to whatsmyip.org you will see all the information that is passed to each webpage you request including geo-location information, details about the type of computer you are using and much more. Whilst not totally accurate they can pinpoint the last location of the computer used to access a webpage which could be your own PC or could be someone else's (if you use a proxy - see below).

How to bypass

Please read the guide under the following section about web server logging as it applies to both.

Logging by the Web Server

Every time you make an HTTP request e.g access a web page, a record is made on the web server that hosts that page to a log file. Each separate file contained within that web page is logged so every image, CSS file and script is logged along with your IP address, the method e.g POST or GET, the URL, bytes sent and received and much much more.

Although its possible to turn off this logging most companies running web servers require these logs for traffic analysis e.g with a tool such as Webtrends as it helps analyse traffic from all agents including robots who do not have JavaScript support. Also many countries now require ISP's to keep log files for up to a year or more in case the data is required at a later date.

How to bypass

As you must assume that the web servers you are visiting sites on have logging enabled then the only way to not get tracked is to go through proxy servers or use tools like the FireFox add-ons Modify Headers or Tamper Data which allow you to change the headers sent from your PC to the webserver in question and act as a mini proxy on your own PC. They cannot however change the REMOTE_ADDR header which holds the IP address of the PC making the request.

Another way is to turn your PC into a webserver through free software like WAMP Server and then create a web based proxy for your surfing. The good thing about this is that in the remote servers log files all they will see for an IP address is 127.0.0.1 which is the local loopback IP address and cannot be used to track anyone as every PC uses that address.

Remember a proxy is just an intermediate server that sits between you and the web server you want to access. If someone was tracking you they would only see your request to the proxy server and not the actual content that the proxy server requests on your behalf.

There are various forms of proxy some that are anonymous and others that pass your IP address along in the HTTP_FORWARDED_FOR, HTTP_X_FORWARDED_FOR or any number of other headers. You can also use code or tools to fill these headers with random IP addresses to make it harder for a tracker to find you as it will look like you have bounced round a lot of proxies when in fact you haven't.

There is also a form of proxy known as an "anonymizer", which is called this because it hides all the users identifying information such as headers that hold the IP and user-agent. There are lots online for you to use.

Anonymizers are not entirely secure. If an anonymizer keeps logs of incoming and outgoing connections and the anonymizer is physically located in a country where it is subjected to warrant searches then there is a potential risk that government officials can reverse engineer and identify all users who used the anonymizer and how they used it.

Most anonymizers state they do not keep logs but there is currently no way to confirm that. However, if the user used another anonymizer to connect to the exposed anonymizer, that user is still anonymous. This is sometimes called daisy-chaining

The safest way therefore is to use a chain of proxy servers to make your requests or use a specialist service like TOR which is designed to make it hard to track Internet usage.

P2P Torrents

People use torrents to download films, music and other software. Sometimes these are illegally obtained copies or pirated software.

The Pirate Bay was one of the most famous sites that people used to obtain torrents and the people behind it are currently involved in legal action as the US movie industry is trying to sue them for facilitating the illegal download of copyrighted material.

Even though they are just a search engine on the same lines as Google or BING (and you can find torrents on those search engines as well!) - it is pretty unfair as the Pirate Bay are not uploading the films themselves they are just a search engine that lists files of a certain type.

When you download torrents you use special software such as uTorrent or Deluge to download all the tiny pieces of the file you want. The idea is that because you are not downloading a whole file from one location but rather tiny bits of it from lots of locations you are not really breaking the law.

When you download you are a "leecher" and when you upload you are a "seeder". The software simplifies all this when you download a file as it connects all the tiny bits up for you so you don't have to worry about where they are coming from. 

Also as you download you are also uploading the bits you have already downloaded so other people can obtain them. You can change your settings to prevent the uploading part of this if you want to by changing the ratio of upload versus download or the rate/speed that you upload (or even turn it off).

The Priate Bay was the biggest site on-line which is why it is being targeted and if you try accessing www.thepiratebay.org in your browser now I bet you it will be blocked by your ISP.

How to bypass

There are many proxies for the Pirate Bay which will allow you to access the site from a different URL. Just search for "Pirate bay proxies" and then pick one. 

You might find an advert at the top of the page counting down - this is a way to access the site once the count is down to 0. Ignore the main part of the page and click on the "view" button that might appear after the countdown in the top right corner which should take you to the pirate bay proxy. 

You might have to try a few out first but I use https://piratereverse.info. As soon as the ISP shuts one down another one will pop up (just like the thousands of people who pointed domains at WikiLeaks when it was blocked) so you will always be able to find a site to get them from whether it's the Pirate Bay or a User Group or discussion board.

Also beware that many torrent tools will be flagged as Trojan down-loaders (even when they are not) and also that ISP's and other government organisations insert their own trackers that log the IP addresses of people downloading the torrents so that they can contact/blacklist/reduce your bandwidth etc. Therefore be careful and pick a good one and read up about trackers before engaging in torrent downloading.

To make the chance of being caught a lot less you can should change your torrent tool settings to go through a proxy server - preferably HTTPS (encrypted) or use any option that forces encryption when transferring files. 

You should also change the port used by the tool in your settings from a random number to 80 or 8080 as these are common webserver ports and make it hard for ISP's to tell what kind of traffic is being transferred. If possible use a "block list" that will mean that all the data packets sent to or from you will bypass known ISP routers where they can be sniffed and identified. More and more ISP's are doing this so this is wise to prevent yourself from being caught.

Read these articles to help you install a torrent down-loader and set-up measures to prevent yourself being blocked.



Cookies

Cookies are small text files that are stored on the clients computer and contain very small pieces of text. They are mainly used by websites to store flags that enable the site to know whether you have previously been to their site or not. Advertisers also use them to track the type of sites you visit so that they can deliver targeted advertising the biggest offender being Google which uses their domination of the market to track the sites users visit so they can target content specific adverts to the user.

Another type of cookie is a session variable which is used by many sites to store a unique ID that refers to a visit on the site. The ID is generated by the web server and the session cookie only stores this ID so that on each request to the server the system knows that the visitors requests belong to one visit.

How to bypass

If you are concerned about tracker cookies then you easily disable site related cookies in your browser but if you disable all cookies then Session variables won't work and you will most likely find yourself getting logged out of member only areas of websites or not being able to login in the first place.

The best option is to disable 3rd party cookies (those set by advertisers) and to delete non essential cookies after using the Internet (Incognito mode in Chrome).

Flash, ActiveX, Java Applets

3rd party components such as Flash, ActiveX controls and Java applets come with their own security concerns. There have been numerous security vulnerabilities reported with these types of component as due to their complexity and power they have more access to the clients computer than a normal web page. They should be seen as mini applications rather than just a fancy banner, game or helpful utility to enable you to upload files to Facebook more quickly.

You shouldn't install these types of application unless you are totally sure they are safe as they could have a lot more control over your computer than you realise. There have even been hacks that have enabled remote users to video and record a user through their webcam without them knowing.

How to bypass

You can use Firefox extensions such as FlashBlock or AdBlocker to disable flash on specific pages or the Opera browsers Turbo mode which speeds up page loads as well as allowing you to choose which flash movies to play. If you decided to choose privacy over anything else then you will end up having a pretty boring web experience as more and more sites use Javascript and Flash to deliver interactive content.

However if you are really security conscious you should use a text browser such as Lynx which won't load images, flash, JavaScript or any other form of plug-in. It will show you the textual content of the pages you visit and will ask if you want cookies to be stored for each request. Due to only loading text and links you will have fast load times so there is a benefit to having a reduced web interface.

You should also regularly check your PC for viruses and spyware. One of the first things modern Trojans do nowadays is download good anti-virus software so that they don't get overwritten by another spyware app!

They also try to disguise themselves as virus checkers to avoid detection. Even the best off the shelf virus checkers don't catch all forms of spyware especially those that have to regularly download virus definition patterns as it means new viruses don't get caught until they have been identified, a pattern created and downloaded by the client.

Virus payloads can also be modified randomly to avoid pattern detection so tools that don't use pattern matching such as hijackthis.exe which runs an analysis of all currently running processes looking for odd behaviour are good tools to use. This tool will generate a report which can then be analysed by members of the special Hijackthis.exe message board for signs of infection.

One of the best removers of Trojans I have found is a tool called SDFix.exe which managed to detect and remove a Trojan that four other tools including an off the shelf app didn't detect. There are also a number of good free products such as MalwareBytests Anti-Malware and AdAware anti adware and spyware software which can be run regularly to check your PC for spyware and viruses.

However keyloggers that are based around hardware such as cable extensions that you don't notice that have been inserted by your employer are undetectable unless you know what you are looking for and will store every key pressed on your PC whilst enabled. Check your cables that come out your computer to see if anything strange is connecting two parts of a wire together.

If you are caught out by such a tool make sure your employer has followed the law by informing you of any anti-privacy measures he or she may have introduced such as monitoring your PC and web usage in your contract. If they haven't then you have a good legal case to sue  them and they are breaking the law by spying on you without your knowledge. The same goes for CCTV, recording devices and other means of logging your activity without your knowledge.

Article 8 of the Human Rights Act that is used in the UK has been successfully used in previous cases by employees who have been sacked due to unknown spying by their employers and should be used by anyone taking their employer to court if they have been sacked due to such technological spying.

Tools to use to aid privacy on the web Firefox Add-Ons
  • Web Developer toolbar. Disable Javascript, cookies, view cookie and header info, modify the DOM, view generated source code, show password fields.
  • Flashblock disables flash movies until you enable them. Allows creation of a white-list of allowed sites.
  • FoxyProxy manage your proxies with an easy to use tool.
  • Tamperdata or Modify Headers acts like a proxy and allows you to modify HTTP requests as they are made from your client.
  • HTTP Fox, Firebug and even the Chrome developer toolbar allows you to see all the data your PC send to websites and the data sent back by the webserver you are accessing. It also shows any redirects or code loaded in that you might not be aware of.
Google Chrome
  • Use Incognito browsing to prevent browser and search history and cookies from being stored.
  • Firefox and IE9 also have privacy modes that can be used to remove cookies and reduce your internet footprint but I would not trust anything Microsoft as it's hooked into the computers main system and parts of the browser are shared with other non Internet based software.
All browsers
  • De-activate Javascript, VBScript (IE only) until you know the site is safe.
  • De-activate 3rd party cookies used by trackers, advertisers and sites wanting to keep track of you as move around the web such as Google Analytics.
  • If you share a PC Clear your cache, autocomplete, download list and history regularly - use CCleaner, AdAware etc.
If you need more details about the various forms of Internet Censorship and how to bypass it then check out the following article that contains a lot of details about the various methods used and how to bypass them.

How to bypass Internet Censorship If you are looking for an up to date list of available proxy servers then you can check out the following links:


The following page has an index where you can find more proxy lists 


If you want to quickly access some web based proxies you can pick from the following list or you can read my guide on creating your own web proxy which comes with an example and some code you can use to get running quickly.

Read the original article at www.darkpolitricks.com.

Monday, 22 March 2010

Write your own Proxy Checker Tool

Creating your own Proxy Checker Tool

Finding good reliable proxies is a hard task and although many sites offer free lists the majority of the proxies on them will be out of date or not working when you get round to testing them. If you are happy with using WebProxies then there are plenty about but they don't help if you are wanting to access content that utilises complex AJAX to deliver movies or other country specific goodies you want to view that is being blocked.

Therefore its a good idea to have a proxy checker tool that can be run on demand to find working proxies. There are many tools you can buy or download for free such as Proxyway that will do this but the problem with any executable is that you don't know what the code is doing behind the scenes.

Any executable that you run on your PC that is contacting multiple servers in Russia and China should be used with caution as these are well known hotspots for hackers and utilising hidden malware inside otherwise useful tools is a well known tactic.

Therefore its always a good idea if you can to write your own code. I have just knocked up a Proxy Checker tool for my own use that not only finds useful proxy lists at the click of a button but also checks the proxies within those lists to see if they are working.

1. The code is written in PHP and uses an HTML form and multiple AJAX requests to load the results into the DOM. This makes the script very usable as your not waiting around to see the results and the page is updated as each result comes in giving it the look and feel of a real time app.

2. If you don't have access to a webserver to run PHP from then install WAMP Server on your PC. This will allow you to run scripts from your localhost plus you can enable all the extensions that a webserver may not let you use such as CURL or FOPEN. I do like running "personal apps" from my localhost as it means I get all the flexibility of a webserver plus no-one else can use the code!

3. The HTML page contains a form with a textarea. Paste in all the URL's containing the ProxyList sites you want to scrape. If you don't have a list of proxy sites then you can use the "Find Proxy List" button to scan the web for some. This is not an extensive search but it will return some lists. Remember good quality proxy lists are hard to come by and a quiet proxy is a quick proxy therefore if you find a good reliable proxy server keep it quiet and don't destroy it!

4. On hitting the "Check Proxies" button the form loops through the Proxy Lists URL's making AJAX calls to a helper script that scrapes any proxy details it can find. I am using a basic scraper function utilising file_get_contents but you can use CURL or fsockopen if you wish or like I do on other sites a custom function that utilises all 3 in case the server has blocked the use of one or more options or if your CURL settings don't allow you to use Proxy Tunnelling.
// A very simple function to extract HTTP content remotely. Requires fopen support on server.
// A better function would check for CURL use and fallback on fsockopen
function getHttpContent($url, $useragent="",$timeout=10, $maxredirs=3, $method="GET", $postdata="",$proxy="") {

$urlinfo = null;

// simple test for a valid URL
if(!preg_match("/^https?:\/\/.+/",$url)) return $urlinfo;

$headers = "";
$status = "";

// create http array
$http = array(
'method'=>$method,
'user_agent'=>$useragent,
'timeout'=>$timeout
);

// add proxy details if required
if(!empty($proxy)){
$http["proxy"] = $proxy;
}

// if we want to POST data format it correctly
if($method=="POST"){
$content_length = strlen($postdata);

$http["content"] = $postdata;
$headers .= "Content-Type: application/x-www-form-urlencoded\r\nContent-Length: $content_length";
}

// now add any headers
$http["header"] = $headers;

// set options
$opts = array('http'=>$http);

// create stream context
$context = stream_context_create($opts);

// Open the file using the HTTP headers set above
$html = @file_get_contents($url, false, $context);

// check global $http_response_header for status code e.g first part is HTTP/1.1 200 OK
if(isset($http_response_header[0])){
// Retrieve HTTP status code by splitting this into 3 vars
list($version,$status,$msg) = explode(' ',$http_response_header[0], 3);
}

// if we have a valid status code then use it otherwise default to 400
if(is_numeric($status)){
$urlinfo["status"] = $status;
}else{
$urlinfo["status"] = "400"; //bad request
$msg = "Bad Request";
}

// only return the HTML content for 200=OK status codes
if($status == "200"){
$urlinfo["html"] = $html;
//put all other headers into array in case we want to access them (similar to CURL)
}elseif(isset($http_response_header)){
$urlinfo['info'] = $http_response_header;
}

// return array containing HTML,Status,Info
return $urlinfo;
}



5. The content is decoded to get round people outputting HTML using Javascript or HTML Encoding it to hide the goodies. It then looks for text in the format of IP:PORT e.g
// call function to get content from proxy list URL
$content = getHttpContent($url, "",10, 3, "GET");

// did we get a good response?
if($content["status"]=="200" && !empty($content["html"])){

// extract content and decode it to get round people using Javascript to hide HTML
$content = urldecode(html_entity_decode($content["html"]));

// now look for all instances of IP:PORT
preg_match_all("/(\d+\.\d+\.\d+\.\d+):(\d+)/",$content,$matches,PREG_SET_ORDER);

6. I then return the list of IP's to the front end HTML page which outputs them into a table with a "TESTING" status. As each unique IP:PORT is inserted into the report another AJAX call is made to test the Proxy Server out.

7. The Proxy test utilises the same HTTP scraper function but this time it uses the IP and PORT details from the Proxy we are wanting to test. The page it calls is one of the many IP Checker tools that are available on the web. You can change the URL it calls but I am using a page that returns the Country after doing a reverse IP check. This way if the proxy is working I know the country details.

8. Once the reverse IP test is carried out on the Proxy the results are returned to the HTML report page and the relevant Proxy listing is updated in the table with either GOOD or BAD.

I have found that a lot of other Proxy checker scripts are only validating that a proxy is working by giving it a PING or opening a socket. Although this may show whether a server is accessible it doesn't tell you whether using it as proxy will work or not.

Therefore the best way to test whether a Proxy is working is to check for a valid response by requesting a page and if you are going to call a page you might as well call a useful page. One that will return the IP's location or maybe one that shows any HTTP_VIA, FORWARDED_FOR headers so you can detect whether the Proxy is anonymous or not.

Remember when you find some good quality proxies store their details as they are worth their weight in gold!


Removed. I'm going to sell this bad boy!