Showing posts with label ISP. Show all posts
Showing posts with label ISP. Show all posts

Thursday, 11 August 2022

Testing For Internet Connectivity

Setup Measures For A Big API System

By Strictly-Software

I have a BOT that runs nightly, and in the SetUp() methods to ensure everything is okay, it runs a number of tests before logging to an API table in my database. This is so I know if the API I am using is available, whether I needed to login to it, when the last job was started and stopped, and when the last time I did log into the API. 

It just helps in the backend to see if there are any issues with the API without debugging it. Therefore the SetUp job has to be quite comprehensive.

The things I need to find out, which are logged into the logfile.log I keep building until the job is finished with major Method Input Params, Return Params, Handled Errors, Changed System Behaviour, SQL Statements that return an error e.g timeout, an unexpected divide by zero error and the like are the following from my SetUp job.

1. Whether I can access the Internet, this is done with a test to a page that just returns an IPv4 or IPv6 address. The major HTTP Status code I look for there is a 200 status code. If I get nowhere but get a status code of 1 that is a WebExceptionStatus.NameResolutionFailure, which means the DNS could not find the location and it's probably due to either your WIFI button being turned off (Flight Mode), or your Network Adapter having issues.

I test for web access with an obvious simple HTTP test to an IP page that returns my address, with 2 simple regular expressions that can ID if it's IPv4 or IPv6.


public string TestIPAddress(bool ipv4=true)
{
    string IPAddress = "";
    // if we are using an IPv6 address the normal page will return it if we want to force
    // get an IPV4 address then we go to the IPv4 page which is the default behaviour due to
    // VPNS and Proxies using IPV4 usually
    string remotePageURL = ((ipv4) ? "https://ipv4.icanhazip.com/" : "https://icanhazip.com/");
   
    this.UserAgent = this.BrainiacUserAgent; // use our BOT UserAgent as no need to spoof
    this.Referer = "LOCALHOST"; // use our localhost as we are running from our PC
    this.StoreResponse = true; // store the reponse from the page
    this.Timeout = (30000); // ms 30 seconds            

    // a simple BOT that just does a GET request I am sure you can find a C# BOT example on my blog or tons on the web
    this.MakeHTTPRequest(remotePageURL);            

    // if status code is between 200 & 300 we should have data
    if (this.StatusCode >= 200 && this.StatusCode < 300)
    {
		IPAddress = this.ResponseContent; // get the IP Address

		// quick test for IPAddress Match, IPv4 have 3 dots
		if(IPAddress.CountChars('.')==3)
		{
		   // An IpV4 address can replace dots and if all numbers then it is IPv4
		   string ip2 = IPAddress.Replace(".", "");
		   if(Regex.IsMatch(ip2, @"^\d+$"))
		   {
				this.HelperLib.LogMsg("IP Address is IPv4: " + IPAddress + ";","MEDIUM"); // log to log file
		   }
		}
		// otherwise if it contains : semi colons (and nos and certain HEX letters)
		else if (IPAddress.Contains(':'))
		{
		   // could be an IpV6 address check only numbers and letters when colons are removed                    
		   if (Regex.IsMatch(IPAddress, @"^[:0-9A-F]+$"))
		   {
				this.HelperLib.LogMsg("IP Address is IPv6: " + IPAddress + ";","MEDIUM"); // log to log file
		   }
		}
    }
    else
    {
		// no response, flight mode button enabled?
		this.HelperLib.LogMsg("We could not access the Internet URL: " + remotePageURL + "; Maybe Flight Mode is on or Adapter is broken and needs a refresh", "LOW");

		IPAddress = "No IP Address returned; HTTP Errror: " + this.StatusCode.ToString() + "; " + this.StatusDesc + ";";

		// call my IsNetworkOn function to test we have a network
		if(this.IsNetworkOn())
		{
		   this.LastErrorMessage = "The Internet Network Is On but we cannot access the Internet";
		}
		else
		{
		   this.LastErrorMessage = "The Internet Network Is Off and we cannot access the Internet";
		}

		this.HelperLib.LogMsg(this.LastErrorMessage,"LOW"); // log error

		// throw the error will be caught in a lower method and stop the script
		throw new System.Exception(this.LastErrorMessage);
    }
}


If that fails then I run this method to see if the Network is available. You can see in the code above that I call it if there is no 200-300 HTTP status response or IP address returned. 

I test it by running the console script with and without the flight mode button on my laptop.

public bool IsNetworkOn()
{
bool IsNetworkOn = System.Net.NetworkInformation;

NetworkInterface.GetIsNetworkAvailable();

return IsNetworkOn;
}
I also have a series of Proxy addresses I can use to get round blocks on IP addresses although I do try and use Karmic Scraping e.g no hammering of the site, caching pages I need to prevent re-getting them, gaps between retries if there is a temporary error where I change the referer, user-agent and proxy (if required), before trying again after so many seconds.

Also before I turn on a global setting in my HelperLib class which is the main class all other objects refer to and pass from one to another, I test to see if the computer I am on is using a global proxy or not. I do an HTTP request to a page that returns ISP info, and I have a list of ISP's which I can check. 

If my ISP comes back with the right ISP for my home, I know I am not on a global proxy, also it tells me whether the IP address from earlier is IPv4 or IPv6, then I know I am not using a global proxy. If Sky or Virgin is returned I know I have my VPN enabled on my laptop and that I am using a global proxy.

If I am using a global proxy all the tests and checks for "getting a new random proxy:port" when doing an HTTP request are canceled as I don't need to go through a VPN and then a proxy. If the Global Proxy is turned off I might choose to get a random proxy before each HTTP call. 

If that setting is enabled in my Settings class. As you can't have config page in console scripts that hook into a DLL, my code in the DLL for my settings is just a class called Settings with a big Switch Statement where I get the setting I need or set it.

Once I have checked my Internet access I then do an HTTP call to the Betfair API Operational page which tells me if the API is working or not. If it isn't then I cannot log in into it. If it is working I do a Login test to ensure I can login this involves:
  • Using 1 of 2 JSON methods to login to the 2 endpoints that give me API access with a hashed API code, and my username & password.
  • If I can log in, I update the API table in the DB with the time I logged in and that I am logged in. There may be times I don't need to login to the API such as when I am just scraping all the Races and Runner info.
  • I then do a Session re-get, I hold my session in a file so I don't have to re-get a new one each time but on a run I like to get a new Session and extract the session code from the JSON return to pass back in JSON calls to get data etc.
  • I then do a database test using my default connection string I just SELECT TOP(1) FROM SYS.OBJECTS and ensure I get a response.

Once I know the API is operational, the DB connectivity is working and that I can scrape pages outside the Betfair API I can set all the properties such as whether to use a random proxy/referer/user-agent if one of my calls errors (with a 5 second wait), if I am using a global proxy I don't bother with the proxies.

Believe it or not that is a lot of code across 5 classes just to ensure everything is okay on running the job from the console or Windows Service. 

I throw my own errors if I cannot HTTP scrape, not just log them, so I can see what needs fixing quickly, and I create a little report with the Time, BOT version, Whether it is live, testing, placing real bets, running from a service or console and other important info I might need to read quickly.

So this is just a quick article on how I go about ensuring a project using a big DLL (with all the HTTP, HelperLib, API etc) code is working when I run it.

Who knows you might want to do similar checks on your large automated systems. 

Who knows. I am just making bank from my AutoBOT which TRADES on Betfair for me!

© 2022 Strictly-Software

Monday, 26 November 2018

Obtaining an external IP address in memory for use in a Firewall Rule

Obtaining an external IP address in memory for use in a Firewall Rule

By Strictly-Software

As I am currently using an ISP which constantly changes my external IP address due to excessive use of DCHP, I have to regularly update external firewalls on servers to allow my computer remote access.

This is obviously a right pain to do and I have no idea why my ISP changes my IP address so much when my old ISP used DCHP and kept it for months at a time.

Therefore I created this little noddy VBScript to sit on my desktop to obtain my IP address and hold it in my clipboard memory ready for me to just open up my firewall and paste it in.

The code uses the MSXML2.ServerXmlHttp object to obtain the IP address from an external website which just prints it on the page and I store the Response.Text into a variable.

I then use WScript.Shell object to open a new Notepad window and write the IP address out into it.

Then I use SendKeys to select the IP address and copy it into the clipboards memory before shutting down Notepad.

This means I can just quickly double click my desktop shortcut icon to obtain the IP address ready to paste it straight into a Firewall rule.

Check this script out and when you are ready and can see that hitting CTRL + V pastes the IP address out elsewhere you should remove the "TEST SECTION" and uncomment the part above it that closes down Notepad. This ensures you are not leaving empty objects around in your computers memory.

It's not exactly amazing code but it's very helpful at this time and saves a lot of time visiting a website manually to get my external IP address.

You might find this useful yourselves!


Option Explicit

Dim IPAddress, objShell
Dim objHTTP : Set objHTTP = WScript.CreateObject("MSXML2.ServerXmlHttp")

'* Obtain external IP address and store it in a variable
objHTTP.Open "GET", "http://icanhazip.com", False
objHTTP.Send
IPAddress = objHTTP.ResponseText

'* Open Notepad
Set objShell = WScript.CreateObject("WScript.Shell")
objShell.Run "notepad.exe", 9

WScript.Sleep 1000

'* Write out the IP address to a blank notepad file
objShell.SendKeys IPAddress

'* select the IPAddress and copy it into memory
objShell.SendKeys "^{a}"
objShell.SendKeys "^{C}"

'* uncomment the following section and remove the "TEST SECTION" when you are ready

'* Close notepad with the IP address in clipboard ready to be pasted
'* Open Save Dialog
'* objShell.SendKeys("%{F4}")
'* Naviagate to Don't Save
'* objShell.SendKeys("{TAB}")
'* Exit Notepad
'* objShell.SendKeys("{ENTER}")

'* TEST SECTION - Proves that the IP address can be pasted elsewhere
WScript.Sleep 1000

'* Proof that the IP address is in the clipboard and can be pasted out
objShell.SendKeys "Test Paste Works"
objShell.SendKeys "{ENTER}"
objShell.SendKeys "^{V}"
objShell.SendKeys "{ENTER}"
objShell.SendKeys "Try a manual CTRL and V to check the IP address is still in memory"
objShell.SendKeys "{ENTER}"

'* Kill objects - DO NOT REMOVE!!
Set objShell = Nothing
Set objHTTP  = Nothing

By Strictly-Software

© 2018 Strictly-Software

Friday, 27 March 2015

Changing your DNS Servers - Problems Accessing The Internet Due To DNS Issues Then Read This!

Changing your DNS Servers

Are You Having Problems Accessing The Internet Due To DNS Issues Then Read This!

By Strictly-Software

Sometimes I get problems at home when my laptop cannot connect to the Internet due to DNS issues.

This happens quite regularly and my ISP is Virgin Media.

If you really don't know what you ISP is then go to this site whatismyipaddress.com.

Also you may get issues when your ISP blocks certain sites that you want to visit e.g. The Pirate Bay or other torrent sites or information clearing houses or anti establishment websites.

Therefore if you are getting DNS issues then you should have a look at some of the various solutions below to see if any are suitable for your needs.

1. Default Set-Up

Your system will usually be set up by your ISP to use DHCP (Dynamic Host Configuration Protocol) which from the article about DHCP says that your computer assigns a different IP address automatically every time you access the Internet.

However my settings have been on DHCP for years and I haven't noticed my IP address change except for a couple of times which can be very annoying. Especially if you have firewall exceptions and programs on other servers like Fail2Ban or DenyHosts to prevent you being banned (SSH, TCP/IP),

The reason your IP address might change is down to DHCP which is used to issue unique IP addresses to computers accessing the ISP as well as automatically configuring other network information on your computer.

As we haven't moved to IPv6 yet and we don't have enough IPv4 addresses for every single device accessing the internet an ISP will share IP addresses as well as using proxy servers or gateways to allow your device, whether it be a phone, tablet, TV, laptop, PC or car, to access the internet.

However be careful before changing your DHCP settings as they could cause problems especially if you are on a shared network.

In most homes and small businesses, the router acts as the DHCP server.

However on large networks, a single computer might act as the DHCP server and changing your DNS settings could cause problems especially if you use internal networks to access websites.

For example at my company I have test sites on internal IP addresses and changing the DNS addresses prevents me from accessing those sites.

You can read more about DHCP here.

However whilst the article says that in most cases a new IP address is assigned to your device every time you connect to the network I find my computers IP address stays the same for quite a while and only changes every four or so months.

If you don't know your IP address then you can easily check your IP address from your own PC without having to open a webpage. Or you can easily use the Internet and run a search for "What is my IP".

If you would like a little script that you can run from the command line or at the click of a button to get a popup with your external IP address then you can read this article on obtaining your IP address from your computer without using a browser.

2. Checking your DNS address from the Command Prompt

You can check your current DNS settings from the command prompt with ipconfig /all which will show you all your network connections and DNS details.

I have shown you my own home computer settings and you can see the DNS settings I am using at the bottom are Google's.


C:\Users\rreid>ipconfig /all

Windows IP Configuration

   Host Name . . . . . . . . . . . . : stard0026w7
   Primary Dns Suffix . . . . . . . : metal.strictly-software.com
   Node Type . . . . . . . . . . . . : Hybrid
   IP Routing Enabled. . . . . . . . : No
   WINS Proxy Enabled. . . . . . . . : No
   DNS Suffix Search List. . . . . . : metal.strictly-software.com

Ethernet adapter Local Area Connection:

   Connection-specific DNS Suffix . : metal.strictly-software.com
   Description . . . . . . . . . . . : Broadcom NetLink (TM) Gigabit Ethernet
   Physical Address. . . . . . . . . : D4-BE-D9-95-40-DB
   DHCP Enabled. . . . . . . . . . . : Yes
   Autoconfiguration Enabled . . . . : Yes
   Link-local IPv6 Address . . . . . : fe80::fc53:2287:bb47:b8a9%11(Preferred)
   IPv4 Address. . . . . . . . . . . : 10.0.7.79(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Lease Obtained. . . . . . . . . . : 23 February 2015 17:06:21
   Lease Expires . . . . . . . . . . : 19 March 2015 15:49:01
   Default Gateway . . . . . . . . . : fe80::20c:29ff:fe77:3876%11
                                                   10.0.7.249
   DHCP Server . . . . . . . . . . . : 10.0.7.244
   DHCPv6 IAID . . . . . . . . . . . : 248823513
   DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-17-02-22-03-D4-BE-D9-95-40-DB

   DNS Servers . . . . . . . . . . . : 8.8.8.8
                                                  8.8.4.4
   Primary WINS Server . . . . . . . : 10.0.7.1
   NetBIOS over Tcpip. . . . . . . . : Enabled

3. Obtaining Different DNS Settings


So one of the solutions to this problem is to change your DNS server IP addresses to public DNS Servers like Googles free addresses 8.8.8.8 and 8.8.4.4 or you can use many others including opendns.com.

This provides security at the DNS level that protects your computer from malware and other threats including probes and hacks which could save you without installing any further software.

If you are a business you have to pay but for personal use you don't and can just use these addresses 208.67.222.222 or 208.67.220.220.

Read more about opendns here.

You can find a big list of other DNS server IP's to use here.

Some of them include:

ProviderPrimary DNS ServerSecondary DNS Server
Level3209.244.0.3209.244.0.4
DNS.WATCH84.200.69.8084.200.70.40
Comodo Secure DNS8.26.56.268.20.247.20
OpenDNS Home208.67.222.222208.67.220.220
DNS Advantage156.154.70.1156.154.71.1
Norton ConnectSafe199.85.126.10199.85.127.10
GreenTeamDNS81.218.119.11209.88.198.133
SafeDNS195.46.39.39195.46.39.40
OpenNIC107.150.40.23450.116.23.211
SmartViper208.76.50.50208.76.51.51
Dyn216.146.35.35216.146.36.36
FreeDNS37.235.1.17437.235.1.177


4. Changing DNS Settings


Every computer will be slightly different and you and you can search Google to find out how to do it on your computer, tablet, phone or even your smart TV.

This example is for Windows 7 but it not so different for other Windows machines:
  • Go to Control Panel and select Network and Sharing Center.
  • Select Change Adaptor Settings.
  • Select the network connection you want to change LAN or WIFI.
  • The panel will open showing you your connection details.that shows the network connection you want to change e.g LAN or WIFI.
  • Scroll down to Internet Protocol Version 4 (TCP/IPv4) (leave IPv6 for later!),
  • Click on that row then Properties,
  • In the bottom half are the DNS settings. Your ISP will most likely have ticked "Obtain DNS Server address automatically" so de-select that,
  • Select "Use the following DNS server addresses" and then enter first your preferred IP address e.g 8.8.8.8 and then in the alternate DNS server put 8.8.4.4
  • Hit OK and the settings should be saved. Test it by trying to get to a web page. If you can access a webpage by the
  • If you are not sure the DNS settings are being used then go to point 2 and check from the command prompt that your DNS settings have been changed.


A good idea might be to set your DNS up so that it uses Googles primary IP 8.8.8.8 for the preferred server and then the OpenDNS IP 208.67.222.222 for the Alliterate DNS Server.

E.G Mix and Match in-case one of the servers goes down..

5. Other Quick Options To Try

Other options you can try to attempt to sort your WIFI out without getting "techie" include just turning off WIFI fr a 30 seconds then back on.

Or you could turn your machines Airplane mode on and waiting a minute before turning it back off.

Turning your router off by unplugging it or taking the power cable out of the back of it. Wait a good 5 minutes then turn it back on.

Using the computers online diagnostic tool to test why your system is down. If it says DNS problems then this article might be for you.



6. Conclusion

Remember all modern devices like phones, tablets and even Smart TV's can connect to the internet and they all have options to change their DNS settings if you are having problems.

Also you must be careful if you are on a work computer as changing your DNS might not be a good idea as you could have internal routing going on to locally hosted sites on the companies internal network.

Therefore changing them might prevent you from viewing some websites (especially dev/demo sites).

You can always test by checking an internal site to see if you get a workable screen or just a "cannot connect" message.


© 2015 Strictly-Software

Wednesday, 14 August 2013

Help Fight Internet Censorship with the Pirate Bay's New PirateBrowser

Help Fight Internet Censorship with the Pirate Bay's New PirateBrowser

To celebrate it's 10th birthday the worlds most infamous censored site, The Pirate Bay, has introduced its own Internet browser to enable people to access its website even if it's being blocked by your ISP.

Most big ISP's have blocked The Pirate Bay for its users claiming it breaches copyright by allowing people to download Torrents of films and music.

If you don't know what a torrent it it's a movie split into thousands of small pieces. Each piece is stored on various computers so that each user is not in theory holding a full version of a film that may be breaching copyright. When you download  the torrent the files are all put together and downloaded from their various locations.

However most ISP's still see this as copyright violation and even though The Pirate Bay is just like Google in the fact they are just a search engine and don't actually host the films or music they have been attacked from all quarters.

Is Your ISP Blocking You?

You can quickly test whether you are being blocked by your ISP by clicking these links which all point to the Pirate Bay Website.

http://thepiratebay.org

http://thepiratebay.sx

https://piratereverse.info

Check More Pirate Bay Proxies

If you want a PHP script that will scan a number of known Pirate Bay Proxies then you can download this one I quickly knocked up from here: Pirate Bay Proxy Checker Script.

Just change the file extension to .php and either run it from your local computer or server. If you are running Windows I recommend downloading WAMP so that you can run an Apache server on your PC.

I also recommend changing the port number the server runs on to 8080 or 8888 so it can run side by side with IIS. This article will explain how to set WAMP up.


Use Pirate Bay Tor Browser

If you are being blocked from the Pirate Bay by your ISP then you have a number of choices.

1. Find a working mirror to the site - e.g search for Pirate Bay Proxies on DuckDuckgo.com or try this page: http://torrentproxies.com/ to find some not blocked by your ISP. Many mirrors will first open up in something called adf.ly with a JavaScript countdown. After a few seconds a link will appear saying "click here to continue". On clicking the page will change (usually to an advert with a blue bar at the top) and in the top right corner will be another countdown saying "Please Wait". Usually after 5 seconds it will say "SKIP AD" and on clicking it you will end up at the Pirate Bay mirror site.

2. Use a proxy to access the Pirate Bay. There are many free proxies out there and you can find a good list at http://nntime.com/proxy-list-01.htm. Use a tool like FoxyProxy to manage your proxy lists and toggle between them. Or you can go into your browsers network / proxy settings and manually enter the IP address and port number you want to use.

3. Download the new Pirate Bay Browser or the Mozilla TOR Browser. Both of which access the TOR network to help disguise your internet footprint by bouncing your HTTP requests through a series of servers.  As the TOR website explains.


"Tor protects you by bouncing your communications around a distributed network of relays run by volunteers all around the world: it prevents somebody watching your Internet connection from learning what sites you visit, and it prevents the sites you visit from learning your physical location. Tor works with many of your existing applications, including web browsers, instant messaging clients, remote login, and other applications based on the TCP protocol."


With all the outrage surrounding the NSA spying on ALL American citizens and the recent revelations that they are using this information not only to catch terrorists but to catch petty criminals, drug dealers, tax avoiders and to spy on and blackmail politicians and judges it is in everyone's best interests that they tighten up on their Internet Security.

As this article I wrote explains there are a number of measures you can take to reduce your Internet footprint and whilst you might be totally invisible you can blend into the background.

So do yourself a favour and use the Pirate Bay browser or the TOR Mozilla Browser. Not only does making use of the TOR network help protect your privacy but the Pirate Bay Browser will let you access a number of Pirate Bay Mirror sites and bypass any censorship that your ISP or country may have introduced.

Download the TOR Browser


Download the TOR Browser


Once installed when you open the .EXE you will connect to the TOR network.

Joining the TOR network

Once connected you can use the Firefox browser to surf the web. All your Pirate Bay Mirror sites and other Torrent sites are linked at the top in the icon bar. Pick one and then carry out your search for movies, applications or other content.

As you can see it's just a search engine like Google which makes it very unfair that the Pirate Bay is being blocked by ISP's as Google also indexes illegal content such as porn and copyrighted material.

This is a search for the US TV show Dexter.

The Pirate Bay Browser

Like most search engines once you run a search you get your results.

The Pirate Bay Search Results

As you can see from the results on the right there are two columns, seeders and leachers.

You want to choose an item with as many seeders (uploaders) and as few leachers (downloaders) as possible to get a quick download.

You can also tell from the names of the file what sort they are e.g if the file has the word CAM in it then it's a poor quality camera in a cinema job. If it's HDTV quality it will be a bigger better quality file and BRRip is a BluRay copy.

Clicking on the file you want will open up the result page. Hopefully you will get comments telling you the quality of the file. Click on "Get This Torrent" to download your torrent file.

Downloading a Torrent from the Pirate Bay

When you want to download a torrent from the Pirate Bay you will first need to have a torrent client. There are many out there including:
www.utorrent.com/downloads/win

bitlord.soft32.com/

www.bittorrent.com/
Once you have one installed your application of choice you can just click on the torrent in your search results and it will open up in the client and start downloading.

As you can see whilst you download the file you will also be uploading at the same time. This way you are not just taking without giving back the files you have already leached. You can control the bandwidth ratio between upload and download as well as set limits on the rate you upload.

Downloading a Torrent in UTorrent

However bewarned, ISP's are also on the lookout for people downloading (leaching) and uploading (seeding) torrents so you need to protect yourself.

Some of the ways include:
-Using someone else's WIFI or network.

-Using an anonymous proxy that doesn't leak identifying information or even better a VPN. A paid for tool like BTGuard which is both a proxy and encryption tool which helps prevent your ISP throttle your traffic.

-Using a block list which will mean that all traffic is bounced around ISP routers so that they cannot scan the traffic.

-Setting your download port to a common port for other traffic such as 80 or 8080  (HTTP) so that it doesn't look obvious you are downloading P2P data. Most people use a "random port" but it is better to look like you are downloading HTTP content especially if you are encrypting your traffic as well.

-Limiting your upload rate to a minimum. Although this violates the spirit of P2P (sharing) the people going after you for stealing copyrighted material are more interested in those spreading (uploading) the content rather than downloading it.

-Ensuring you force outgoing traffic to be encrypted. This will help prevent your ISP see that you are using BitTorrent traffic and may prevent them throttling your bandwidth.

-Set a download cap on your traffic. Even if you are encrypting your traffic some ISP's may see the amount you are downloading and throttle it if they think you are up to no good.

-Using an application like PeerBlock which will block traffic from known bad IP addresses such as P2P blocklists, known bad IP addresses, spyware, FBI and copyright monitoring sites and so on. It is worth downloading and just sits in the background running as you do you're downloading.
As with most security measures a wide range of different measures is much better than just relying on one system.

Hopefully this article will help you make use of The Pirate Bay if you have been blocked so far from accessing it.

Wednesday, 6 March 2013

Internet Censorship and Privacy - How they track you

Internet Censorship and Privacy - How they track you

This was taken from the www.darkpolitricks.com site.

Internet Censorship and Surfing Anonymously

By Dark Politricks

Sometimes it feels like the good old days of the Internet and being anonymous have passed - and you would be right.

With restrictive and snooping laws being passed all over the world, firewall filters wrapped around whole countries, Twitter users sued for Re-Tweeting libellous claims and big tech companies working hand in hand with the biggest security forces on the planet there really is no way to escape.

However there are ways to minimise your "footprint" and if you are not a serious criminal or terrorist then you shouldn't have anything to fear.

However if you are then you're probably being watched through your webcam right now whilst your iPhone's microphone is being channelled into GCHQ or Langley for analysing - tough luck!

From a users perspective the Internet contains a myriad of security and privacy issues which if the user is not aware of could cause potential problems on all manner of levels.

For the privacy conscious person who wants to be able to surf the net without worrying about someone looking at the content they have visited in real time or at a future date e.g your work, government or Police then there are a number of issues they need to be concerned about.

With the recent dismissal of the head of America's most powerful spy, David Petraeus, and knowledge about the way he was caught it is good to know the way's you are tracked so you can choose whether you want to take that risk.

As with most web content if you wanted to be 100% anonymous on the web it will be pretty hard to do.

If you want to stay totally anonymous you should probably move underground somewhere as there is always a satellite up there somewhere and with Google Earth you cannot even escape commercial companies anymore. So moving to the woods to live in a hut without electricity or broadband is not even an option anymore!

However there are various forms of tracking that you should be aware of so that you can limit the risks to you whilst surfing or using the Internet or phone. These don't have to be to hide from the Government but could just be to prevent your personal details from being sold to advertisers or having horrible popup boxes show when you close a window.

This is not a comprehensive list but it is a start and it is also one that is constantly changing as technology changes. When I wrote this originally tablets (iPad etc) were not that common. Now they are just another place to accumulate your browsing history and a tool to be used against you if you are ever in that unfortunate position.

Emails

This is is how the head of the CIA was caught out. He wasn't sending the emails but he was saving them as a draft under an anonymous Google account and then letting his mistress login and read them.

This way there was no Internet trail as when you send an email the mail is routed from server to server and the IP addresses of the mail servers it travelled through are recorded in the headers of your mail. 

Check it yourself. If you use a mail client find the option to "View All Mail headers" and then view an email that has been sent to you. At the top you should see the details of the route that the email took which should show the originating mail server, the receiving mail server and the IP addresses of any it traveled through in-between.

When people use this draft save only technique they are trying to avoid this trail. However you are defeating yourself in the first instance by:

a) Using an online mail server such as Gmail or Hotmail. All the data including drafts are saved on THEIR servers i.e "in the cloud" so if they disable your account, or are served with a warrant there is a good chance your draft emails will be accessed and read along with your sent, read and junk emails.

b) If you don't hide your origin when signing up for a throwaway Gmail or Hotmail account (you have to fill in a form to get one in the first place) then they will still get your IP address unless you have gone through known secure Proxy servers or used someone else's computer without their knowledge (e.g an open Wifi router). Do a scan on your PC / Phone now and see if there are any around you. Open ones won't have a lock symbol next to them.

c) Remember that Microsoft computers store all deleted emails, web history and other files even when you think they have been deleted on your computer. Here is a very old article from 2000 which shows even back then Microsoft was hiding emails, web searches and other files from users. The scripts and batch files you might find if you search for "Microsofts Really Hidden Files" probably won't work anymore but they probably have no need for such old methods anymore especially when people are buying computers and defaulting them to backup everything to the cloud.

How to bypass

Don't use "cloud based" systems that are well known to have links with the US security services (Microsoft and Google have - as I have shown and both these mega companies are actively helping the US spy agencies with their own huge database). Read this article on why we are sleep walking into a surveillance society by consent. Then ask yourself whether using any of these big Internet companies software is safe, especially as they seem to gobble up smaller companies by the minute. Why create back-doors when you can walk in the front I asked myself when Microsoft bought SKYPE.

It may pain you but by using any Internet based email service it means recording the data as it leaves the device you write your message on, storing it on a computer system you have no control over and by signing the Terms and Conditions you have allowed them to "own" your data and use it for advertising and God knows what else.

Plus nothing leaves the Internet, you can view cached versions of Google (or any site) all the way back to the 90's on this site: http://archive.org/web/web.php

If you don't want your own website to appear on this search engine that archives everything forever then they are pretty good about obeying the robots.txt directives so you can put this in your robots.txt file (read about it here) to prevent that site indexing your site.

# alexa archiver
User-agent: ia_archiver
Disallow: /

To be really sure you can block the IP 207.241.224.41 in your .htaccess file or at your firewall if you wanted to stop them crawling your robots.txt file at all.

Also use throwaway email addresses if you can or even create your own with some basic scripting (not hard if you can be bothered) and put it on a server in another "more freedom friendly country" and use proper proxies to access the webpage front end to send your emails (the part about proxies come later on).

The need for these disposable email systems sprung out of the need for a quick email address to sign up to a site or set up an anonymous blog or anything else that you don't want all the spam emails that follows. I also have found with some basic hacking you actually use them to send AND receive email - it all depends on how good the programmer is.

Either do a search for "Disposable Email Addresses" to find the latest ones or check out guerrillamail.com or Mailinator  however a word of warning - there are lots of disposable email accounts out there who knows who really owns them? If you do use them make sure their URL starts with https:// (this means data is encrypted from your PC to their server).

As for Internet files you can use "cleaner" tools like CCleaner to remove cookies, old registry files, old programs, start up applications and Internet history easily. Plus you should never use Internet Explorer anyway, as there are a myriad of more security conscious browsers such as FireFox out there which are much better on the privacy front as they are not tied into the operating system of your computer.

Using the Cloud to backup your "Secure" data

Storing anything on anyone else's computer means you don't have control over it. Therefore cloud based storage systems should be avoided for anything personal or secure. Even phones nowadays have settings to automatically back up your numbers, texts, photos and videos.

It might be good if you ever lose your computer or phone but remember that if the cloud based backup server is in the USA then they are probably sniffing everything inputted into it anyway.

This is  the same for Facebook, Google, Tumblr and any other social media site. As soon as you put anything on that site THEY own it and if they are served a subpoena or warrant to hand the data over there is nothing you can do.

How to bypass

It may pain you but just don't use Facebook, Google, Twitter or any other social media system if you are going to put anything dodgy on it.

The same goes for dropbox and any other web based storage centre that may have to hand over your data to the authorities one day.

If you must keep files secret then keep a portable external hard drive at home and backup all your files to that device before hiding it. At least that way you have control and ownership over the backed up data and you are the only person who knows where it is located.

Using your computer hardware to spy on you

There have been many cases lately where computers and phones have been used against their owners to spy on them.

Last year there was a big outcry about iPhones "secret" database that logged all the GPS positions you had been to with your phone. Even without GPS they can use phone mast triangulation to find a near enough point that your phone pinged the mast.

Also we had the case of cops using tools to download this data as they pulled motorists over or arrested people and then illegally accessed this database of locations to find out where you had been.

As for computers we had a school in Lower Merion school district in Philadelphia that was accused of spying on students in their bedrooms via school issued laptops and the webcams built into them. Would you want a headmaster in his office alone at night watching your kids in their bedroom?

How to bypass

Take the battery out of your phone whenever you don’t want to be tracked.

As the earlier report shows cellphone triangulation tracking takes less power than GPS tracking and even when your phone is turned off a tiny amount of battery charge is available to the phone which is enough to log your presence at a nearby tower and then log your presence down to the nearest 100 metres or so. 

Either that or use pay phones or pay passers by to use their phone when you need to make a call when your phone is unavailable.

On your computer turn off the microphone and webcam with your settings e.g on Windows it's in Control Panel. On Windows 7 it will be under Speech Recognition and Audio Devices.  To be extra safe wrap masking tape over the webcam when you don't want to use it as well as the speaker (blue-tack or something else that would muffle the sound). Anything that can be useful to you can be useful to someone with control of your computer.

To test if your microphone is working either go into your computers settings e.g control panel or go to the old Google Search Engine (if it's still available at http://www.google.com/webhp?hl=all ) hit the microphone symbol in the input box and talk.

If you see the blocks under the microphone move up and down and then a result similar to what you said appear in the box - the microphone is still on. If it's off it will say so.

Javascript urchins

These are little bits of script that are added to the source code of the HTML page you are visiting. They use JavaScript to record identifying features about the user and their browser such as the user-agent, system details and location by calling a script on another server that then logs these details to a central database. A good example is Google Analytics which most sites including this one use to tell the owner about the amount of traffic they receive and where it is coming from.

How to bypass

Turning off Javascript will prevent this logging from occurring. You can do this in most browsers through their Tools > Options settings but you can get toolbars and add-ons like the FireFox Web Developer Toolbar or the NoScript add-on that do this for you.

Webbugs

Similar to urchins these are little images, usually so small they cannot be seen, that point back to a web server and run some code whenever the image is loaded by a client. They tend to be used by email marketing tools and are embedded within HTML emails so that they can record who has actually opened the email and track the email if its forwarded it on. 

They can also exist on web pages or within desktop applications and as the image is hosted remotely whenever it is loaded it records the location of the application or user who is loading it.

How to bypass

Many email clients if they don't do it automatically have the option to display emails as plain text which would prevent these webbugs from working. I use Thunderbird which is free and you can set to ask you first whether to load any remote content at all whether they are images, scripts or anything not already embedded within the email.

 In Browsers you can disable images easily with the Web Developer toolbar, Google Chromes privacy settings or by using a text browser like Lynx.

Server Side logging by the page

Most pages on the web nowadays are more than pure HTML/CSS and contain code that runs server side e.g .asp, .php, .jsp, .aspx etc.

When the page is requested the web server parses the page and runs any code before returning the generated HTML to the client. This code has access to a lot of information about the client requesting the page such as IP address which can be used for GEO tagging, User-agent details, accepted file types and other information contained within the headers. They could choose to log this information to a database or file if they wanted to even if the IIS or Apache web server had its own logging disabled.

For example if you got to whatsmyip.org you will see all the information that is passed to each webpage you request including geo-location information, details about the type of computer you are using and much more. Whilst not totally accurate they can pinpoint the last location of the computer used to access a webpage which could be your own PC or could be someone else's (if you use a proxy - see below).

How to bypass

Please read the guide under the following section about web server logging as it applies to both.

Logging by the Web Server

Every time you make an HTTP request e.g access a web page, a record is made on the web server that hosts that page to a log file. Each separate file contained within that web page is logged so every image, CSS file and script is logged along with your IP address, the method e.g POST or GET, the URL, bytes sent and received and much much more.

Although its possible to turn off this logging most companies running web servers require these logs for traffic analysis e.g with a tool such as Webtrends as it helps analyse traffic from all agents including robots who do not have JavaScript support. Also many countries now require ISP's to keep log files for up to a year or more in case the data is required at a later date.

How to bypass

As you must assume that the web servers you are visiting sites on have logging enabled then the only way to not get tracked is to go through proxy servers or use tools like the FireFox add-ons Modify Headers or Tamper Data which allow you to change the headers sent from your PC to the webserver in question and act as a mini proxy on your own PC. They cannot however change the REMOTE_ADDR header which holds the IP address of the PC making the request.

Another way is to turn your PC into a webserver through free software like WAMP Server and then create a web based proxy for your surfing. The good thing about this is that in the remote servers log files all they will see for an IP address is 127.0.0.1 which is the local loopback IP address and cannot be used to track anyone as every PC uses that address.

Remember a proxy is just an intermediate server that sits between you and the web server you want to access. If someone was tracking you they would only see your request to the proxy server and not the actual content that the proxy server requests on your behalf.

There are various forms of proxy some that are anonymous and others that pass your IP address along in the HTTP_FORWARDED_FOR, HTTP_X_FORWARDED_FOR or any number of other headers. You can also use code or tools to fill these headers with random IP addresses to make it harder for a tracker to find you as it will look like you have bounced round a lot of proxies when in fact you haven't.

There is also a form of proxy known as an "anonymizer", which is called this because it hides all the users identifying information such as headers that hold the IP and user-agent. There are lots online for you to use.

Anonymizers are not entirely secure. If an anonymizer keeps logs of incoming and outgoing connections and the anonymizer is physically located in a country where it is subjected to warrant searches then there is a potential risk that government officials can reverse engineer and identify all users who used the anonymizer and how they used it.

Most anonymizers state they do not keep logs but there is currently no way to confirm that. However, if the user used another anonymizer to connect to the exposed anonymizer, that user is still anonymous. This is sometimes called daisy-chaining. 

The safest way therefore is to use a chain of proxy servers to make your requests or use a specialist service like TOR which is designed to make it hard to track Internet usage.

P2P Torrents

People use torrents to download films, music and other software. Sometimes these are illegally obtained copies or pirated software.

The Pirate Bay was one of the most famous sites that people used to obtain torrents and the people behind it are currently involved in legal action as the US movie industry is trying to sue them for facilitating the illegal download of copyrighted material.

Even though they are just a search engine on the same lines as Google or BING (and you can find torrents on those search engines as well!) - it is pretty unfair as the Pirate Bay are not uploading the films themselves they are just a search engine that lists files of a certain type.

When you download torrents you use special software such as uTorrent or Deluge to download all the tiny pieces of the file you want. The idea is that because you are not downloading a whole file from one location but rather tiny bits of it from lots of locations you are not really breaking the law.

When you download you are a "leecher" and when you upload you are a "seeder". The software simplifies all this when you download a file as it connects all the tiny bits up for you so you don't have to worry about where they are coming from. 

Also as you download you are also uploading the bits you have already downloaded so other people can obtain them. You can change your settings to prevent the uploading part of this if you want to by changing the ratio of upload versus download or the rate/speed that you upload (or even turn it off).

The Priate Bay was the biggest site on-line which is why it is being targeted and if you try accessing www.thepiratebay.org in your browser now I bet you it will be blocked by your ISP.

How to bypass

There are many proxies for the Pirate Bay which will allow you to access the site from a different URL. Just search for "Pirate bay proxies" and then pick one. 

You might find an advert at the top of the page counting down - this is a way to access the site once the count is down to 0. Ignore the main part of the page and click on the "view" button that might appear after the countdown in the top right corner which should take you to the pirate bay proxy. 

You might have to try a few out first but I use https://piratereverse.info. As soon as the ISP shuts one down another one will pop up (just like the thousands of people who pointed domains at WikiLeaks when it was blocked) so you will always be able to find a site to get them from whether it's the Pirate Bay or a User Group or discussion board.

Also beware that many torrent tools will be flagged as Trojan down-loaders (even when they are not) and also that ISP's and other government organisations insert their own trackers that log the IP addresses of people downloading the torrents so that they can contact/blacklist/reduce your bandwidth etc. Therefore be careful and pick a good one and read up about trackers before engaging in torrent downloading.

To make the chance of being caught a lot less you can should change your torrent tool settings to go through a proxy server - preferably HTTPS (encrypted) or use any option that forces encryption when transferring files. 

You should also change the port used by the tool in your settings from a random number to 80 or 8080 as these are common webserver ports and make it hard for ISP's to tell what kind of traffic is being transferred. If possible use a "block list" that will mean that all the data packets sent to or from you will bypass known ISP routers where they can be sniffed and identified. More and more ISP's are doing this so this is wise to prevent yourself from being caught.

Read these articles to help you install a torrent down-loader and set-up measures to prevent yourself being blocked.



Cookies

Cookies are small text files that are stored on the clients computer and contain very small pieces of text. They are mainly used by websites to store flags that enable the site to know whether you have previously been to their site or not. Advertisers also use them to track the type of sites you visit so that they can deliver targeted advertising the biggest offender being Google which uses their domination of the market to track the sites users visit so they can target content specific adverts to the user.

Another type of cookie is a session variable which is used by many sites to store a unique ID that refers to a visit on the site. The ID is generated by the web server and the session cookie only stores this ID so that on each request to the server the system knows that the visitors requests belong to one visit.

How to bypass

If you are concerned about tracker cookies then you easily disable site related cookies in your browser but if you disable all cookies then Session variables won't work and you will most likely find yourself getting logged out of member only areas of websites or not being able to login in the first place.

The best option is to disable 3rd party cookies (those set by advertisers) and to delete non essential cookies after using the Internet (Incognito mode in Chrome).

Flash, ActiveX, Java Applets

3rd party components such as Flash, ActiveX controls and Java applets come with their own security concerns. There have been numerous security vulnerabilities reported with these types of component as due to their complexity and power they have more access to the clients computer than a normal web page. They should be seen as mini applications rather than just a fancy banner, game or helpful utility to enable you to upload files to Facebook more quickly.

You shouldn't install these types of application unless you are totally sure they are safe as they could have a lot more control over your computer than you realise. There have even been hacks that have enabled remote users to video and record a user through their webcam without them knowing.

How to bypass

You can use Firefox extensions such as FlashBlock or AdBlocker to disable flash on specific pages or the Opera browsers Turbo mode which speeds up page loads as well as allowing you to choose which flash movies to play. If you decided to choose privacy over anything else then you will end up having a pretty boring web experience as more and more sites use Javascript and Flash to deliver interactive content.

However if you are really security conscious you should use a text browser such as Lynx which won't load images, flash, JavaScript or any other form of plug-in. It will show you the textual content of the pages you visit and will ask if you want cookies to be stored for each request. Due to only loading text and links you will have fast load times so there is a benefit to having a reduced web interface.

You should also regularly check your PC for viruses and spyware. One of the first things modern Trojans do nowadays is download good anti-virus software so that they don't get overwritten by another spyware app!

They also try to disguise themselves as virus checkers to avoid detection. Even the best off the shelf virus checkers don't catch all forms of spyware especially those that have to regularly download virus definition patterns as it means new viruses don't get caught until they have been identified, a pattern created and downloaded by the client.

Virus payloads can also be modified randomly to avoid pattern detection so tools that don't use pattern matching such as hijackthis.exe which runs an analysis of all currently running processes looking for odd behaviour are good tools to use. This tool will generate a report which can then be analysed by members of the special Hijackthis.exe message board for signs of infection.

One of the best removers of Trojans I have found is a tool called SDFix.exe which managed to detect and remove a Trojan that four other tools including an off the shelf app didn't detect. There are also a number of good free products such as MalwareBytests Anti-Malware and AdAware anti adware and spyware software which can be run regularly to check your PC for spyware and viruses.

However keyloggers that are based around hardware such as cable extensions that you don't notice that have been inserted by your employer are undetectable unless you know what you are looking for and will store every key pressed on your PC whilst enabled. Check your cables that come out your computer to see if anything strange is connecting two parts of a wire together.

If you are caught out by such a tool make sure your employer has followed the law by informing you of any anti-privacy measures he or she may have introduced such as monitoring your PC and web usage in your contract. If they haven't then you have a good legal case to sue  them and they are breaking the law by spying on you without your knowledge. The same goes for CCTV, recording devices and other means of logging your activity without your knowledge.

Article 8 of the Human Rights Act that is used in the UK has been successfully used in previous cases by employees who have been sacked due to unknown spying by their employers and should be used by anyone taking their employer to court if they have been sacked due to such technological spying.

Tools to use to aid privacy on the web Firefox Add-Ons
  • Web Developer toolbar. Disable Javascript, cookies, view cookie and header info, modify the DOM, view generated source code, show password fields.
  • Flashblock disables flash movies until you enable them. Allows creation of a white-list of allowed sites.
  • FoxyProxy manage your proxies with an easy to use tool.
  • Tamperdata or Modify Headers acts like a proxy and allows you to modify HTTP requests as they are made from your client.
  • HTTP Fox, Firebug and even the Chrome developer toolbar allows you to see all the data your PC send to websites and the data sent back by the webserver you are accessing. It also shows any redirects or code loaded in that you might not be aware of.
Google Chrome
  • Use Incognito browsing to prevent browser and search history and cookies from being stored.
  • Firefox and IE9 also have privacy modes that can be used to remove cookies and reduce your internet footprint but I would not trust anything Microsoft as it's hooked into the computers main system and parts of the browser are shared with other non Internet based software.
All browsers
  • De-activate Javascript, VBScript (IE only) until you know the site is safe.
  • De-activate 3rd party cookies used by trackers, advertisers and sites wanting to keep track of you as move around the web such as Google Analytics.
  • If you share a PC Clear your cache, autocomplete, download list and history regularly - use CCleaner, AdAware etc.
If you need more details about the various forms of Internet Censorship and how to bypass it then check out the following article that contains a lot of details about the various methods used and how to bypass them.

How to bypass Internet Censorship If you are looking for an up to date list of available proxy servers then you can check out the following links:


The following page has an index where you can find more proxy lists 


If you want to quickly access some web based proxies you can pick from the following list or you can read my guide on creating your own web proxy which comes with an example and some code you can use to get running quickly.

Read the original article at www.darkpolitricks.com.