A New Test For The Brave Browser and New Security Focused Browsers

How Can We Test For Brave?

By Strictly-Software

I use the Brave browser as well as a few other Chromium-based security-focused browsers such as (Opera, CCleaner, and DuckDuckGo) most of the time due to their inbuilt security measures. My preferred browser is Brave due not only to its Shield which removes trackers, cookies, adverts, and has a measure of built-in fingerprint spoofing. Plus it has its own search engine, which can be accessed from the address bar, and it doesn't block certain sites like Google and other search engines do.

It also blocks Google's Accelerated Mobile Pages and takes you to the original publisher's site, can have strict upgrades to HTTPS URLs when linked to an unsecured domain, as well as its incognito pages are based on the TOR engine.

I never use Chrome anymore as it used to be a quick plain browser but has now got bogged down with too many options. Also, I don't like Google which relies on heavy use of adverts and selling user information for revenue, plus its other links to intelligence agencies, and its censorship. This has pushed mainstream media articles ahead of legitimately more relevant sites. I also don't like it's use of banned reading lists that try to prevent people from viewing sites the US Intel Establishment has deemed unsuitable like RT.com or Infowars.com. 

Plus I don't know what they now do server side to try and identify users due to all these new privacy-based browsers, and the number of privacy plugins/extensions, that could help protect you. However, it's good to know you can have a measure of protection without bogging the browser down with plugins such as fingerprint spoofers like Trace and AdBlockers like MalwareBytes or the DuckDuckGo plugin. Also, I like being able to earn cryptocurrency from Brave which rewards the user, not the site for viewing small adverts.

However, the issue with Brave is that it hides as Chrome and doesn't have its own user agent. It used to, and the hope is that in the future it will again but at the moment it just shows a Chrome user-agent. 

For example, my latest Brave user-agent is:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36

There have also been a lot of changes to how Brave identified itself in the Window and Navigator sub-object that used to make Brave identifiable through inspection of certain objects. These seem to have changed back and forth many times and you can see some of the old objects that used to be checked in my older article on looking for Brave here.

Why bother looking for Brave, or indeed any browser when feature detection should be used rather than user-agent sniffing or other means to find the name of the Browser?

Well, Brave prides itself on security, hiding adverts, removing tracking code and cookies, and now has a certain level of fingerprint sniffing protection to stop sites from using properties identified by JavaScript or Server Side code to change Response headers to make identifying unique users and traffic to sites much harder.

Therefore you may want to write Chromium extensions that add extra protection to Brave browsers or do the reverse, add code back in to replace removed adverts, or help track link clicks with removed Ping attributes or other devious plans your boss wants you to run on sites accessed by people using Brave. 

I don't know, I just like to play with the code and see what object detection features each browser reveals. They may all be based on Chromium but they can all have unique features.

An updated function from my previous article makes use of the fact that Brave now identifies itself in the window.navigator object, specifically the brave object in navigator and the isBrave property within that. 

First I rule out any Mozilla browsers e.g Firefox by looking for a well-known Mozilla property, and ensure the browser is Chromium based by checking for a Webkit property. It is similar to the two line old function but uses a new Webkit property as the old one has been removed and is wrapped in a function.

function IsBrave(){
	
	let w=window,n=w.navigator; // shorten key objects

	// as many tests we know browsers now support; prove its not Mozilla; prove its Chromium based and has Brave properties in window and navigator objects
	let isBrave = !("mozInnerScreenX" in w) && ("chrome" in w && "onwebkitanimationiteration" in w && "brave" in n && "isBrave" in n.brave) ? true : false;
	
	return isBrave;
}

And if you want to check for the new CCleaner browser there isn't any specific objects in the navigator object I can see but it does have a unique user-agent e.g:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/121.0.0.0 Safari/537.36 CCleaner/121.0.0.0

However, the Duckduckgo browser, which comes with an email protection service, allowing you to use a specific @duck.com email address that only forwards non-spam emails to your real address, does have a specific object if you want to check for it.

// detect duckduckgo
let duckduckgo = ("duckduckgo" in navigator) ? true : false;

However, we all should be detecting objects to allow for features rather than the old method of user-agent sniffing, but if you did want to identify certain browsers there are unique objects, usually on the navigator object that can be checked.

You should know that there are lots of ways you can protect yourself from tracking and unique identification nowadays but it does rely on the device you are using to browse as well as the model of that device.

For instance, even if you are now using IPv6 addresses on your router which makes unique identification a lot easier than the older DCHP method of picking a free IPv4 address as local to you as possible when accessing the web, your system can be set to use temporary IPv6 addresses.

This means that your real unique identifying IP address is never shown to outside sites. A look at ipconfig /all in your command prompt will show you whether you are using them or not and a search online will show you how to change the settings if you're not using them yet.

Also, if you are using Android or iOS phones or devices that use Google as an integral part of their system, you can delete the unique advertising ID that exists to allow sites to uniquely track you whether or not your browser or plugin removes Google tracking codes. 

On iOS the ad identifier is also called "IDFA" and on Android, "AAID". As these IDs can only be accessed using server-side code, Java or Python, and a Google library, modifying the DOM won't stop you from being tracked. 

You can easily remove these IDs in the Privacy / Advertising section of your phone's settings if you have Android 12 or above. iOS devices are more complicated but this article explains how to remove them.

If you want to check what properties exist for a browser object then these two lines of code can help you out rather than a loop.

const keys = Object.keys(window);
console.log(keys);

This just outputs all the keys such as events, other objects like document and navigator, and properties as well into the developer tools console area that all modern browsers have.

Remember as Brave, CCleaner, DuckDuckgo, Edge, Opera and Chrome are all based on the same Chromium browser they are the same standard-compliant browsers. However, as I stated in my last article, it is amazing how many modern sites still break when I use a user-agent switcher and change my string to IE6 for example. 

They really shouldn't if they were using feature detection by checking for the existence of objects before running certain code.

By Strictly-Software

What do you do when you have multiple blogs under one Google Blogger Account?

Changing The Names Of Replies And Posts

By Strictly-Software

If you are a blogger user and you have multiple blogs each with different names then you might run into the problem that when you post, or reply to a message using your Google account it will put your main Blogger account name in as the author, or comment author.

Now the way I have got round this is to use JavaScript in the footer of the blog. Add a JavaScript / HTML gadget to the footer of your blog and then you can fix this issue.

I will show you two ways you can fix this using old JavaScript and newer JavaScript .

To fix the name of post authors you can use this code. Say your blogger author name is Dave Williams but you are using pseudo names for your blog, say "Strictly-Software" for example then you don't want Dave Williams posted at the end of each post, you want Strictly-Software to appear.

Wrap these in SCRIPT blocks, whether a short <script>....</script> block or a full old style <script type="text/javascript"><!-- ..... //--></script> tags. Up to you.

This looks for all the spans with a class of "fn" which appears under the post where it says "Posted by ..... at 9:58pm" etc. So we loop through all the fn classes and change the innerHTML to the value we want.

 
(function(){
var x = document.getElementsByClassName("fn");
var i;
for (i = 0; i < x.length; i++) {
    x[i].innerHTML = "Strictly-Software";
}
})()

Now what about comments? 

If you are replying to someone and you don't want to use the Name/URL option to put your blogs name in, then you can use the Google option so it knows you own the comment, but then you can use some code to identify the links which contain the Google Authors name and if found update them to the name you want.

Same as before with the SCRIPT tags, although I put both together one after the other in one script block at the bottom of my layout.

Now, this code uses a more modern document.querySelectorAll method to find all elements that match the selector we pass in which will be an anchor tag. This is because all comments have the name of the person commenting wrapped in a comment tag IF you use a Google Account to post it. 

Obviously, Anonymous and Name/URL comments can't as there is nowhere to link to.

(function(){
	var x = document.querySelectorAll("a");
	var i;
	for (i = 0; i < x.length; i++) {		
		if(x[i].innerHTML=="Dave Williams"){
			x[i].innerHTML="Strictly-Software"
		}		
	}
})();

As you can see I am looping through all the elements that the querySelectorAll("a"); returns. 

If we had used just querySelector it would have only found the first A tag on the page and we need to find and replace all comments on the page which have Dave Williams as the current contents and if found, we replace it with Strictly-Software by using the innerHTML of the element.

I just thought I would post this as I had to do this tonight on one of my other blogs as I noticed the comment author was wrong. You could use the 2nd method for getting the author name as well if you wanted but I thought I would show you two ways of doing this.

Hope this helps if you have the same issue.

By Strictly-Software

Testing For Internet Connectivity

Setup Measures For A Big API System

By Strictly-Software

I have a BOT that runs nightly, and in the SetUp() methods to ensure everything is okay, it runs a number of tests before logging to an API table in my database. This is so I know if the API I am using is available, whether I needed to login to it, when the last job was started and stopped, and when the last time I did log into the API. 

It just helps in the backend to see if there are any issues with the API without debugging it. Therefore the SetUp job has to be quite comprehensive.

The things I need to find out, which are logged into the logfile.log I keep building until the job is finished with major Method Input Params, Return Params, Handled Errors, Changed System Behaviour, SQL Statements that return an error e.g timeout, an unexpected divide by zero error and the like are the following from my SetUp job.

1. Whether I can access the Internet, this is done with a test to a page that just returns an IPv4 or IPv6 address. The major HTTP Status code I look for there is a 200 status code. If I get nowhere but get a status code of 1 that is a WebExceptionStatus.NameResolutionFailure, which means the DNS could not find the location and it's probably due to either your WIFI button being turned off (Flight Mode), or your Network Adapter having issues.

I test for web access with an obvious simple HTTP test to an IP page that returns my address, with 2 simple regular expressions that can ID if it's IPv4 or IPv6.

public string TestIPAddress(bool ipv4=true)
{
    string IPAddress = "";
    // if we are using an IPv6 address the normal page will return it if we want to force
    // get an IPV4 address then we go to the IPv4 page which is the default behaviour due to
    // VPNS and Proxies using IPV4 usually
    string remotePageURL = ((ipv4) ? "https://ipv4.icanhazip.com/" : "https://icanhazip.com/");
   
    this.UserAgent = this.BrainiacUserAgent; // use our BOT UserAgent as no need to spoof
    this.Referer = "LOCALHOST"; // use our localhost as we are running from our PC
    this.StoreResponse = true; // store the reponse from the page
    this.Timeout = (30000); // ms 30 seconds            

    // a simple BOT that just does a GET request I am sure you can find a C# BOT example on my blog or tons on the web
    this.MakeHTTPRequest(remotePageURL);            

    // if status code is between 200 & 300 we should have data
    if (this.StatusCode >= 200 && this.StatusCode < 300)
    {
		IPAddress = this.ResponseContent; // get the IP Address

		// quick test for IPAddress Match, IPv4 have 3 dots
		if(IPAddress.CountChars('.')==3)
		{
		   // An IpV4 address can replace dots and if all numbers then it is IPv4
		   string ip2 = IPAddress.Replace(".", "");
		   if(Regex.IsMatch(ip2, @"^\d+$"))
		   {
				this.HelperLib.LogMsg("IP Address is IPv4: " + IPAddress + ";","MEDIUM"); // log to log file
		   }
		}
		// otherwise if it contains : semi colons (and nos and certain HEX letters)
		else if (IPAddress.Contains(':'))
		{
		   // could be an IpV6 address check only numbers and letters when colons are removed                    
		   if (Regex.IsMatch(IPAddress, @"^[:0-9A-F]+$"))
		   {
				this.HelperLib.LogMsg("IP Address is IPv6: " + IPAddress + ";","MEDIUM"); // log to log file
		   }
		}
    }
    else
    {
		// no response, flight mode button enabled?
		this.HelperLib.LogMsg("We could not access the Internet URL: " + remotePageURL + "; Maybe Flight Mode is on or Adapter is broken and needs a refresh", "LOW");

		IPAddress = "No IP Address returned; HTTP Errror: " + this.StatusCode.ToString() + "; " + this.StatusDesc + ";";

		// call my IsNetworkOn function to test we have a network
		if(this.IsNetworkOn())
		{
		   this.LastErrorMessage = "The Internet Network Is On but we cannot access the Internet";
		}
		else
		{
		   this.LastErrorMessage = "The Internet Network Is Off and we cannot access the Internet";
		}

		this.HelperLib.LogMsg(this.LastErrorMessage,"LOW"); // log error

		// throw the error will be caught in a lower method and stop the script
		throw new System.Exception(this.LastErrorMessage);
    }
}

If that fails then I run this method to see if the Network is available. You can see in the code above that I call it if there is no 200-300 HTTP status response or IP address returned. 

I test it by running the console script with and without the flight mode button on my laptop.

public bool IsNetworkOn()
{
bool IsNetworkOn = System.Net.NetworkInformation;

NetworkInterface.GetIsNetworkAvailable();

return IsNetworkOn;
}

I also have a series of Proxy addresses I can use to get round blocks on IP addresses although I do try and use Karmic Scraping e.g no hammering of the site, caching pages I need to prevent re-getting them, gaps between retries if there is a temporary error where I change the referer, user-agent and proxy (if required), before trying again after so many seconds.

Also before I turn on a global setting in my HelperLib class which is the main class all other objects refer to and pass from one to another, I test to see if the computer I am on is using a global proxy or not. I do an HTTP request to a page that returns ISP info, and I have a list of ISP's which I can check. 

If my ISP comes back with the right ISP for my home, I know I am not on a global proxy, also it tells me whether the IP address from earlier is IPv4 or IPv6, then I know I am not using a global proxy. If Sky or Virgin is returned I know I have my VPN enabled on my laptop and that I am using a global proxy.

If I am using a global proxy all the tests and checks for "getting a new random proxy:port" when doing an HTTP request are canceled as I don't need to go through a VPN and then a proxy. If the Global Proxy is turned off I might choose to get a random proxy before each HTTP call. 

If that setting is enabled in my Settings class. As you can't have config page in console scripts that hook into a DLL, my code in the DLL for my settings is just a class called Settings with a big Switch Statement where I get the setting I need or set it.

Once I have checked my Internet access I then do an HTTP call to the Betfair API Operational page which tells me if the API is working or not. If it isn't then I cannot log in into it. If it is working I do a Login test to ensure I can login this involves:

• Using 1 of 2 JSON methods to login to the 2 endpoints that give me API access with a hashed API code, and my username & password.
• If I can log in, I update the API table in the DB with the time I logged in and that I am logged in. There may be times I don't need to login to the API such as when I am just scraping all the Races and Runner info.
• I then do a Session re-get, I hold my session in a file so I don't have to re-get a new one each time but on a run I like to get a new Session and extract the session code from the JSON return to pass back in JSON calls to get data etc.
• I then do a database test using my default connection string I just SELECT TOP(1) FROM SYS.OBJECTS and ensure I get a response.

Once I know the API is operational, the DB connectivity is working and that I can scrape pages outside the Betfair API I can set all the properties such as whether to use a random proxy/referer/user-agent if one of my calls errors (with a 5 second wait), if I am using a global proxy I don't bother with the proxies.

Believe it or not that is a lot of code across 5 classes just to ensure everything is okay on running the job from the console or Windows Service. 

I throw my own errors if I cannot HTTP scrape, not just log them, so I can see what needs fixing quickly, and I create a little report with the Time, BOT version, Whether it is live, testing, placing real bets, running from a service or console and other important info I might need to read quickly.

So this is just a quick article on how I go about ensuring a project using a big DLL (with all the HTTP, HelperLib, API etc) code is working when I run it.

Who knows you might want to do similar checks on your large automated systems. 

Who knows. I am just making bank from my AutoBOT which TRADES on Betfair for me!

© 2022 Strictly-Software

Extending Try/Catch To Handle Custom Exceptions - Part 2

Extending Core C# Object Try/Catch To Display Custom Regular Expression Exceptions - Part 2

By Strictly-Software

This is the second part of our custom Try/Catch overwrite program that is aimed to catch Regular Expression errors that would normally be thrown up by !IsMatch (doesn't match) and so on.

 

So why would you want to do this when the C# .NET RegEx functions already return a true or false or 0 matches etc which you could use for error detection?

 

Well, you can read the first article above this one for info however you may have a case where you have a BOT that is automatically running every day all day on certain sites where the HTML has been worked out and expression especially made to match them. 

If for example the HTML source changes and the piece of data you are collecting suddenly does not get returned anymore as your expression does not match anything instead of just logging to a file which means scanning and checking for errors once you notice your fully automated 24/7/365 system has stopped working 2 months later. 

Therefore instead of just logging the results of IsMatch you may want to throw up an error and stop the program so that a new expression can be made to match the HTML ASAP without lots of data being missed.

 

If you want more info on the Try/Catch class, read the first example article. Here we build just a basic BOT that is going to utilise the RegExException class we made last time. It is not a perfect BOT, you may get some warnings come up, I did it in VS 2021, and it's a specific BOT as its aims are to get the META Title, Description, and the full source of the URL that is passed into it. 

Any errors with the regular expressions not matching anymore are thrown up using our RegExException TryCatch cobject so that they detail the expression, text, or whichever of the 3 methods we made. If you want to use it as the basis of your own BOT you can but has been specifically designed to demonstrate this RegEx TryCatch example so you may want to edit or remove a lot of content first before building back on top of it if you want to make your own BOT in C#.

 

So here is the BOT

public class HTML
{
	public string URL  // property
	{ get; set; }

	public string HTMLSource  
	{ get; set; }

	public string HTMLTitle
	{ get; set; }

	public string HTMLDesc
	{ get; set; }

	public void GetHTML()
	{
	    string HTMLSource = "";
	    string HTMLTitle = "";
	    string HTMLDesc = "";

	    // if no URL then all properties are blank
	    if (String.IsNullOrWhiteSpace(this.URL))
	    {
			HTMLTitle=HTMLDesc = HTMLSource = "";
			return;
	    }

	    try
	    {
			HttpWebRequest request = (HttpWebRequest)WebRequest.Create(this.URL);
			HttpWebResponse response = (HttpWebResponse)request.GetResponse();

			if(response.StatusCode == HttpStatusCode.Forbidden)
			{
		    	throw new Exception(String.Format("We are not allowed to visit this URL {0} 403 - Forbidden", this.URL));
			}
			else if(response.StatusCode == HttpStatusCode.NotFound)
			{
		    	throw new Exception(String.Format("This URL cannot be found {0} 404 - Not Found", this.URL));
			}
            // 200 = OK, we have a response to analyse
            else if (response.StatusCode == HttpStatusCode.OK)
            {
                Stream receiveStream = response.GetResponseStream();
                StreamReader readStream = null;
                if (response.CharacterSet == null)
                {
                    readStream = new StreamReader(receiveStream);
                }
                else
                {
                    readStream = new StreamReader(receiveStream, Encoding.GetEncoding(response.CharacterSet));
                }

                string source = this.HTMLSource = readStream.ReadToEnd();

                if (!String.IsNullOrWhiteSpace(source))
                {
                    // extract title with a simple regex                        
                    string reTitle = @"^[\S\s]+?<title>([\S\s]+?)</title>[\S\s]+$";
                    // Get the match for META TITLE if we can using a wrapper method that takes a regex string, source string and group to return
                    string retval = this.GetMatch(reTitle, source, 1);
                    if (!String.IsNullOrWhiteSpace(retval))
                    {
                        this.HTMLTitle = retval;
                    }
                    else // failed to find the <title>Page Title</title> in the HTML source so throw a Regex Exception
                    {
                        throw new RegExException(reTitle, new Exception(String.Format("No match could be found for META Title with {0}", reTitle)));
                    }

                    // META DESCRIPTION
                    string reDesc = @"^[\s\S]+?<meta content=[""']([^>]+?)['""]\s*?name='description'\s*?\/>[\s\S]*$";
                    // Get the match for META DESC if we can using a wrapper method that takes a regex string, source string and group to return
                    retval = this.GetMatch(reDesc, source, 1);
                    if(!String.IsNullOrWhiteSpace(retval))
                    {
                        this.HTMLDesc = retval;
                    }
                    else // failed to find the <title>Page Title</title> in the HTML source so throw a Regex Exception
                    {
                      throw new RegExException(reDesc, new Exception(String.Format("No match could be found for META Description with {0}", reDesc)));
                    }
                }

                response.Close();
                readStream.Close();
            }
	    }
	    catch(WebException ex)
	    {
          	// handle 2 possible errors
          	if(ex.Status==WebExceptionStatus.ProtocolError)
          	{
              Console.WriteLine("Could not access {0} due to a protocol error", this.URL);
          	}
          	else if(ex.Status==WebExceptionStatus.NameResolutionFailure)
          	{
              Console.WriteLine("Could not access {0} due to a bad domain error", this.URL);
          	}
	    } // catch anything we haven't handled and throw so we can code in a way to handle it
	    catch (Exception ex)
	    {
			throw;
	    }
	}


	private string GetMatch(string re,string source,int group)
	{
	    string ret = "";

	    // sanity test
	    if (String.IsNullOrWhiteSpace(re) || String.IsNullOrWhiteSpace(source) || group == 0)
	    {
			return "";
	    }
	    // use common flags these could be passed in to method if need be
	    Regex regex = new Regex(re, RegexOptions.IgnoreCase | RegexOptions.Compiled | RegexOptions.Multiline);

	    if (regex.IsMatch(source))
	    {
          	MatchCollection matches = regex.Matches(source);

          	Console.WriteLine(String.Format("We have {0} matches", matches.Count.ToString()));

		foreach (Match r in matches)
		{
		    if (r.Groups[group].Success)
		    {
				ret = r.Groups[group].Value.ToString().Trim();
				break;
		    }
		}                
	    }
	    // return matched value
	    return ret;
	}
}

 

So as you can see this HTML Class is my basic BOT. It has 4 properties, URL (which is just to set the URL of the code to get) then: HTMLSource, HTMLTitle, and HTMLDesc which as you can see are all abbreviations of the 3 main bits of the content we want from our BOT, the META Title, META Description, and the whole HTML source. We set these using the code in our HTML classes GetHTML() which does the main work but uses another method GetMatch() to try and get the match of the parameters passed in e.g the expression, the source code to get any match from, and the no of the match to get.

 

For example, there may be bad XHTML/HTML all over the web where people have used ID"s multiple times or put elements that should only exist once in multiple times. With this last parameter, you can ensure you are getting the match you want, usually 1 for the first.

 

Hopefully, you can see what the class is doing and how the results of the IsMatch method can fire off a custom RegExException or not and how it is formatted.

 

In the next article, we will show the content that uses this class to return data and put it all together

 

 

© 2022 By Strictly-Software 

Extending Try/Catch To Handle Custom Exceptions - Part 1

Extending Core C# Object Try/Catch To Display Custom Regular Expression Exceptions - Part 1

By Strictly-Software

Have you ever wanted to throw custom exception errors whether for business logic such as if a customer isn't found in your database, or whether more complex logic has failed that you don't want to handle but raise so that you know about it.

 

For me, I required this solution due to a BOT I use that collects information from various sites every day and uses regular expressions to find the pieces of data I require and extract them. The problem is that the site often changes its HTML source code and it will break a regular expression I have specifically crafted to extract the data using it.

 

For example, I could have a very simple regular expression that is getting the ordinal listing of a name using the following C# regular expression:

 

string regex = @"<span class=""ordinal_1"">(\d+?)</span>";

Then one day I will run the BOT and it won't return any ordinals for my values and when I look inside the log file I find it will have my own custom message due to finding no match such as "Unable to extract value from HTML source" and when I go and check the source it's because they have changed the HTML to something like this:

<span class="ordinal_1__swrf" class="o1 klmn">"1"<sup>st</sup></span> 

 

This is obviously gibberish many designers add into HTML to stop BOTS crawling their data hoping that the BOT developer has written a very specific regex that will break and return no data when met with such guff HTML. 

 

Obviously, the first expression whilst meeting the original HTML alright is too tight to handle extra guff within the source. 

 

Therefore it required a change in the expression to something a bit more flexible in case they added even more guff into the HTML:

 

string regex = @"<span class=""ordinal_+?[\s\S]+?"">""?(\d+?)""?<sup";

 

As you can see I have made the expression looser, with extra question marks ? in case quotes are or are not wrapped around values, and using non-greedy match any character and non-character expressions like [\s\S]+? to handle the gibberish from the point it appears to where I know it has to end at the closing quote or bracket.

 

So instead of just logging the fact I have missing data from crawls I wanted to raise the errors with TRY/CATCH and make the fact that a specific piece of HTML no longer matches my expression an exception that will get raised so I can see it as soon as it happens. 

 

Well with C# you can extend the base object TRY/CATCH so that your own exceptions based upon your own logic can be used. In future articles, we will build up to a full C# Project with a couple of classes, a simple BOT and some regular expressions we can use to test what happens when trying to extract common values from the HTML source on various URL's to throw exceptions.

 

Creating The TRY CATCH CLASS

First off the TRY / CATCH C# Class where I am extending the base object to call, usual messages but using String,Format so that I can pass in specific messages. I have put numbers at the start of each method so that when running the code later we can see which exceptions get called.

 

I have just created a new solution in Visual Studio 2022 called TRYCATCH and then named my RegExException that extends the Exception object. You can call your solution what you want but for a test, I don't see why just following what I am doing is not okay.

[Serializable]
public class RegExException : Exception
{
public RegExException() { }

public RegExException(string regex, string msg)
   : base(String.Format("1 Regular Expression Exception: Regular Expression: {0}; {1}", regex, msg )) { }

public RegExException(string msg)
    : base(String.Format("2 {0}", msg)) { }

public RegExException(string regex, Exception ex)
    : base(String.Format("3 Regular Exception is no longer working: {0} {1}", regex,ex.Message.ToString())) { }

}

 

As you can see the Class has 3 overloaded methods which either just take a single message, a regular expression string and a message, and a regular expression and an exception these values are placed in specific places within the exception messages.

 

You would cause a Regular Expression Exception to be thrown by placing something like this in your code:

 

throw new RegExException(String.Format("No match could be found with Regex {0} on supplied value '{1}'", re, val));

 

Hopefully, you can see what I am getting at, and we will build on this in a future post.

 

© 2022 By Strictly-Software 

 

 

Make Your First "Hello World" Brave / Chrome / Edge Extension

Making your 1st Chrome / Brave Extension....

By Strictly-Software

This article is about how to go ahead and make your first extension for  Brave, Chrome, or Edge. As I no longer use Chrome but use Brave instead which like Microsoft Edge is based on the core Chromium Open Source standards-compliant browser, any mention of Brave in this article can be applied to Chrome or Edge as well.

The good thing about choosing to write your first extension for a Chromium-based browser is that once you have done it, it can be applied to 3 major browsers, Chrome, Brave, and Microsoft's replacement for Internet Explorer, Edge. So if you want to maximize your efforts choosing to develop an extension for Chromium gives you much more scope to apply it to other major browsers unlike Mozilla for Firefox. 

Not only do I like Brave for paying me not any site owner, to watch adverts in cryptocurrency (BAT), it is more secure, with an easy way to allow or prevent tracking cookies, 3rd party cookies, scripts, adverts, popups, and more from following you around the web.

It does this by stripping out all the tracking codes by default before rendering the page.

By doing this it also saves bandwidth and therefore time. It also uses TOR as it's an incognito window so you are more secure when trying to avoid people by going through the onion to access many pages that Brave will sometimes automatically redirect you to.

You can download the Brave Browser from this link and start to strip trackers and earn cryptocurrency for ignoring tiny little adverts that are shown to you in the corner of the screen right now.

Setting Up Your Extension

1. Create a folder somewhere on your machine where all the files for the extension will be located. I just created a folder in documents called "Extensions" and then within it for this test a "HelloWorld" sub folder.

2. Create a manifest.json file with basic info about your extension. The manifest.json file tells Chrome important information about your extension, like its name and which permissions it needs.

The manifest version should always be 2, because version 1 is unsupported as of January 2014. So far our extension does absolutely nothing, but let’s load it into Chrome / Brave anyway.

Let's add some info into it which describes the extension.

{
  "manifest_version": 2,
  "description": "Hello World Extension",
  "name": "Strictly-Software Hello World Extension",
  "version": "0.1.1"
}

3. Create a content script which is "a JavaScript file that runs in the context of web pages." This means that a content script can interact with web pages that the browser visits.

So open a text file and make the script, as we are doing a hello world all we need to do is something simple such as showing an alert box whenever we go to a new URL that shows a message e.g:

// content.js
alert("Hello from Strictly-Softwares Brave/Chrome extension!");

Save the file in the same folder as all of your other files as content.js. 

As we want the alert box to show on every URL we access we need to add a bit of script in the manifest file which tells it to act on <all_urls> you can see this in the below example as we also tell the manifest.json file about our JavaScript that needs to run on all the URLS we access.

{
  "manifest_version": 2,
  "description": "Hello World Extension",
  "name": "Strictly-Software Hello World Extension",
  "version": "0.1.2022"
  "content_scripts": [
    {
      "matches": [
        "<all_urls>"
      ],
      "js": ["content.js"]
    }
  ]
}

If we wanted the extension code to only run on certain pages then we can use regular expression like syntax which you can explore later to define the URLS that our code acts on.

Also if we were loading multiple JS files into our manifest file, we would use commas within the square brackets to denote them e.g to add jQuery go and get the version of the plugin you want e.g from https://ajax.googleapis.com/ajax/libs/jquery/3.3.1/jquery.min.js and save a local copy into your extension folder. 

There is no point loading it from a remote site each time as it takes time to load, plus they have stopped using the "latest" version in their filename which you used to be able to reference to get the most up to date version from Google's or jQuery's CDN, however they stopped it due to the amount of requests they were getting which was almost on an unintentional DOS scale.

For framework files you should always get a version that you know works, and save it locally or on a CDN, so that it can be cached and not loaded from a remote site each time. Also there is no need to keep updating it to the next version when one comes out. If it works for the project you are working on then leave the script reference alone. 

This is something many developers didn't understand by always loading in the latest version remotely, this took network calls, time to load, and it often introduced new bugs into their code. It is far better to store all your code locally so it can be cached and so that you know it works. So to add jQuery into the extension we would do this.

"js": ["jquery-3.3.1.js", "content.js"]

The manifest would look like this if you did add extra files in e.g:

{
  "manifest_version": 2,
  "description": "Hello World Extension",
  "name": "Strictly-Software UserAgent Extension",
  "version": "0.1.2022",
  "content_scripts": [
    {
      "matches": [
        "<all_urls>"
      ],
      "js": ["jquery-3.3.1.js", "lightbox.js", "scripts.js", "content.js"]
    }
  ]
}

4. Add a logo for your extension at the size of 24x24 this will appear in your toolbar when you pin it so you can use the extension. At the same time make some more icons that will be used on the brave://extensions/ page when loading up your extension, for example when you click on it to see the details (Name, description, version etc), and elsewhere.

You can use this page to convert an image, screenshot, logo into the sizes you need > https://icoconvert.com/.

The icon for the toolbar should be called just icon.png which you can put in the same folder as the other images which you should name with the size on to not get confused. 

At the same time make .png logos of the sizes 16x16, 48x48 and 128x128. These are put at the top level in the manifest as you can see below as they are used in different places. The toolbar icon is down at the bottom as you can see.

So the final manifest file will look like this.

{
  "manifest_version": 2,
  "description": "Hello World Extension",
  "name": "Strictly-Software UserAgent Extension",
  "version": "0.1.2022",
  "icons": {
    "16": "icon16.png",
    "48": "icon48.png",
    "128": "icon128.png"
  },
  "content_scripts": [
    {
      "matches": [
        "<all_urls>"
      ],
      "js": ["content.js"]
    }
  ],
  "browser_action": {
    "default_icon": "icon.png"
  }
}

5. Now in Brave/Chrome go to extensions and open it up. 

In Brave it will be a URL like brave://extensions/ and Chrome chrome://extensions/. In the top right corner turn developer mode on. Then in the top left corner hit the "Load Unpacked" button and select your folder containing all your files and images.

You will see your extension appear in the list of already loaded extensions with one of your logos showing e.g:

5. Clicking on "Details" will show the information from your manifest and a different size logo e.g:

6. Now select the extension icon in the toolbar to get up your drop down menu of extensions and select the pin so it's pinned to your toolbar e.g:

7. Now try going to any URL and you should get an alert box pop up straight away with the message you put in the content.js file. Even before other JavaScript loaded content fires from a DOM or Window onload event, this alert should fire straight away e.g:

8. And there you go, your first Chrome/Brave extension. Obviously, this just a Hello World test showing you how you can create a basic extension and if you are going to make an extension you will need to read up more on all the different features and actions possible.

However, this is a good guide for someone wondering about how to make an extension or insert code that fires before the page is loaded. 

I have an idea for what I need to use this for such as overwriting JavaScript objects and putting my own properties into them so that they load when the page does before any other JavaScript code as well as being accessible to any local code that tries to access them. You can read a bit about this in a later article, injecting a script at the top of your HTML page.

If you want more information about what you can do just head to the main Chrome extensions page for developers at https://developer.chrome.com/docs/extensions.

Have fun...

By Strictly-Software

Changing Blogger With JavaScript

Using JavaScript To Edit Blogger Layout

By Strictly-Software

If you are using Googles blogger to output content then there will often be times where you want to edit the layout depending on the page you are on, or the content that is shown.

I recently had a site where I needed to edit the content in multiple places depending on whether a GoFundMe post was being shown on the homepage, or it's own page. As I also had a mini GoFundMe logo in the sidebar I didn't want both images to be shown at the same time.

I also had a Custom Google Search Bar which only searched a certain number of websites related to the content on the blog.

Therefore the logic I needed was to.

1. Once the page had loaded I needed to ensure if a post of GoFundMe was shown on the page then I needed to remove the sidebar image in ID #Image4.

2. Just doing a test for #Image4 doesn't work as the widgets change names server side before output therefore I need to list out the pages / URLS that the post appears on where I know that the image in the side bar and the main post image both exist.

3. I need to add text to a special Google Search Bar when the site has loaded, sometimes this can be slow depending on Google's Servers, so therefore I decided to put it in a window.onload event to ensure all other images and external items had loaded before I edited the input bar.

4. On focus of the search bar I need to clear it to allow people to put their own search terms.

5. On leaving the search bar if it's empty then I need to put the default text back in, a simple onblur, onfocus toggle.

As the widget I added (an HTML/JavaScript widget), was at the bottom of the page the likelihood of the code running after these items had loaded was high so the chances of the code running before the search bar being loaded was low. 

However on the chance it did run, the window.onload event unlike DOMLoad events ensure that all external items have loaded. If I tried a DOMLoad event there was a small chance that the code would run before the items it was affecting had loaded.

Therefore to test that the page was showing a post with the GoFundMe on it and therefore removing a server side loaded item from the page I used Regular Expressions. 

I looked at all the places the pages loaded this post and they were the homepage, the homepage for mobile or non mobile sites, with or without the Google # link for the search bar tab being in the URL, plus certain labels that only this post appeared on.

The Regular Expression I came up with that worked using my test page I made in HTML & JS that worked was this.

^https:\/\/sitename.blogspot.com\/?(\??(m=\d)?(#gsc.tab=\d+)?|\S+?GoFundMe\S+?|\S+?\/(?:Donate|Donations|GoFundMe)\S+?)?$

This expression uses the document.location.href value to run on and is anchored from the start to end of input with ^ and $.

It always contains the main domain in the URL and I have made the ending / optional with a ? e.g escaped with a forward slash first and putting a question mark after it e.g \/?.

The next part is whether or not a querystring ? exists which is why there is an escaped question mark first to denote a real question mark, followed by another one to say it "May or may not be there". 

The next part is a check to see if there has been a toggle used on the site to change from a mobile view to a website view. They use a simple querystring value for this m=0 (website), m=1 (mobile), so I have (m=\d) which denotes m=(any digit), just incase they change or append digits.

This is the 1st part of an OR section wrapped in a group. I could have put ?: at the start of the group to say "do not store", but as I am not back referencing or using the group in replacements then I didn't bother.

The whole OR is broken down into this logic.

The logic contained within the OR (expression|expression|expression) is the equivalent to this long explanation:

(\??[optional back slash](m=\d)?[optional mobile/website URL toggle to any digit](#gsc.tab=\d+)[optional anchor that denotes Google has loaded up its search bar code as if you are quick you can catch a document.location.href value without this on it]. Then a wrap around the main post called GoFundMe with \S+? which makes it look for non space characters 1 or more times before or after the words GoFundMe which appear on the posts page in the middle of the URL e.g https://sitename.blogspot.com/2021/10/httpssitename.blogspot.comGoFundMe.html.html?m=0#gsc.tab=0 and then finally a look for the 3 labels that are used to find the page and appear in the URL like so https://sitename.blogspot.com/search/label/Donations?m=0 (as you can see when I copied the URL the page hadn't loaded in the Google Search Bar which is why there is no #gsc.tab=0 in the URL) so this is why I have made that part optional as well.)

So that regular expression handles the following URL's whether or not the Google code has loaded or there is code to force it show a mobile site or website version of the blog.

https://sitename.blogspot.com

https://sitename.blogspot.com/

https://sitename.blogspot.com/?

https://sitename.blogspot.com/?m=0 or https://sitename.blogspot.com/?m=1 

https://sitename.blogspot.com/#gsc.tab=0

https://sitename.blogspot.com/?m=1#gsc.tab=0

https://sitename.blogspot.com/2021/10/httpssitename.blogspot.comGoFundMe.html.html#gsc.tab=0

I made a quick function call to querySelector so that it is short, I always used G when it was document.getElementById(val), so I used it again as to me it means GET G(x) = GetMe(x) a reference to the small framework I made which is on the blog somewhere e.g

G = function(v){return document.querySelector(v)};

Then I can reference objects by class, selector or ID easily with a one letter function e.g

var el=G(".gsc-search-button > input");

G("#Image4").remove();

The final code I inserted into the JavaScript/HTML widget was this.

<script type="text/javascript">
G = function(v){return document.querySelector(v)};
window.addEventListener("load", function() 
{
	if(G("#gsc-i-id1")){
		G("#gsc-i-id1").value = "Search for similar articles here";
	}
	G("#gsc-i-id1").addEventListener("focus",function(){
		G("#gsc-i-id1").value="";
	});
	G("#gsc-i-id1").addEventListener("blur",function(){
		if(G("#gsc-i-id1").value==""){
			G("#gsc-i-id1").value = "Search for similar articles here";
		}
	});	
});

// As these elements are loaded server side they will exist when any JavaScript gets to run so no need to put inside the Window.Onload event
var exp = new RegExp(/^https:\/\/sitename.blogspot.com\/?(\??(m=\d+)?(#gsc.tab=\d+)?|\S+?GoFundMe\S+?|\S+?\/(?:Donate|Donations|GoFundMe)\S+?)?$/,"gi");
if(exp.test(document.location.href)){
	G("#Image4").remove();
}

// search button never appears wide enough to say "Search" so expand
var el=G(".gsc-search-button > input"); // Use selector to get child input of class gsc-search-button which is used numerous times

if(el){el.style.width="60px";} // just a safety test to ensure it's there - which it should be
</script>

It works and I am happy to see no double GoFundMe images on the page and the focus/blur code working fine.

It is just an example of using JavaScript to modify the DOM when you cannot use Server Side code to initially output it.

© 2021 By Strictly-Software

Do You Want Multiple Wickr Accounts?

How to get another Wickr Account To Work

By Strictly-Software

If you are like me you might have an Android phone with Wickr the encrypted chat app installed on it.

However if you have been blocked by someone, locked out, forgotten your password or just want another Wickr account it's not easy as Wickr is tied to your Google account and device and if you try and install it from Google Play Store on a different device it will just say "Already Installed". 

You have to either remove the app from your phone and start again or use a different phone with a different Google Account linked to it as Google Play always knows what devices you have installed and which apps are on it.

I tried the obvious route first, use an Android Emulator such as NOX, and then created a new Google Account and Email address and then went to the PlayStore to download Wickr. The only thing was the link was broken and it kept sending me to an error page.

I tried downloading the APK file from a site I searched for Wickr Download APK, but even after installing it, it got to the 99% of "encrypting your device" e.g my NOX Emulator on Windows 8.1, and just hung forever.

Therefore I couldn't get the emulator to work. I could have installed another one but whats the point of slowing down a PC with multiple Android emulators just to watch free TV from Navisport etc?

Therefore I actually did a search for "Download Wickr For Windows 8.1" and all I got were help articles telling me to download further emulators and then install from Google Play. This was obviously a no go.

Therefore I actually tried the following which did work on Windows 8.1 from this link.

The Windows 10 link is the first one on the left. The rest are APK, Mac and IOS files.

To install on a Windows 8.1 machine do the following, without the need for an Android Emulator.

1. Create a new Google Account, username, password, add it to the 12 you already have.
2. Log out of any Google account you might be in e.g especially if you are using Chrome.
3. Go to this link https://pro-download.wickr.com/#/version/pro
4. Select "Windows - For use with Windows 10"  and download the application.
5. Install the MSI file called WickrMe-5.4.3.msi.
6. It may take a while but it should start to install, even on Windows 8.1.
7. Once installed, create a new Wickr account and username.
8. Find your old contacts and start chatting from your laptop or PC.

The direct download link can be found here if you click here.

It works for me, so it should work for you, don't spend hours trying to getting your emulator to install an APK file that just hangs. Try the Windows 10 Desktop MSI file first.

By Strictly-Software

Boggle - Online Game

Playing Boggle Online - A Game For XMAS In Pure JavaScript, CSS and HTML

By Strictly-Software

I made this online version of Boggle, the shake letters and make words from letters that join together as my Dad was learning JavaScript, HTML and CSS. So I made it purely in those languages.

You too can play the game if you want online on any up to date browser that is standard compliant and runs JavaScript on my site at https://www.strictly-software.com/games/boggle.html

The version I made uses a single Boggle object with a number of settings, properties and timers. I have split all the functions out into their own methods and added some features that let you edit the game such as:

-The time the game plays for. It defaults to 03:00 minutes but you can change it to any time in the 00:00 format.

-There is a drop down box with Easy, Medium and Hard for you to select. Each option has a different array of letters that are randomly picked for the game. You are more likely to get multiple X, Z and Qu's in the hard level than the easy one.

-There is an option to select the MAXIMUM red letters than appear in a game. It defaults to 3. You might not get 3 but you won't get over 3. You can change this to another numerical value up to 9. When you pick a word containing a red letter you get double points.

-There is a random tick box which means that the letters chosen for the game are purely picked at random from the alphabet array defined in the Boggle object.

If your browser is working (I have tested it on my TV, Laptop, Tablet and phone) then the countdown timer should turn from green to yellow after 90 seconds and then start flashing red during the last 30 seconds before beeping at the end.

If your TV doesn't do all this then it is down the the TV's browser. Test it online on a PC/Laptop first to ensure it all works.

We played a few games of it at Christmas and it worked perfectly.

It is an ideal way for someone learning HTML, JavaScript and CSS to understand the code, tweak it and learn about selectors, short cuts, regular expressions and so on.

It makes use of my lightweight framework which supports chaining, setting HTML, removing nodes callbacks, CSS selectors and much more. 

You can read about the Getme.js framework and how I made it here > http://blog.strictly-software.com/2009/10/build-your-own-framework.html.

I use SS instead of $ like jQuery and other frameworks and I am of the mindset that pure JavaScript should be used as much as possible. 

An example of chaining is joining multiple functions together with one initial selector to pick the array of nodes to work on.

SS("DIV .mycontrols input).setAtts({style:"color:black;"}).setHTML("Game Over!);

Where the node list (array) from the first selector is passed into the second function so that only those relevant nodes/elements are acted upon. Read the Getme.js article for more details.

All the files can be downloaded to your local machine to play offline and In Chrome to see the code in action go to More Tools > Developer Tools and open the Console in the bottom part of the screen.

Turn the debug variable in the Boggle.html file to true and you will see all the debug to help you understand which functions are being called and if any errors are shown they will appear in red. 

Play around and make the CSS, HTML different and add extra functions or features with JavaScript.

Play it online here > https://www.strictly-software.com/games/boggle.html

Enjoy the game, we did!

Obtaining an external IP address in memory for use in a Firewall Rule

Obtaining an external IP address in memory for use in a Firewall Rule

By Strictly-Software

As I am currently using an ISP which constantly changes my external IP address due to excessive use of DCHP, I have to regularly update external firewalls on servers to allow my computer remote access.

This is obviously a right pain to do and I have no idea why my ISP changes my IP address so much when my old ISP used DCHP and kept it for months at a time.

Therefore I created this little noddy VBScript to sit on my desktop to obtain my IP address and hold it in my clipboard memory ready for me to just open up my firewall and paste it in.

The code uses the MSXML2.ServerXmlHttp object to obtain the IP address from an external website which just prints it on the page and I store the Response.Text into a variable.

I then use WScript.Shell object to open a new Notepad window and write the IP address out into it.

Then I use SendKeys to select the IP address and copy it into the clipboards memory before shutting down Notepad.

This means I can just quickly double click my desktop shortcut icon to obtain the IP address ready to paste it straight into a Firewall rule.

Check this script out and when you are ready and can see that hitting CTRL + V pastes the IP address out elsewhere you should remove the "TEST SECTION" and uncomment the part above it that closes down Notepad. This ensures you are not leaving empty objects around in your computers memory.

It's not exactly amazing code but it's very helpful at this time and saves a lot of time visiting a website manually to get my external IP address.

You might find this useful yourselves!

Option Explicit

Dim IPAddress, objShell
Dim objHTTP : Set objHTTP = WScript.CreateObject("MSXML2.ServerXmlHttp")

'* Obtain external IP address and store it in a variable
objHTTP.Open "GET", "http://icanhazip.com", False
objHTTP.Send
IPAddress = objHTTP.ResponseText

'* Open Notepad
Set objShell = WScript.CreateObject("WScript.Shell")
objShell.Run "notepad.exe", 9

WScript.Sleep 1000

'* Write out the IP address to a blank notepad file
objShell.SendKeys IPAddress

'* select the IPAddress and copy it into memory
objShell.SendKeys "^{a}"
objShell.SendKeys "^{C}"

'* uncomment the following section and remove the "TEST SECTION" when you are ready

'* Close notepad with the IP address in clipboard ready to be pasted
'* Open Save Dialog
'* objShell.SendKeys("%{F4}")
'* Naviagate to Don't Save
'* objShell.SendKeys("{TAB}")
'* Exit Notepad
'* objShell.SendKeys("{ENTER}")

'* TEST SECTION - Proves that the IP address can be pasted elsewhere
WScript.Sleep 1000

'* Proof that the IP address is in the clipboard and can be pasted out
objShell.SendKeys "Test Paste Works"
objShell.SendKeys "{ENTER}"
objShell.SendKeys "^{V}"
objShell.SendKeys "{ENTER}"
objShell.SendKeys "Try a manual CTRL and V to check the IP address is still in memory"
objShell.SendKeys "{ENTER}"

'* Kill objects - DO NOT REMOVE!!
Set objShell = Nothing
Set objHTTP  = Nothing

By Strictly-Software

© 2018 Strictly-Software

Stopping BOTS - A Multi Layered Approach

Stopping BOTS - A Multi Layered Approach

By Strictly Software

Some people don't mind BOTS of all shapes and form roaming their sites but if you actually look into what they are doing should you be worried about their actions?

Have you examined your log files lately to see what kind of BOTS are visiting and how much bandwidth they are using?

Here are a few of the reasons you might want to care about the type of actions carried out by automated crawlers (BOTS):

1. They eat bandwidth. Social media BOTS especially who jump onto any link you post on Twitter causing Twitter Rushes. This is where 50+ BOTS all hit your site at the same time and if you are not careful could use up all your memory and cause a frozen system if not configured properly. There are plenty of articles about Twitter Rushes on this site if you use the search option down the right hand side to find more details.

2. Bandwidth costs money. If you are a one man band or don't want high server costs then why would you want social media BOTS, many that provide no benefit to you, costing you money just so they can provide their own end users with a service?

3. Content theft. If a user-agent identifying itself as IE6 is hitting a page a second is it really a human using an old IE browser visiting that many pages? Of course not. However for some reason IE 6 is the most popular user-agent used by script kiddies, scrapers and hackers. Probably because they have just downloaded an old crawler script off the web and run it without the knowledge to edit the code and change the agent. Look for user-agents from the same IP hitting lots of pages per minute and ask yourself are they helping your business or just slowing your site down by not obeying your robots.txt crawl-delay command?

4. Hacking. Automated hackbots scan the web looking for sites with old OS systems, old code and potential back doors. They then create a list of sites for their user and come back to penetrate these sites with SQL/XSS injection hacks. Some might show up in GET requests in the log file but if they are tampering with FORM elements then any POSTED data containing hack vectors won't show up. Hiding key response parameters such as your server brand and model and the scripting language you use are good simple measures to prevent your sites name ending up on this list of potential targets to hack and can easily be configured in config files on your system.

Therefore you should have a defence against these type of automated BOTS. Of course you also have the human hacker who might find a sites contact form, view the source, tamper with the HTML and work out a way to modify it so he can send out mass emails from your server with a custom script. Again security measures should be implemented to stop this. I am not going to talk about the basics of security when it comes to preventing XSS/SQL injection but the site has many articles on the topic and basic input sanitation and database login security measures should stop these kinds of hack.

So if you do want to stop automated BOTS from submitting forms, registering to your site, applying for jobs and anything else your site might do the following list might be helpful. It is just an off the head list I recently gave to someone on LinkedIn but could be helpful if expanded to your own requirements.

On my own sites I use a multi pronged approach to stop BAD BOTS as well as bandwidth wasting social media BOTS, hack bots and even manual hackers tampering with the forms. It saves me money as well as increases performance by allowing legit users only to use the site. By banning over 50% of my traffic which is of no benefit to me I can give the 50% of useful traffic a better user experience.

1) We log (using Javascript), whether the user has Javascript enabled e.g an AJAX call on the 1st page they hit that sets a session cookie using Javascript. As most BOTS don't use Javascript we can assume if they have Javascript enabled they are "probably" human.

2) We also use Javascript (or the 1st page HTTP_ALL header in IE) to log whether Flash is enabled and the version. A combo of having Flash running and Javascript isbetter than just Javascript on it's own.

3) I have my own logger DB that records browser fingerprints and IP's, Useragent, Javascript, Flash, HTTP settings, installed apps, browser extensions, Operating System and other features that can almost uniquely identify a user. The problem is of course an IP often changes either through DCHP or the use of proxies, VPN's and hired VPS boxes for an hour or two. However it does help in that I can use this combination data to look up in my historical visitor database to see what rating I gave them before e.g Human, BOT, SERP, Hacker, Spammer, Content Thief and so on. That way if the IP has changed but the majority of the browser finger print hasn't I can make an educated guess. If I am not 100%  sure however I will then go into "unsure mode" where security features such as CAPTCHAS and BOT TRAPS are introduced just in case. I can then use Session variables if cookies are enabled to store the current status of the user (Human, BOT, Unknown etc), or use my visitor table to log the browser footprint and current IP and do lookups on pages where I need to use defensive measures if cookies are not enabled.

4) These Session/DB settings are then used to decide whether to increment banner hit counters, write out emails in images or with Javascript so that only humans can see them (to prevent BOT email scrapers), and other defensive measures. If I know they are 100% human then I may chose not to deploy these measures.

5) On forms like contact forms I often use BOT Traps. These are input elements that are in the flow of the form with names like email_extra that are hidden with CSS only. If the BOT submits a value for this hidden input I don't submit the form, or I do but without carrying out the desired action and not let the BOT know that nothing happened.

6) A lot of forms (especially contact forms) can be submitted by just entering an email address for all fields (name, email, password etc). Therefore I check that the field values are different e.g not the same value for an email AND password field. I also ensure the name matches a name pattern with a regular expression.

7) I have built my own 2 stage CAPTCHA system which can be turned on or off on the fly for forms where I don't know if the user is 100% human OR I can decide to just always have it on. This is based around a maths question, where the numbers are in 3 automatically created images, grey and blurry like normal CAPTCHA's The user has to first extract the right numbers from the images then carry out an automated sum from those numbers e.g add number 1 to number 2 and deduct number 3. This works very well as it requires a human brain to interpret the question and not just use OCR techniques to extract the CAPTCHA image values. There are so many OCR breakers out there that a standard CAPTCHA where you enter the word on the picture can easily be cracked automatically now.

8) If there is textarea on the form, contact, application etc, then I use my RUDE word table which has hundreds of variants of rude words and the regular expression next to it to detect them. This can obviously be updated to include pharmacy pill names, download movies, porn and other spam words.

9) I also have a number of basic regular expressions if the user wants light detection that checks for certain strings such as "download your xxx now", "buy xxx for just $£", and words like MP3s, Films, Porn, Cialis and other common spam words that would have no place on a site not selling such goods.

10) I always log any blocking so I can weed out any false positives and refine the regular expressions etc.

11) I also have an incremental ban time so the 1st time anyone gets banned is for 1 hour, then 2, then 4 then a day etc etc.The more times they come back the longer they get banned.

12) Sometimes I use JavaScript and AJAX to submit the form instead of standard submit buttons. As Javascript is so commonly used now (just look at Google), then most people have it enabled otherwise the majority of sites just wouldn't work or would have minimum features. It would require a human hacker to analyse your page to break it and then write a custom BOT just to hack the form when a technique like this is used. To get round this you can use a rolling random key created server side, inputted into a hidden element with Javascript on page load and then examined on form submission to ensure it is correct. If it's not then the person has tampered with the form by entering an old key not the new key and can be banned or blocked.

13) Another good way to stop automatic hack BOTs (ones that just roam the web looking for forms to try and submit and break out of to send emails etc - contact forms), is to not use FORM tags in your server side code but have compressed and encrypted JavaScript that on page load converts the <div id="form">....</div> into a real FORM with an action, method etc. Anyone viewing the non generated source code like most BOTS, won't see a FORM there to try to hack. Only a generated HTML source view (once the page has loaded), would show them this, which most BOTS would not be able to view.

14) Honeypots and Robots.txt logging is also useful e.g log any hit to the robots.txt file and for any BOTS that don't visit it before crawling your site. You can then make a decision to ban them for breaking your Terms Of Service for BOTS that should state they should obey your Robots.txt rules.

15) As BAD BOTS usually use the links in the DISALLOW section of Robots.txt to crawl anyway. Then putting a fake page in the list of URLs is a good idea. This page should be linked to from your site in a way that humans cannot see the link and accidentally visit it (and if they do it should have a Javascript link on it to enable them to get back to the site). However BAD BOTS will see the link in the source and crawl it. As they have broken your TOS and followed a URL in your DISALLOW list they are being doubly "bad", so you have the right to send them off to a honeypot (many exist on the web that either put emails out for them to extract then wait for an email to be sent to that address to prove they are an email scrapper bot) OR they get sent to an unbreakable maze like system which auto generate pages on the fly so that the BOT just keeps going around in circles crawling page after page and getting nowhere. Basically wasting their own bandwidth.

16) HTACCESS Rules in your .htaccess file should identify known bad bots as well as IE 6, 5 and 5.5 and send them off to a 403 page or a 404 so they don't realise they have been sprung. No-one in their right mind should be using these old IE browsers anymore however most downloadable crawlers used by script kiddies still use IE 6 as a user-agent for some reason. My guess is that they were written so long ago that the code hasn't changed or that people had to support IE 6 due to Intranets being built in that technology e.g using VBScript as the client side scripting language.

By using IE 6 as a UA they get access to all systems due to sites having to support that ancient horrible browser. However I ban blank user-agents, user-agents less than 10 characters long, any that contain known XSS/SQL injection vectors and so on, There is a good PHP Wordpress plugin called Wordpress Firewall that if you turn on all the features and then examine the output in your .htaccess file will show you some useful rules such as banning image hot linking that you can then nick for your own file.

17) Sending bad bots back to their own server is always a good trick so that they get no-where on your own site. Another good trick is to send them to a site that might scare the hell out of them once they realise they have been trying to hack or DDOS it https://www.fbi.gov/wanted/cyber or the METS Cyber Crime department.

These are just a few of the security measures I use to stop BOTS. It is not a comprehensive list but a good starting point and these points can be expanded and automated depending on who you think is visiting your site.

Remember most of these points are backed up with detailed articles on this site so have a search if anything spikes your interest.

Hope this helps.

By Strictly Software


© 2016 Strictly Software

IE Bug with non Protocol Specific URLS

IE Bug with non Protocol Specific URLS

By Strictly-Software

I have recently come across a problem that seems to only affect IE and non protocol specific URLS.

These are becoming more and more common as they prevent warnings about insecure content and ensure that when you click on a link you go to the same protocol as the page you are on.

This can obviously be an issue if the site you are on is on an SSL but the site you want to go to doesn't have an HTTPS domain or vice-versa. However most big sites have both domains and will handle the transfer by redirecting the user and the posted data from one domain to another.

An example would be a PayPal button that's form posts to

action="//www.paypal.com/"

In the old days if you had a page on an SSL e.g https://www.mysite.com and had a link to an image or script from a non secure domain e.g http://www.somethirdpartysite.com/images/myimg.jpg you would get pop ups or warning messages in the browser about "non secure content" and would have to confirm that you wanted to load it.

Nowadays whilst you don't see these popups in modern browsers if you check the console (F12 in most browsers) you will still see JavaScript or Network errors if you are trying to load content cross domain and cross protocol.

S2 Membership Plugin

I came across this problem when a customer who was trying to join one of my WordPress subscription sites (that uses the great free S2 Membership Plugin), complained to say that when he clicked a payment button that I have on many pages, was being taken to PayPal's homepage rather than the standard payment page that details the type of purchase, price and options for payment.

It was working for him in Chrome and FireFox but not IE.

I tested this on IE 11 Win7 and Win8 myself and found that this was indeed true.

After hunting through the network steps in the developer toolbar section (F12) and comparing it to Chrome I found that the problem seemed to be IE doing a 301 redirect from PayPals HTTP domain to their HTTPS one. 

After analysing the response and request headers I suspect it is something to do with the non UTF-8 response that PayPal was returning to IE for some reason, probably because Internet Explorer wasn't requesting it as such for some reason.

Debugging The Problem

For the techies. This is a breakdown of the problem with network steps from both Chrome and IE and relevant headers etc.

First the Paypal button code which is encrypted by S2Member on the page. You get given the button content as standard square bracket short codes which get converted into HTML on output. Looking at the source on the page in both browsers on one button I could see the following.

1. Even though the button outputs the form action as https://www.paypal.com it seems that one of my plugins OR WordPress itself, although I suspect a caching plugin obviously, is changing my links (I haven't been able to narrow it down - or WordPress) is removing any protocols conflicts by using non protocol specific URLS.

So as my site doesn't have an SSL any HREF, SRC or ACTION that points to an HTTPS URL was being replaced with // e.g https://www.paypal.com on my page http://www.mysite.com/join was becoming //www.paypal.com in the source and generated source.

2. Examining the HTML of one of the buttons you can see this in any browser. I have cut short the encrypted button code as it's pointless outputting it all.

<form action="//www.paypal.com/cgi-bin/webscr" method="post">
<input type="hidden" name="cmd" value="_s-xclick">
<input name="encrypted" type="hidden" value="-----BEGIN PKCS7-----MIILQQYJKoZIhvcNAQcEoIILMjCCCy4CAQExgg..."

3. Outputting a test HTML page on my local computer and running it in IE 11 WORKED. This is was probably because I explicitly set the URL to https://www.paypal.com so no redirects were needed.

4. Therefore logically the problem was due to the lack of an HTTPS in the URL.

5. Comparing the network jumps.

1. Chrome

Name - Method - Status    - Type                     - Initiator
webscr - POST - 307      - x-www-form-urlencoded    - Other
webscr - POST - 200 - document             - https://www.paypal.com/cgi-bin/webscr

2. IE

URL         - Protocol  - Method - Result - Type       - Initiator
/cgi-bin/webscr         - HTTP      - POST   - 301       -           - click
/cgi-bin/webscr         - HTTPS    - POST   - 302       - text/html  - click
https://www.paypal.com/home - HTTPS   - POST   - 200       - text/html  - click

Although the titles are slightly different you can see they just are different words for the same thing e.g Status in Chrome or Result in IE both relate to the HTTP Status code the response returned.

As you can see Chrome also had to do a 307 (HTTP 1.1 successor to the 302 temporary redirect) from HTTP to HTTPS however it ended up on the correct page. Whereas in IE when I first clicked on the button it took me to the payment page in HTTP but then did a 301 (permanent) redirect to it in HTTPS and then a 302 (temporary) redirect to their home page.

If you want to know more about these 3 redirect status codes this is a good page to read.

The question was why couldn't IE take me to the correct payment page?

Well when I looked at the actual POST data that was being passed along to PayPal from IE on the first network hop I could see the following problem.

cmd=_s-xclick&encrypted=-----BEGIN㲠----MIILQQYJKoZIhvcNAQcEoIILMjCCCy4CAQExgg...

Notice the Chinese character after the BEGIN where it should say PKCS7?

In Chrome however this data was exactly the same as the form e.g

cmd:_s-xclick
encrypted:-----BEGIN PKCS7-----MIILQQYJKoZIhvcNAQcEoIILMjCCCy4CAQExgg...

Therefore it looked like for some reason the posted data was being misinterpreted by IE whereas in Chrome it was not. Therefore I needed to check what character sets and response was being sent and returned.

Examining Request and Response Headers

When looking at the HTTP Request headers on the first POST to PayPal in IE I could see that the Accept-Language header was only asking for en-GB e.g a basic ASCII character set. Also there was quite a lack of request headers compared to IE. I have just copied the relevant ones that can be compared between browsers.

IE Request Headers

Key         Value
Content-Type: application/x-www-form-urlencoded
Accept-Language: en-GB
Accept-Encoding: gzip, deflate
Referer: http://www.mysite.com/join-now/
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko
Request: POST /cgi-bin/webscr HTTP/1.1
Accept: text/html, application/xhtml+xml, */*
Host:         www.paypal.com

Chrome Request Headers

Key                 Value
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Encoding:gzip, deflate
Accept-Language:en-GB,en-US;q=0.8,en;q=0.6
Cache-Control: max-age=0
Connection: keep-alive
Content-Length:4080
Content-Type: application/x-www-form-urlencoded


And the responses for the Content-Type header which I think is key.

IE

Content-Type: text/html

Chrome

Content-Type: text/html; charset=UTF-8


So whilst Chrome is saying it will accept more language sets and gets back a charset of UTF-8 IE is only saying it will accept only en-GB and gets back just text/html.

I even tried seeing if I could add UTF-8 in as a language to accept in IE but there was no option to, so I tried adding Chinese which obviously use extended character sets and was the problematic character.
 However this made no difference even though the Accept-Language header was now:

Accept-Language: en-GB,zh-Hans;q=0.9,zh-CN;q=0.8,zh-SG;q=0.7,zh-Hant;q=0.6,zh-HK;q=0.4,zh-MO;q=0.3,zh-TW;q=0.2,zh;q=0.1

Conclusion

Therefore I came to the conclusion that I could not force IE to change it's behaviour and I doubt any phone calls to IE HQ or even PayPal would solve the issue. Therefore to allow IE users to be able to still pay on my site I needed a workaround.

1. I added to my technical test page which I make all users check before paying a test for IE and a warning about possible problems. This went alongside tests to ensure JavaScript and Cookies were enabled as both are needed for any modern JavaScript site.

2. I added in some JavaScript code in my footer that run on DOM load that looped through all FORM elements and checked the ACTION attribute. Even though when examining the results in the console they showed http://www.paypal.com rather than what I saw in the source //www.paypal.com I added some code in to ensure they always said HTTPS.

The function if you are interested is below and it seems to have fixed the problem in IE. If I view the generated source now I can see all form actions have HTTPS protocols.

// on DOM load loop through all FORM elements on the page
jQuery(document).ready(function () {	
	// get all form elements
	var o,,e=document.getElementsByTagName("FORM");
	for(var i=0,l=e.length;i<l;i++)
	{
		// get the action attribute
		o = e[i].action;

		// if current action is blank then skip
		if(o && o!="")
		{
			// if the start of the action is http (as non protocol specifc domains show up as http)
			// then replace the http with https
			if( /^http:\/\/www.paypal.com/.test(o) )
			{
				e[i].action = o.replace("http:","https:");
			}		
		
		}
	}	
});

So whilst this is a just a workaround for the IE bug it does solve the issue until Internet Explorer sorts itself out. Why they have this problem I have no idea.

I am outputting all my content as a UTF-8 charset and Chrome is obviously handling it correctly (along with Firefox and Safari).

So I can only presume it's an IE bug which isn't helped by an unknown (as yet) plugin (or WordPress) changing cross protocol URLs to the now standard //www.mysite.com format.

Therefore if you come across similar problems with redirects taking you to the wrong place, check your headers, compare browsers and if you spot something strange going on, try a JavaScript workaround to modify the DOM on page load.


© 2015 Strictly-Software