The Journey of the Machine Learning Technology that Powers DNSFilter

In 2018, DNSFilter acquired the company Webshrinker, a company that utilizes machine learning technology to provide website screenshot and domain intelligence API services. We wanted to highlight the journey of how Webshrinker began and how far the technology has come.

How Webshrinker got started

Adam Spotton, now Head of Data Science at DNSFilter, started Webshrinker in 2012 as a screenshot-as-a-service company. Initially, the product was created to fulfill a need for educational use. A provider of curated lists of educational websites needed to provide thumbnails of the website resources but there were thousands of websites to screenshot.

Rather than do work manually or use an incomplete solution (there was only one other screenshot-as-a-service at that time), Adam built Webshrinker to automatically take screenshots of sites that could then be embedded in the educational resource solution, reducing countless hours of manual work.

In 2015, Webshrinker grew as domain categorization was added and became the main driver of the product.

Using cloud computing and machine learning technology, Webshrinker classifies websites into sets of categories by scanning URLs. Users of Webshrinker are able to take the URL-to-category mapping (available to them as either an API or downloadable database) and use it within their software or services.

After a Kickstarter campaign in 2016, DNSFilter became a Webshrinker customer and began integrating Webshrinker’s domain categorization into the DNSFilter product.

Iterating on technology

Webshrinker has gone through many iterations but one of the main transformations has been the technology it uses to simulate a full browser session. Initially, it used Selenium. Then we moved onto a combination of PhantomJS and AWS Lambda. But this caused rendering issues that affected the screenshot API.

Now Webshrinker simulates an actual Chrome browser without using third-party libraries like Selenium.

This is one of the most important aspects of Webshrinker: its ability to imitate an actual person who starts a browser session. This is particularly important when it comes to threat detection.

When Webshrinker processes a page, it opens a browser simulation. If Webshrinker was not programmed to handle browser simulations in a way that mimicked human behavior and a human-initiated web visit, deceptive sites could actually deceive Webshrinker.

In addition to changes to the browser simulator over time, we also regularly pull lists of domains that have recently been flagged as malicious by the system so we can spot-check them for accuracy.

For instance, we might take 400 domains marked as phishing sites and run them through aggregate tools like VirusTotal. (VirusTotal is our standard benchmark because it is a collaboration of over 150 antivirus and domain scanning tools). This gives us an indication of how many false positives (non-malicious sites marked malicious) that Webshrinker is flagging.

This is an example of a VirusTotal spot check done on a Webshrinker-flagged malicious site:


Doing these routine spot checks allows us to see where Webshrinker might be less accurate so we can continue to optimize the technology. As it stands right now, Webshrinker’s false positive rate is at roughly 4%.

Continued improvements

No software product is ever really finished, but especially not machine learning technology. We are actively working on making improvements to Webshrinker that will make it even more accurate. All of the spot checks implemented serve to make Webshrinker a stronger categorization tool.

We are also always optimizing the way that Webshrinker detects threats and feeding it sites to learn from. Doing this, we enable Webshrinker’s machine learning to better sort domains into malicious and non-malicious categories.

Planned improvements include creating additional, more granular categories. Eventually, image analysis will play a larger role in scanning the page during categorization to identify subcategories.

If you have a use for Webshrinker’s services, get in touch with us.

Part 2 to in this series shows how Webshrinker’s website categorization works.

  • There are no suggestions because the search field is empty.
Latest posts
Migrating from Cisco Umbrella to DNSFilter: It Pays to Make the Switch Migrating from Cisco Umbrella to DNSFilter: It Pays to Make the Switch

Navigating the complexities of cybersecurity challenges today means more than just being alert; it requires a readiness to adapt and embrace superior technologies for better protection of your digital assets. The recent announcement of Cisco Umbrella Roaming Clients end-of-life (EOL) on April 2, 2024, and its end-of-support (EOS) on April 2, 2025, has encouraged several organizations to consider the next steps in maintaining robust cybersecurity ...

Zero-Day Attacks: What Are They? Zero-Day Attacks: What Are They?

The term “zero-day attacks” is thrown around frequently with a lot of concern—and rightfully so. In today’s world where even the most menial tasks are conducted online, there is always some cyber threat lurking in the dark shadows of the internet. Picture this: A burglar finds a secret doorway to your house and decides to pay you a visit. All your assets are now accessible to him, even without your knowledge.

Mid-Winter Nights Hallucinations: Some Thoughts on Our New GenAI Category Mid-Winter Nights Hallucinations: Some Thoughts on Our New GenAI Category

AI, LLM, generative content, NLP, big data, neural processing, machine learning, GPT. In 2023 it's undeniable that these were some of the most heard terms from various businesses, news outlets and the social media sphere. Ultimately this alphabet soup can mean just as much as it sometimes doesn’t—and, as often is the case, the internet leans into the trend.Sites popped up everywhere—some reputable while others less so—promising cyberpunk profile ...

Explore More Content

Ready to brush up on something new? We've got even more for you to discover.