Welcome to part 5 of our series of blog posts dissecting each of the datasets available as part of Abusix Mail Intelligence.
This time we’re looking at our Domain Blocklist. This list is built solely by messages hitting our trap infrastructure and is 100% automated.
How the Domain Blocklist is being built:
To begin, I’ll need to refer you back to part 1 of our series and specifically to the different trap types we use.
The domain blocklist uses messages hitting our main trap pools, however, exceptions are made to allow other trap types to contribute where we see patterns in the URLs that indicate Phishing, Compromised websites, Freenom TLDs, newly observed domains, or where the domain is commonly used for abuse (typically free hosting or DDNS services that offer free sub-domains).
Domains and URLs are extracted from these messages, domains on the white/welcome-list are excluded and all other remaining domains are then listed. Any bare IPs that are seen in anchor tags are also listed. (Read more on our transaction from the terminologies blacklist and whitelist in this blog post)
If any short URLs are found, these are extracted and the domains that these URLs point to are also listed.
We try to be as careful as possible to avoid things like opt-in confirmations or messages sent by malware which typically use a genuine message thread and attach malware to these, therefore the domains contained would be genuine.
All domains are stripped to their “organizational domain” using the Mozilla Public Suffix List. This allows us to easily normalize domains to match against our white/welcome-list and also prevents attackers from using subdomains to avoid listings.
We maintain a manual list of exceptions to the Public Suffix List for cases where we only want to list subdomains – an example of this would be for a hosting service that offers sub-domains to their customers where we don’t want to list the organizational domain, but instead list the abusive customers subdomain.
Reasons for being listed & how to avoid getting listed on our Domain Blocklist:
Common reasons for being listed in our domain blocklist list are:
- Professional spam
- Compromised website
- Abusable web forms
Let’s deal with each of these (except the professional spam case) and how you can avoid these problems.
Compromised Website
The most common compromises we see are WordPress sites. All of these could be avoided if they were kept up to date with the latest patches, plugins etc.
What typically happens is that once the site is compromised – spam, phishing, or malware is uploaded to your site and then URLs to these are then used directly in spam that is sent out in high volumes from other hosts, which causes your domain to be listed.
Abusable Web Forms
Spammers constantly search the internet for services to exploit in some way to send spam, phishing, or malware.
They do this by using automated bots that are used to look on your site for sign-up/registration forms, mailing-list sign-ups, or “Send to a friend” features to see if they can be abused to send messages to an innocent 3rd party.
If your forms append your site URLs, this can cause your domain to be listed when the form is abused.
That’s it for this week, hope that is useful!
Until next time – stay safe.
Steve
