Welcome to part 2 of our series of blog posts dissecting each of the datasets available as part of Abusix Mail Intelligence. This time we’re looking at our IP blacklist. This list is built solely by messages hitting our trap infrastructure and is ~98% automated, but is the only list in our set where we maintain some manual entries for some of the worst repeat offenders.
How the IP Blacklist is being built:
Let’s start by talking about traps. All of our traps are domains that we have managed through a lifecycle. Nearly all of our traps will start off in our “recycled” pool and will remain there for an absolute minimum of one year.
At Abusix, we have a policy that we never use typo domains (e.g. domains that are “almost” the same as a well-known domain) as traps that can cause blacklisting events as we feel they make very poor traps and are likely to cause false-positives.
All emails sent to “recycled” traps will be SMTP hard rejected at the end of DATA whilst the messages sent will still go through our infrastructure. We also have domains donated to us via our blackhole.mx service. As we have no idea about the history of these domains or how long they will be pointed at us, they always remain in the same state as “recycled”.
We closely monitor the metrics of all of our traps and trap pools, e.g. how long have we owned them, how much daily traffic and whitelisted traffic hits them etc. The traffic from all of our traps is separated into different “origins” so that our detectors know exactly which trap pool the message came from. Once we’ve had the domain for at least a year and we’re happy with the metrics, it’s moved into our “trap” pool which means that we stop rejecting all mail sent to it, and it will start to be used to generate “blacklisting” events.
To build our IP blacklist, any messages hitting our “trap” pool from IP addresses that are not whitelisted, will be immediately included. Additionally, any messages which come into our trap network, regardless of the trap pool that attempts to use our traps as a relay are automatically included.
Any IP added to the list will then remain listed for 5.2 days after the last event that we saw from it, or until it is delisted. (If you need to delist your IP/domain from one of our lists, you can use our free lookup service here.)
To help catch additional spam where the spam is sent from a spread of IP addresses (so-called “snowshoe” spam), we record any hits on any trap pool from IPs in a /24 over 5.2 days and if we see “trap” hits on the same /24, then all the IP addresses that we’ve seen for that /24 during that period are also listed.
Additionally, any new IPs we see hitting any trap pool is also listed
Reasons for being listed & how to avoid getting listed on our IP Blacklist:
Common reasons for being listed in our IP blacklist:
- Professional spam
- Compromised or infected hosts (e.g. PCs, laptops, servers, routers, IoT devices)
- Compromised email accounts on genuine mail servers
- Abusable web forms
- Poorly maintained mailing lists
Let’s deal with each of these (except the professional spam case) and how you can avoid these problems.
Compromised or Infected hosts
Keep your devices up-to-date with the latest software, firmware, and plugin updates. Make sure you run Anti-Virus.
Currently, the most common compromises we see are from Windows PCs, Mikrotik routers, and WordPress. All of these could be avoided if they were kept up to date with the latest patches, firmware update, plugins etc.
Compromised email accounts
Weak passwords, passwords reused on other sites (which have their databases stolen), stolen credentials via malware, or stolen credentials via Phishing are some of the most common causes.
Using password managers and using unique passwords for each site is one of the best defenses to all of these.
On the server-side, enforcing good password strength and uniqueness using services like haveibeenpwned.com are some strategies that can prevent your users’ credentials from being used to send spam via your service.
Abusable web forms
Spammers constantly search the internet for services to exploit in some way to send spam, phishing, or malware.
They do this by using automated bots that are used to look on your site for sign-up/registration forms, mailing-list sign-ups, or “Send to a friend” features to see if they can be abused to send messages to an innocent 3rd party.
Here is a good (bad) example:
To: <spam target>
Subject: [Your Company Name] Your username and password
X-PHP-Originating-Script: 1032:class-phpmailer.php
Date: Tue, 18 Aug 2020 16:30:23 +0000
From: WordPress <[email protected]>
Message-ID: <[email protected]>
X-Priority: 3
X-Mailer: PHPMailer 5.2.1 (http://code.google.com/a/apache-extras.org/p/phpmailer/)
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Type: text/plain; charset="UTF-8"
Username: date Lori and Jenny www[dot]rb2020[dot]sitew[dot]org Password: dWnBdsHjWbe7 http://yourdomain.com/wp-login.php
This is a website registration page that has been abused to send spam to our trap network. In this case, the spammer used one of our email addresses to register and then used the username field to insert their message payload. In this case a dating spam website.
This will also cause significant issues for the website owner:
- The customer database is poisoned with bad sign-ups that send spam to potentially a significant amount of people, resulting in the host being blacklisted.
- If the database is used for a mailing-list, then every message sent will result in messages hitting our traps and the site being blacklisted each time.
- Cleaning the database of these bad sign-ups will be extremely difficult.
To avoid this, all sign-up forms and any forms that might cause email to be sent should:
- Have some sort of anti-bot sign-up defense like a reCAPTCHA challenge.
- Confirm that the user is actually genuine by sending an email with an activation link, this ensures that if the user typo’s their own address (this happens!).
- Validate all inputs and ensure that URLs are stripped where necessary.
Doing all of these will reduce the chance of your web forms being abused to send spam and poison your databases.
Poorly managed mailing lists
Automated sign-ups (e.g. bots), users typoing their addresses at sign-up, or messages being sent to very old customers (e.g. >1 year) are a common cause of getting blacklisted.
As with web forms, your mailing list sign-up procedure should do the following to avoid these common pitfalls:
- Have some sort of anti-bot sign-up defense like a reCAPTCHA challenge.
- Confirm that the user is genuine and has entered the correct email address by sending an email with an activation link that must be clicked before they are added to the list.
- Periodically ask the user if they wish to remain on the list periodically and remove anyone that doesn’t reconfirm.
- Automatically remove email addresses from the list that are rejected or bounce consecutively or have no interaction with your messages (e.g. never click any links).
- If you haven’t sent mail to an address for over a year, then you should not presume permission or that the address hasn’t changed hands since.
M3AAWG publishes a full Sender Best Practices guide which is worth reading.
Until next time – stay safe.
Steve