Welcome to Part 7 of our series of blog posts dissecting each of the datasets available as part of Abusix Mail Intelligence. This time we’re looking at our Short URL Hash list. 

How the Short URL Hash List is being built:

This list is completely automated and is being built from messages hitting our main trap network. For frequently abused domains (e.g. those commonly used for spam, phishing and malware) or for free domains, we look for these domains in our entire trap network.

What is this Short URL Hash List designed for:

Short URLs are a major problem. You have absolutely no idea where they are going to take you prior to clicking them and the services themselves are absolutely useless at taking abusive links down. We can’t list the shortener domain itself as that would cause lots of false-positives.

The spammers, phishers and malware authors have turned to URL shorteners for these reasons and because it makes it much harder to filter the messages.   

To determine where a short link points, you have to make an external web connection to that shortener service.  This doesn’t scale if you handle anything more than a handful of messages per minute, so it’s difficult for a spam filter to do this in real-time.

We created the Short URL hash list to provide immediately actionable intelligence that scales to hundreds of messages per second.  As soon as your spam filter sees what looks like a short URL, it can check to see if we’ve seen that short URL in spam.

Abusix Mail Intelligence is the only commercial mail reputation provider to produce this data, however because we were the first to offer this, it will take time for the spam filter vendors to add support for it. At the time of writing only rspamd supports these lookups.

Reasons for being listed on the Short URL Hash List:

Common reasons for being listed in the Short URL Hash List:

• Spam, phishing and malware.
• Compromised websites are abused to host redirectors, phishing, or malware pages, and shortened links to these sites are then used.

Hope that is useful.

Until next time – stay safe.

Steve