Automatic MIME attachments triage

Published: 2015-11-17
Last Updated: 2015-11-23 19:44:58 UTC
by Xavier Mertens (Version: 1)
2 comment(s)

Today, we are all receiving a huge amount of spam containing multiple malicious documents (cfr my last diary). If most of them are part of massive spam campaigns and are already well known, others could be very interesting (because new or targeting specific people/groups). Being the happy owner of domain names and email addresses for years (some for 15+ years!), they are listed in all spammers' databases around the world. This helps me to receive a lot of emails. I'm collecting mails tagged as spam or send to unknown recipients. That's a kind of email honeypot.

The amount of mails received being important, it's boring to process them manually. I needed a way to automate the processing of all those attachments. That's the purpose of "mime2vt.py". This Python script extracts MIME types from email flows and checks them against virustotal.com. This is a basic check that helps in the triage of all those malicious files. Today, the implemented features are:

  • MIME attachments are optionally dumped in a directory (for later investigations)
  • VirusTotal API key used to:
    • Check if the file is known (its score is returned and logged)
    • Submit unknown files
  • Non relevant MIME types can be excluded (ex: image/png,image/gif,image/jpeg,text/plain,text/html)
  • Results are logged via Syslog
  • Zip archives are inspected
  • Virustotal results are send to an Elasticsearch instance (optional)

I recently updated the script with bug fixes and new features:

  • URLs are extracted from emails 
  • MD5 hashes are stored in a local database 

As the scripts reads the mail from STDIN, it's easy to implement it in any environment. Example, I'm a fan of procmail:

:0
* ^X-Spam-Flag: YES
{
    :0c
    | /usr/local/bin/mime2vt.py -d /var/mime/%y/%m/%d -c /etc/mime2vt.conf
    :0
    spam
}

In the example above, emails flagged as spam are piped to mime2vt. A copy of the MIME attachments will be saved in /var/mime/2015/11/16 (a daily repository). Here is an example of detected file:

Nov 16 12:12:50 marge mime2vt.py[9238]: DEBUG: config_file = /etc/mime2vt.conf
Nov 16 12:12:50 marge mime2vt.py[9238]: Processing zip archive: 8e10533a4b624f90b646a971fcf063c5.zip
Nov 16 12:12:50 marge mime2vt.py[9238]: DEBUG: Extracted MD5 06a4059da943b09f13ab2909824968de from Zip
Nov 16 12:12:51 marge mime2vt.py[9238]: DEBUG: VT Response received
Nov 16 12:12:51 marge mime2vt.py[9238]: File: dhl16112015_6987878544212.exe 
(06a4059da943b09f13ab2909824968de) Score: 3/54 Scanned: 2015-11-16 09:51:31 (2:21:20)

If the file is already known by VirusTotal, its score is returned as well as the scan time (+ time difference). Optionally, the Virustotal JSON reply can be indexed by Elasticsearch to generate live dashboards:

The script is updated quite often but it already available on github.

Xavier Mertens
ISC Handler - Freelance Security Consultant
PGP Key

Keywords:
2 comment(s)

Comments

Hi Xavier. I like the idea of such a script. I am a newbie on postfix. I have built my own email server following the tutorial on https://www.exratione.com/2014/05/a-mailserver-on-ubuntu-1404-postfix-dovecot-mysql/.

It is basically a postfix, dovecot, mysql, Amavis, ClamAV and SpamAssassin integration.
Can you give me some pointers how or where to integrate your script on it?

Would it not also be great to run the oledump-py script of Didier to let the recipient know of what is embeded in the attachement(s)?

Greatful for any advice you can give.
Thanks
We can discuss this topic off-list if you want.
About the automatic scan of OLE documents, it's ongoing, I've a new version of the script almost ready... :-)

/x

Diary Archives