Tracking HTTP POST data with ELK
Last Updated: 2015-11-06 11:17:01 UTC
by Xavier Mertens (Version: 1)
The Apache web server has a very modular logging system. It is possible to customize what to log and how. But it lacks in logging data submitted to the server via POST HTTP requests. Recently, I had to investigate suspicious HTTP traffic and one of the requirements was to analyze POST data. If you already have a solution which performs full packet capture, you're lucky but it could quickly become a pain to search for information across gigabytes of PCAP files. In the past, ngrep ("network grep") helped me to capture interesting traffic like this:
This command will capture all Wordpress login attempts and write packets to a PCAP file. We have now only useful data but, here again, processing them remains a pain. Hélas, sniffing packets is not relevant against HTTPS traffic!
So, I searched for an alternative and decided to use the combination of mod_security and ELK. mod_security being available on most distributions (and I hope used!), it will reduce the need to deploy new software components. The first step is to collect the POST data. By default, mod_security logs only blocked HTTP requests. To log all the POST requests, modify/add the following lines in your mod_security.conf (/etc/apache2/mods-enabled/mod-security.conf on Ubuntu distributions):
SecRule REQUEST_METHOD "POST" "id:1000,phase:2,ctl:auditEngine=On,nolog,pass"
The first directive specify which sections will be logged. In our case, the most important if the "C" which contains the request body). Once done, restart Apache and check the log for new events. Here is a typical Wordpress login attempt:
[06/Nov/2015:10:45:09 +0100] Vjx2pcOa89sAAAbH5QsAAAAJ 22.214.171.124 40119 127.0.0.1 8080
POST /wp-login.php HTTP/1.1
User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:23.0) Gecko/20100101 Firefox/23.0
HTTP/1.1 200 OK
Expires: Wed, 11 Jan 1984 05:00:00 GMT
Cache-Control: no-cache, must-revalidate, max-age=0
Set-Cookie: wordpress_test_cookie=WP+Cookie+check; path=/
Content-Type: text/html; charset=UTF-8
Stopwatch: 1446803109179197 352941 (- - -)
Stopwatch2: 1446803109179197 352941; combined=260, p1=192, p2=61, p3=1, p4=1, p5=5, sr=50, sw=0, l=0, gc=0
Producer: ModSecurity for Apache/2.6.3 (http://www.modsecurity.org/); OWASP_CRS/2.2.8.
The mod_security log format is not the most friendly to read but it is very complete! Hopefully, somebody developed a Logstash parser for mod_security. It is easy to deploy. Once indexed, events will be available in Kibana for investigations. Here is a Wordpress login brute force attack:
And you will find all the HTTP requests details easily browsable:
The combination mod_security/ELK will help you:
- To collect and store for a long retention period your POST data for further investigations (even if the web traffic is encrypted)
- To beautify data (ex: automatic geolocalization as seen above)
- To correlate events with other sources.
WARNING: mod_security is an Apache module and has access to all data in clear text. Keep in mind that HTTP POST requests could contain sensitive information (passwords, ID's, CC number, SSN, etc). You can sanitize sensitive data directly within mod_security via the sanitseArg directive.
ISC Handler - Freelance Security Consultant
Remember that Wireshark can relatively easily decrypt HTTPS traffic.
You can extract the key from the server, and I believe you can use the ephemeral keys from clients to decrypt their specific connections.
Nov 7th 2015
7 years ago
Nov 9th 2015
7 years ago
Nov 10th 2015
7 years ago