We all know that web applications are the new firewall. However, so far we had a hard time collecting web application logs. The hard part is to balance ease of install of a sensor (without disrupting the web application), fidelity of the log information and privacy.

With firewall logs, it is pretty simple. A rejected packet in a firewall has very little information and privacy isn't a big issue. Web application are different as the actual "meat" of the log event is in the request content, which may contain personal information. Parsing web logs isn't so easy either. Administrators frequently customize log formats for special purposes.

To balance these different issues we decided to focus on errors, but instead of parsing logs, we set up a little php script that you can add to your error page. In its current form, the script will work with PHP web servers (tested with Apache) that support the curl extension. Curl is installed by default in current versions of PHP.

Now all you need is an "error page". In Apache, just use the ErrorDocument configuration directive. For example:

ErrorDocument 404 /error.html

Will redirect users to "/error.html" in case of a 404 error [1].  You may already have a page like that configured. All you need to do is add the php snippet to the end, sending us the intended URL, the user agent and the IP address of the client access the missing page.

The hope is to collect data from automated probes, similar in how DShield's firewall logs reflect portscan activity.

In particular if you are running a personal / home web server: Please consider adding the collector script.

Once we get a few submitters, we will start adding continuously updated reports to the site, just like we do for the DShield data. However, we can't do this until we have at least a dozen submitters (better 100 or more) . We can not publish "one off" errors as they will likely be specific to your site and again could cause privacy issues.

Why do we only support PHP? Well, that's the language I know. Feel free to submit a .Net/Java/Ruby/Perl or whatever version of the script.

Simple steps to sign up:

1. Login to retrieve your authentication key here https://isc.sans.edu/myinfo.html
2. Download the php snippet here https://isc.sans.edu/tools/404project.html
3. paste it into your Error Document
4. test...

Please contact us if you have any questions.

[1] http://httpd.apache.org/docs/2.0/mod/core.html#errordocument

------
Johannes B. Ullrich, Ph.D.
SANS Technology Institute
Twitter