CSAM Month of False Positives - Our ISP Says We're Hosting a BotNet!

Published: 2014-10-08
Last Updated: 2014-10-08 15:17:05 UTC
by Rob VandenBrink (Version: 1)
4 comment(s)

It's a note that many of us have received.  If we're unlucky, it's a note that your (not-a-packet-expert) boss has received and we've had to explain it.  It usually goes like this:

We recently received 1 complaint(s) regarding the article below. This issue appears to have originated from your domain (IP address: x.x.x.x).
 
Please take the appropriate action with your customer or the account in question, enforcing your Acceptable Use Policy.
 
Regards,
 
Sincerely,

Bob the Administrator

Internet Abuse Team
Some ISP Communications
Email abuse@someisp.com


Please include your original email to SOMEISP.com in any replies.
 

We have traced activity to your connection that is likely caused by Botnet traffic. Botnets are programs that allow virus writes to take control of other computers. Please take all appropriate steps to eliminate this virus. The communication request may include attempted synchronization between the virus and the controller, spamming activities, network scanning, or spreading the virus to other connections.

Here is a general description of botnets: http://en.wikipedia.org/wiki/Botnet

We have included the name of the bot below where possible.

--Incident--
IP: x.x.x.x Port 41425
Timestamp: 2014-08-17 00:10:02 GMT
Infection: zeus-p2p

-


A note like this takes a fair amount of explanation when discussing it with your client (or manager), who usually isn't a packet guru.  Usually the narrative goes something like this:

While (maybe) it's a good thing that an ISP is looking at traffic to find malicious patterns, unfortunately, in this case the ISP just plain gets it wrong.  Their entire conclusion is based on the source port.  Unfortuntately, the source port in most TCP communications is what is called an "ephemeral" port - it's either the next free port available after 1024, or a random free port above 1024. 

Add to that, in outbound communications, most TCP sessions are run through a firewall that NAT each session ( NAT = Network Address Translation), so that each of these ephemeral ports is mapped to yet a different ephemeral port.

As you can see, from the outside network's perspective, the source port could be just about anything.  So if you are looking for a "suspicious" port, you'll find it as a source port on any corporate firewall most days.

OK, so should the abuse@ folks look at the port at the other end?  Umm- sort-of.  Say someone is sending you an email or browsing to your website - in this case, the ephemeral port is at their end, and the target port is at your end.

Really, what you want to do is look at the ports at both ends - but not to make a final determination.  You might use that method to narrow down your search, before you then look at the contents of the actual packets.  An even better approach is to *start* by looking at the contents of the packets - but if you're an ISP that adds up to a lot of packets which needs a corresponding amount of CPU and disk to deal with it.

If you are an organization who needs to get a handle on outbound traffic (which is just about every organization), consider implementing an IPS on the traffic to/from the inside interface of your firewall - at that location you can see the true IP addresses and ports of all traffic, both source and destination. 

Another great thing to look at implementing is a Netflow collector (or sFlow or jFlow or IPFIX, depending on your gear) - a tool like this helps you to slice and dice your network traffic statistics to get quick answers, or in many shops  netflow graphical display is the most used feature, allowing a quick, visual way to identify "odd" traffic or trends.

Also an egress filter goes a long way to containing problems - if you only allow permitted traffic and have a default deny policy for all "mystery" traffic, you'll find that your problems are more likely to stay contained within your network, and if you are looking at your logs you'll be alerted much earlier in the process.

What you'll find with proper monitoring like this (IPS, Netflow, egress filtering and log monitoring) is that very likely you *do* have infected machines here and there.  For instance we did the "log check" thing last week at a client site and found 1 station trying to infect the neighbors with shellshock.  The difference between your logs and the ISP's logs though is that your logs will properly identify the problem workstation, and the alerts in your logs will be a lot less likely to be a false positive.

The sad thing that I've found is that once you get a false positive note like this from an ISP, you're generally in for a few weeks of them before you can escalate to a support person who understands enough to know that they're getting it wrong, and has the clout to fix the ISP's process.

Please, use our comment form to let us know if you've gotten a note like this, and if it was a false positive or not. 

===============
Rob VandenBrink
Metafore

Keywords:
4 comment(s)

Comments

In addition to this, organizations can also have the Shadowserver Foundation keep an eye on their networks:
https://www.shadowserver.org/wiki/pmwiki.php/Involve/GetReportsOnYourNetwork
I have had this happen at least 3 times in the last 5 or 6 years. The real challenge was one time the ISP only provided the source IP (which was the global NAT address for all 8,000 internal PCs) and when asked what were the destination addresses so I could correlate the IP in our logs and pinpoint the problem system, they stated they couldn't provide that to me for privacy reasons. Doing my best to remain professional, I then asked the ISP if they could tell me at least the destination ports (knowing this would be a much more difficult correlation but doable by matching up timestamps and destination service ports). I was then told that they only monitored source port. Still remaining professional, I simply said we would look into it. A week later I was attending a tech conference and it just so happened that the ISP we use had a booth on the exhibition floor selling their business internet services. I took the opportunity to talk to the service development manager there and explained my experience. His facial expression was priceless. He simply handed me his business card and asked me to email him the details of the notification from his company. Proud of myself, I emailed him that evening after dinner. By morning, I woke up to an email from the ISP with destination IP addresses. I handed this off to my team and within 15 minutes they were able to pinpoint the offending PC. A single bit of information such as destination IP address could have saved us a weeks worth of going back and forth.
I've definitely experienced this on similar level. We were running a IDS tool that alerted that a critical server was "beaconing to known c&c sites" over an ephemeral port. Further investigation revealed that the port being identified was indeed the source port. By this time, management was involved and asking all sorts of questions. It took a decent sized write up to show them why this was a false positive. When it was all said and done, I spent too much time explaining what the issue was than actually analyzing the traffic.
Unlike Virgin media which don't seem to care. I had a mail server that was under assault on a daily basis for months, from or through one of their assigned IP addresses that I traced to Dudley in the West Midlands. I must have written at first warn them, and subsequently to complain in excess of thirty times and never got as much as a response.

Diary Archives