Living in Florida, afternoon thunderstorms are a regular occurrence with Florida having the highest lightning density of any state in the US [1]. In my time in Florida, I had close or direct strikes damage equipment twice. The most recent incident was about a month ago. So I am sharing here some of the things that work and don't work.

This most recent strike didn't hit the house directly as far as I can tell. But the house was engulfed in a large flash lasting several seconds. I am including a video clip of the strike as it was captured on a security camera. Lightning can cause damage in different ways. First of all, a direct strike will of course inject significant voltage and current, causing equipment to melt and even fire. A few neighbors were affected by such a strike hitting a cable TV line, melting several cable modems, and causing one small fire. The more likely damage is however not from a direct strike. Even a close strike, like the one I experienced, can cause damage as the strong electric field will induce currents that will in particular affect low voltage equipment like networks. Networks again are particularly sensitive. Longer cables may pick up more of the electric field.

First a list of the equipment the strike damaged (none of the equipment was visibly damaged):

• Cable modem.
• Firewall (only the port connected to the cable modem)
• One PoE switch lost its PoE function but still passed traffic.
• A second PoE switch lost a couple of ports.
• a small PoE powered switch was completely dead
• Apple TV wired network port was dead but works otherwise
• The projector powered on but displayed no picture
• The receiver connected to the projector would no longer output an HDMI signal
• The subwoofer connected to the receiver was dead.

I may have missed a couple of things, but needless to say, the damage was substantial.

Surge Protectors / UPS

All equipment but the project was connected to a UPS. But I don't think the UPS played a role here. It may have prevented worse damage. So far, it looks like all the damage was caused via network ports (the receiver was connected to the dead PoE switch, but there is also a long HDMI cable from receiver to project that may have played a role).

I do have a surge protector that is well-grounded as the cable enters the house. Note that cable companies usually only ground the cable, and do not install a surge protected. Coax surge protectors are a bit tricky. "Gas Tube" surge protectors that are sometimes used can wear out over as little as 5 years or even built up a charge that itself can cause damage. The surge protector in my case had no visible damage and still appears to work fine, For damage caused by current induced by a close-by lightning discharge, surge protectors are not of too much use. 

PoE Equipment

My Power-over-Ethernet (PoE) equipment did a lot worse than other equipment. There are various anecdotes that can be found that may support that PoE is most sensitive to lightning. The Ethernet standard does include some requirements for over-voltage protection [2] but of course, there are limits, PoE in particular adds additional components like transformers that are vulnerable to excessive voltages. 

Fiber

My network uses some fiber runs in part to electrically isolate network segments. Part of the network is in a separate building with its own power feed and ground point. Back in my physics days, I dealt a lot with sensitive electronics close to high voltage systems and ground loops were an ongoing issue, so I decided early on to use fiber for some of the longer connections (still well within the 100m copper ethernet limit). This strategy worked very well and likely helped contain the damage. Most of the damage appears to have happened either by currents induced by the high potentials of the lightning, or by voltage spikes traveling via network cables.

Recovery

Initial debugging showed that the firewall and the cable modem were out. I do have an LTE modem connected to the firewall. Only the port connected to the cable modem was damaged, and the LTE modem worked well, but due to the cheap data plan I am using, the LTE modem ran out of data within about an hour. Comcast sent a technician next day to replace the modem (I am using static IP addresses which requires a leased modem). 

For the firewall, I did have a spare that was not powered on and I replaced the damaged firewall. The damaged switches worked well enough initially.For the most part, only PoE devices (couple of security cameras and wireless access points) didn't work. I had an old spare switch around, but it was a different type and would have required significant configuration so I decided to wait the two days until the replacement switch arrived. Luckily a replacement switch was readily available.

Lessons Learned

Fiber works! It probably protected my main workstation and with that the most valuable asset that would have been expensive to replace. It is hard to tell if UPSs played a role. Another important lesson is to have some powered-off "cold standby" equipment. Automatic failover and such is nice to have, but in this case, the failover switch/firewall would likely have been damaged as well. As for backup internet connectivity, I will be trying the unlimited 5G home internet which just became available. The speed of the LTE modem was barely usable and having limited data plans was a pain. I am going to deploy a few more ethernet surge protectors on ports that connect to outdoor PoE cameras. Maybe going to replace some of these ethernet cables with shielded cables (but the cameras all survived)

Final Note

A couple of days after I replaced the cable modem, I had another odd network issue: All of a sudden, only IPv6 connections worked, and IPv4 failed. At the time, I was just reconfiguring IPv6 on the firewall, as my IPv6 allocation changed with the modem swap. So I suspected the firewall, undid my last change, but still no luck. It took me a couple of hours until I realized that IPv4 still went over the LTE modem, while IPv6 used the cable modem, and the LTE modem had just run out of data again. But due to multiple equipment changes along the way, this was the last thing I checked "retracing" my configuration path.

Video of the lightning strike

[1]https://www.weather.gov/media/safety/09-18Flashes_Flash_Density_State.pdf
[2]https://incompliancemag.com/article/designing-ethernet-cable-ports-to-withstand-lightning-surges/

--

Johannes B. Ullrich, Ph.D. , Dean of Research