Wednesday, July 6, 2005

Airport/3c905 Packet Loss Solved

Airport ExpressI was experience 30-60% packet loss on my Airport Extreme network, here's how I diagnosed and solved that problem by forcing the link speed to a slower rate.

After my firewall/gateway Linux box died due to a bad hard disk, I temporarily used my Airport Extreme as my primary network access point. When I finally fixed my firewall/gateway I had a problem, I was experiencing very high packet loss on the wireless lan. This was a major issue as my beautiful wife makes exclusive use of the wlan for her internet access. This made her problem, my problem.

3c905b ethernet adapterFirst, a bit about network layout. My network is set up quite simply, there is an incoming DSL connection to the internet with a few static IPs. This connects to an interface on the gateway/firewall machine. This machine has two other interfaces, one of which is connected to the WLan, the other of which is connected to the wired network. Obviously one is trusted, the other is not and the firewall operates in all three directions. The Airport Extreme operates in bridging mode passing all traffic from the firewall to the wlan. It is connected directly to the firewall via its dedicated ethernet card.

The first evidence that there was a problem came when there was very slow/laggy web performance and interactive services (i.e. ssh) that appear to 'lag' for several seconds at a time. This intermittent performance was proven out when I ran ping from one of the laptops in two steps. The first was to ping 100 packets to the gateway itself.

laptop$ ping -c 100

This experiment, repeated a few times, showed that packet loss varied between 30% and 60%. Since there was another device in the chain, namely the airport, I repeated this test going from the laptop to the airport and from the gateway to the airport. What I discovered was that there was no trouble (<1% packet loss) between the Airport and the laptop, but the major packet loss was present when pinging the Airport from the gateway. This isolated the network link between the Airport and the firewall as the culprit.

laptop$ ping -c 100
gateway$ ping -c 100

Always going for the simplest solution first, I immediately swapped the ethernet cable between the two devices for a different one and repeated the tests. This reduced the packet loss to 30%-50%, but it was still entirely unacceptable. Therefore I decided to use a program called ethereal to analyze the traffic on that link. Using its command-line version, tethereal, I set up a dump of all icmp.

gateway$ tethereal -i eth2 'icmp'

Running the ping session again from my laptop to the gateway I was able to see that all of the ICMP Ping Request packets arrived at the gateway, and the gateway was transmitting Response packets for them. This means that there was no troubles sending data from the airport to the gateway, but the responses from the gateway were getting lost. This corresponds to what I had previously seen with Airport Express devices on another network where very small packets like those sent to acknowledge receipt of data were getting lost by the Airport. When tiny packets get lost, my first thought is always that some device isn't able to keep up with the flow of traffic, and so I sought out a means to slow the network link down.

Just to be sure that this isn't a well known issue, I went online to the keeper of all answers and searched for information on packet loss between 3c59x adapters and the Apple Airport Extreme. In general, you always want to type the obvious into Google. I didn't find any direct answers but that search did lead me to the page discussing the 3c59x driver and its options. This was helpful because my plan was to reduce the speed of the connection between the two devices. The page also linked to the mii-diag diagnostic program which could read and set the configuration of the 3c905B-TX cards in my system.

Using this program, I forced the speed of the ethernet adapter to 10baseT. Not the ideal world in a device which is capable of running at 54Mbps, but not only do I have only one device which uses the higher speed, but I'll take 0% packet loss at 1/5th the base speed any day.

UPDATE 2005-08-13

Make sure that if you are running your Airport Extreme in bridging mode that you use the WAN, not the LAN ethernet port. This trick solved a packet loss problem that I was having on another network when the Airport Extreme was plugged directly into a Cisco Gigabit Switch while operating in bridging mode.

No comments:

Post a Comment