SummarySome exinda 4010's cause network outage by not going to the bypass state. This has been determined as a driver issue with the e1000e drivers for the NICs in a 4010
OverviewExinda has a server model 4010. It can be described as a 4710 or a 4810 Exinda by some users as well. In a few 4010 devices we have seen that the bridge interfaces start flapping intermittently and then stop passing traffic.
The bridges don't go into a bypass state. Instead it gets stuck at a point where it is trying to pat bypass, but bypass never kicks in.
In these cases users end up rebooting the Exinda in order to restore network connectivity.
CauseWe have determined the cause to be a driver issue with the e1000e drivers the 4010 NICs use.
The following are the 3 symptoms to look out for
1. Users see NIC flapping/bridge warnings (going up and down)
2. The following log messages
bypassd: TID 139811876615936: [bypassd.NOTICE]: br2,br4,br6 (Fw8756) Bypass timer timed out.
bypassd: TID 139811876615936: [bypassd.NOTICE]: br2,br4,br6 (Fw8756) Bypass timer timed out.?
3. If the NIC doesn't go into bypass then network connectivity is lost. Exinda has to be rebooted in order to restore network connectivity.
a) Bypass not working is the major symptom of the issue we are talking about here.
b) The log message [bypassd.NOTICE]: br2,br4,br6 (Fw8756) Bypass timer timed out: Essentially means that the bypassd process could not pat the watchdog timer for >8s. The watchdog timer for bypass does not really have anything to do with the NICS themselves. So this indicates that the process was deadlocked in the kernel for between 8s (bypass timeout) and 30s (system watchdog timeout)
ResolutionCurrently Exinda TAC is replacing the 4010 Exindas with equivalent Exinda models like 4062 and 3062 Exindas.
Before we do that there are some steps for troubleshooting and determining if this is the driver issue or not.
1. Manually put the bridge to bypass, see if it works well. If bypass works when NIC is set to bypass manually then the above symptoms could be a one time thing
2. Check the cables and speed and duplex settings on the Exinda.
3. Contact Exinda TAC and we will help with proceeding with a replacement.