SummaryExinda firmware 7.4.1 and earlier can possibly reboot when it is unable to correctly parse commands and other information necessary for the functionality
The Exinda firmware, as with all programs, requires the ability to parse text, that is to read through text and find what is necessary for its operations, stripping out what is not and discarding it. In some conditions, it is possible to see that the parsing mechanism in the Exinda fails, which can cause some performance issues and ultimately cause the appliance to reboot.
When dealing with this problem, messages such as the following will appear in the logs:
Mar 9 09:38:55 exinda kernel: [ 494.714272] Command error before '(null)' (line=1981 field=4): <28> Command is truncated
Mar 9 09:39:06 exinda kernel: [ 505.374703] Unable to find command for 'ad <undecipherable> line=2527 field=0)
Mar 9 09:39:06 exinda kernel: [ 505.470005] Command error before '(null)' (line=970 field=3): <28> Command is truncated
Where the number in brackets is the number of seconds since the last power on, and the undecipherable characters can appear as a mixture of anything - characters, symbols, question marks, etc.
CauseIn the background, the parser is working to try and obtain the information it needs to operate and do its functions from the input information. If, for some reason, the Exinda cannot parse the information properly, it goes into an errored state. While it tries to recover, sometimes it is not possible. If system resources are already allocated and expecting a result from something that has failed to be parsed, if will be empty (null). If that memory space is accessed or if it is attempted to be deallocated, it can run into a fatal error which will cause the device to reboot.
ResolutionIn ExOS version 7.4.2, this bug has been fixed. Upgrading to 7.4.2 will prevent this from occurring - in general, as well as the conditions that force the reboot.
Additional InformationIt is believed that this 'ad' command that it is trying to parse has some affect on the Active Directory Integration feature of the appliance. If running into problems with the AD Connector on v7.4.1 firmware and these errors are seen, please upgrade to v7.4.2 to rule out this as the cause of those problems.
If, on v7.4.2, this problem still occurs, please contact Exinda TAC for troubleshooting purposes.
Internal NotesThe following will appear in a kdump:
[ 4648.627863] Unable to find command for 'a \xffffff90\xffffffd7\xffffff88\xffffffff\xffffffffX\xffffffdb\xffffff90\xffffffd7\xffffff88\xffffffff\xffffffff' (line=1147 field=0)
[23147.409005] Command error before '(null)' (line=2139 field=3): <22> Failed to parse IPv4 address with mask
[23750.249561] Command error before '(null)' (line=1159 field=3): <28> Command is truncated
[43081.553456] Unable to find command for 'ad +\x04\xffffff88\xffffffff\xffffffff' (line=2353 field=0)
If the device reboots, the trace in the kdump:
[57985.651444] Call Trace:
[57985.654646] [<ffffffffa01e8d80>] ? cmd_parser_write+0x140/0x230 [proc_utils]
[57985.663092] [<ffffffffa01e8d80>] cmd_parser_write+0x140/0x230 [proc_utils]
[57985.671347] [<ffffffff811c5a13>] proc_reg_write+0x43/0x70
[57985.677946] [<ffffffff8116ae3f>] vfs_write+0xdf/0x1c0
[57985.684158] [<ffffffff8116b02f>] SyS_write+0x5f/0xa0
[57985.690272] [<ffffffff816aaaf9>] system_call_fastpath+0x16/0x1b
(Note that it is possible that the call trace could include cmd_parse at the top, ahead of cmd_parser_write)