Versions / Builds Affected
20141117, 20150218Status
OpenTT / JIRAID
2976How to Identify
The customer might report the following generic symptoms:
- marc.store.exe (the Store service) is using up all available memory
- marc.store.exe (the Store service) crashes
- the event logs contain generic crash entries for such crashes
- other processes (even non-GFI processes) might crash as a side effect of the high memory consumption
Identifying this particular known issue is not straight forward. In order to do so, here is some background:
- the folder structure of a mailbox in GFI Archiver is stored across all archive stores
- when the folder structure of a mailbox in GFI Archiver is requested (for example by the Outlook Connector or when viewing the mailbox in "folder view" under the archive tab) marc.store.exe queries all DBs to retrieve the folder structure for the mailbox from each of them
- this data then must be consolidated and processed in memory before it is being passed back to the client module which requested it
- at this consolidation processing, some kind of loop must occur which causes the high memory consumption
Note that:
- this is not a common situation
- so far it was only the data from a single mailbox triggering this situation - and not multiple mailboxes
- the issue is triggered when a client requests the folder structure of the particular user (for example, the user runs the Outlook Connector which does such a request every 5min by default or the user opens the web page and views their mailbox in "folder view")
To identify this known issue debug logs and event logs are needed which cover multiple crashes. Check for the following:
PART - A
1. Check the event logs and note down the time stamps of the the last few crashes of marc.store.exe
2. Open: ..\Store\DebugLogs\AttendantService.log
3. Look for crash entries similar to the one below
2015-02-16,09:15:59,854,1,"#0000233C","#0000002F","error ","Attendant Service","Error: CRITICAL Unhandled error, terminating:True"
2015-02-16,09:15:59,854,1,"#0000233C","#0000002F","error ","Attendant Service","Error details: System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.
; at System.String.Concat(String str0, String str1)
; at Store.Dal.Elements.Folder.ReplaceParentPath(String oldPath, String oldValue, String newValue)
; at Store.Core.EmailMessageRetrieval.ListAllFolders(String userGuid, Boolean filesOnly)
; at SyncInvokeListAllFolders(Object , Object[] , Object[] )
; at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs)
; at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)
; at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)
; at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage31(MessageRpc& rpc)
; at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)
; at System.ServiceModel.Dispatcher.ChannelHandler.DispatchAndReleasePump(RequestContext request, Boolean cleanThread, OperationContext currentOperationContext)
; at System.ServiceModel.Dispatcher.ChannelHandler.HandleRequest(RequestContext request, OperationContext currentOperationContext)
; at System.ServiceModel.Dispatcher.ChannelHandler.AsyncMessagePump(IAsyncResult result)
; at System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
; at System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)
; at System.ServiceModel.Channels.TransportDuplexSessionChannel.TryReceiveAsyncResult.OnReceive(IAsyncResult result)
; at System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
; at System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)
; at System.ServiceModel.Channels.SynchronizedMessageSource.ReceiveAsyncResult.OnReceiveComplete(Object state)
; at System.ServiceModel.Channels.SessionConnectionReader.OnAsyncReadComplete(Object state)
; at System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
; at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
; at System.Net.Security.NegotiateStream.ProcessFrameBody(Int32 readBytes, Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
; at System.Net.Security.NegotiateStream.ReadCallback(AsyncProtocolRequest asyncRequest)
; at System.Net.FixedSizeReader.CheckCompletionBeforeNextRead(Int32 bytes)
; at System.Net.FixedSizeReader.ReadCallback(IAsyncResult transportResult)
; at System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)
; at System.ServiceModel.Channels.ConnectionStream.IOAsyncResult.OnAsyncIOComplete(Object state)
; at System.ServiceModel.Channels.PipeConnection.OnAsyncReadComplete(Boolean haveResult, Int32 error, Int32 numBytes)
; at System.ServiceModel.Channels.OverlappedContext.CompleteCallback(UInt32 error, UInt32 numBytes, NativeOverlapped* nativeOverlapped)
; at System.Runtime.Fx.IOCompletionThunk.UnhandledExceptionFrame(UInt32 error, UInt32 bytesRead, NativeOverlapped* nativeOverlapped)
; at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)"
4. Note down the thread ID from the crashing thread - here it was: #0000002F
5. Open Store\DebugLogs\EmailMessageRetrieval.log & bak
6. Search through the file and only consider lines with the same thread ID as found earier (#0000002F)
7. The last line should be a >>ListAllFolders call and it should have occurred a little while before the crash - like:
2015-02-16,09:13:08,910,1,"#0000233C","#0000002F","info ","EmailMessageRetrieval",">>ListAllFolders userGuid: 44f624cdc786f34ebb175f6e0452bdfd;filesOnly: False;"
8. Keep a record of the userGuid: 44f624cdc786f34ebb175f6e0452bdfd
9. Go through the steps above once again - but for another crash occurrence
10. If you find that the last line in the thread was also a >>ListAllFolders call and it did refer to the very same userGuid, then this is a strong indication that the customer is affected by this known issue
PART - B
1. Lookup the which user belongs to the userGuid found earlier: 44f624cdc786f34ebb175f6e0452bdfd (this can be done easily by opening ..\Core\DebugLogs\usercntmaad.txt and searching for the GUID (44f624cdc786f34ebb175f6e0452bdfd)
2. Ask the customer to disable the Outlook Connector for this very user and also to ensure that this user does not open their mailbox in "folder view" under the archive tab
3. If only a single user is triggering the issue, then the crashes should stopWorkaround / Fix Details
PATCH FOR 2015 SR1 BUILD 20150218
http://ftp.gfisoftware.com/patches/ARC2015/20150218/ARC2015_SR1_PATCH_20150713_2976.zip
WORKAROUND
Have the user with the mailbox which was identified earlier stop using the Outlook Connector (disable or uninstall it) and ensure that they don't use the archive tab in folder viewRequired Actions
Apply patch
