Case Study: Tracking .NET Exceptions with Event Tracing for Windows (ETW)

In the past I have debugged customers code ,where the code throws tons of exceptions. This is a huge performance problem. Tess has amazing post on why throwing ton of exceptions are bad. To debug this I had to resort to using Windbg and  getting call stacks of exceptions. The biggest issue was ,there were thousands of exceptions thrown from different points which made debugging extremely hard. I couldn’t just break-point based on exception type because the same type of exception were thrown from n different points. Windbg would  peg cpu for getting call stacks for each of these exceptions. I so wish I had ETW when I had to figure out those exceptions. FYI all these exception were handled and that made even harder.

Now with CLR 4.0 having ETW, this is so much easy to diagnose the same problem. Here is the sample code that I am going to use.

using System;
using System.Threading;
class Program {
 private void ProcessArgs() {
 for (int i = 0; i < 20; i++) {
 try {
 throw new ArgumentNullException(i.ToString());
 }
 catch (Exception e) {
 Console.WriteLine(e);
 }
 }
 }
 private static void Main(string[] args) {
 var p = new Program();
 ThreadPool.QueueUserWorkItem((x) => p.ProcessArgs());
 ThreadPool.QueueUserWorkItem((x) => p.ThrowNullReference());
 Console.Read();
 }
 private void ThrowNullReference() {
 for (int i = 0; i < 20; i++) {
 try {
 throw new NullReferenceException(i.ToString());
 }
 catch (Exception ex) {
 Console.WriteLine(ex);
 }
 }
 }
}

Here is the code to trace CLR exceptions

xperf -start clr -on e13c0d23-ccbc-4e12-931b-d9cc2eee27e4:0x00008000:5 -f clrevents.etl

After which I started the console application. Then stopped the etw and then dumped the contents to a csv file.

xperf -stop clr
xperf -i clrevents.etl -o clrexceptions.csv

Here is a sample exception trace from ETW

Microsoft-Windows-DotNETRuntime/Exception /Start ,    3856446,        "Unknown" (27776),      30268,   0, , , , , "System.ArgumentNullException", "Value cannot be null.", 0x00480270, 0x80004003, 16, 15

Attached the application to Windbg. With in the etw trace is the Instruction Pointer where the exception was raised . And for above example it was raised at 0x00480270.

Issued a command to disassemble the sourcecode at the specific instruction pointer within Windbg

!u  0x00480270 

And here is the output from the above command

0:007> !u  0x00480270
Normal JIT generated code
Program.ProcessArgs()
Begin 00480200, size ad

C:\Users\naveen\Documents\Visual Studio 2010\Projects\ConsoleApplication1\Program.cs @ 4:
00480200 55              push    ebp
00480201 8bec            mov     ebp,esp
00480203 57              push    edi
00480204 56              push    esi
00480205 53              push    ebx
00480206 83ec30          sub     esp,30h
00480209 8bf1            mov     esi,ecx
0048020b 8d7dd8          lea     edi,[ebp-28h]
0048020e b906000000      mov     ecx,6
00480213 33c0            xor     eax,eax
00480215 f3ab            rep stos dword ptr es:[edi]
00480217 8bce            mov     ecx,esi
00480219 33c0            xor     eax,eax
0048021b 8945e8          mov     dword ptr [ebp-18h],eax
0048021e 894ddc          mov     dword ptr [ebp-24h],ecx
00480221 833d3c31310000  cmp     dword ptr ds:[31313Ch],0
00480228 7405            je      0048022f
0048022a e8464be768      call    clr!JIT_DbgIsJustMyCode (692f4d75)
0048022f 33d2            xor     edx,edx
00480231 8955d0          mov     dword ptr [ebp-30h],edx
00480234 c745d400000000  mov     dword ptr [ebp-2Ch],0
0048023b 90              nop

C:\Users\naveen\Documents\Visual Studio 2010\Projects\ConsoleApplication1\Program.cs @ 5:
0048023c 33d2            xor     edx,edx
0048023e 8955d8          mov     dword ptr [ebp-28h],edx
00480241 90              nop
00480242 eb4d            jmp     00480291
00480244 90              nop

C:\Users\naveen\Documents\Visual Studio 2010\Projects\ConsoleApplication1\Program.cs @ 6:
00480245 90              nop

C:\Users\naveen\Documents\Visual Studio 2010\Projects\ConsoleApplication1\Program.cs @ 7:
00480246 8d4dd8          lea     ecx,[ebp-28h]
00480249 e882330568      call    mscorlib_ni+0x2635d0 (684d35d0) (System.Int32.ToString(), mdToken: 06000cd4)
0048024e 8945cc          mov     dword ptr [ebp-34h],eax
00480251 b98c475968      mov     ecx,offset mscorlib_ni+0x32478c (6859478c) (MT: System.ArgumentNullException)
00480256 e8c51de8ff      call    00302020 (JitHelp: CORINFO_HELP_NEWSFAST)
0048025b 8945c8          mov     dword ptr [ebp-38h],eax
0048025e 8b55cc          mov     edx,dword ptr [ebp-34h]
00480261 8b4dc8          mov     ecx,dword ptr [ebp-38h]
00480264 e8b7bbfc67      call    mscorlib_ni+0x1dbe20 (6844be20) (System.ArgumentNullException..ctor(System.String), mdToken: 06000795)
00480269 8b4dc8          mov     ecx,dword ptr [ebp-38h]
0048026c e835ffd668      call    clr!IL_Throw (691f01a6)

C:\Users\naveen\Documents\Visual Studio 2010\Projects\ConsoleApplication1\Program.cs @ 9:
00480271 8945c4          mov     dword ptr [ebp-3Ch],eax
00480274 8b45c4          mov     eax,dword ptr [ebp-3Ch]
00480277 8945d0          mov     dword ptr [ebp-30h],eax
0048027a 90              nop

C:\Users\naveen\Documents\Visual Studio 2010\Projects\ConsoleApplication1\Program.cs @ 10:
0048027b 8b4dd0          mov     ecx,dword ptr [ebp-30h]
0048027e e841416268      call    mscorlib_ni+0x8343c4 (68aa43c4) (System.Console.WriteLine(System.Object), mdToken: 06000918)
00480283 90              nop

C:\Users\naveen\Documents\Visual Studio 2010\Projects\ConsoleApplication1\Program.cs @ 11:
00480284 90              nop
00480285 e8ae1fbc68      call    clr!JIT_EndCatch (69042238)
0048028a eb00            jmp     0048028c
0048028c 90              nop

C:\Users\naveen\Documents\Visual Studio 2010\Projects\ConsoleApplication1\Program.cs @ 12:
0048028d 90              nop

C:\Users\naveen\Documents\Visual Studio 2010\Projects\ConsoleApplication1\Program.cs @ 5:
0048028e ff45d8          inc     dword ptr [ebp-28h]
00480291 837dd814        cmp     dword ptr [ebp-28h],14h
00480295 0f9cc0          setl    al
00480298 0fb6c0          movzx   eax,al
0048029b 8945d4          mov     dword ptr [ebp-2Ch],eax
0048029e 837dd400        cmp     dword ptr [ebp-2Ch],0
004802a2 75a0            jne     00480244

C:\Users\naveen\Documents\Visual Studio 2010\Projects\ConsoleApplication1\Program.cs @ 13:
004802a4 90              nop
004802a5 8d65f4          lea     esp,[ebp-0Ch]
004802a8 5b              pop     ebx
004802a9 5e              pop     esi
004802aa 5f              pop     edi
004802ab 5d              pop     ebp
004802ac c3              ret

Now we see the call stacks of where the exception was raised, even without hooking for exceptions within the debugger.  The key reason for doing this is, for this specific case study I could have got all the unique Instruction pointers from the trace and just take one memory dump.With this I could have managed to get the call stacks all the exceptions. This would be non-invasive, which would save us lot of time and effort.

With ETW we can get all the exception even the ones that were handled, along with Instruction Pointer which helps us trace the root cause of the issue.

With this in hand ,we could easily write a small automation tool to extract the Instruction Pointer from the trace and then disassemble the source code and get the call stack.

All of this can be done on .NET 4.0 (clr.dll) and Silverlight (coreclr.dll).

In my forth coming posts ,I will share few more cool things that can be done with ETW.