# The Hitchhikers Guide to Windows Error Reporting

### What is Windows Error Reporting(WER)?

Windows Error Reporting is a feature Microsoft introduced in Windows XP that allows the Operating System to notify Microsoft of Application faults. The feature has evolved to allow third party enterprises to set custom reporting servers, and application developers to build out custom reports for their applications. It is tightly integrated into the Windows Exception handling mechanism for both user mode and kernel mode components.

WER contains several components working together in the OS. It starts with an application crash, then a series of events occur:

1. WER service is started
2. WER determines the cause of the crash and how to process it
3. WER gathers relevant information from the faulted process
4. WER prepares a report of the crash
5. WER queues this report
6. WER sends the report to a remote location

A lot happens between 1-6 but we will not be focusing on the entire architecture of WER. To set expectations, we'll mostly be focused on numbers 1-3.

If you want to get a head start and understand whats going to be discussed here today you can refer to the MSDN documentation of WER [here](https://docs.microsoft.com/en-us/windows/win32/wer/windows-error-reporting)

#### What we will not discuss
WER is a very involved process that goes through several layers of modules and services. What we will not discuss here today are the features that involve reporting generation of faults, and setting up corporate report servers.

Anything related to the networking portion of WER we'll willfully ignore, however reference material will be provided for the curious.

WER is also used to perform Kernel Level dumps but we will not go over this as well.

#### What we will discuss
We will discuss the internals of how the WER service and third party applications communicate. We will also go over some of the features WER provides application developers that can change the behavior of WER. Exception Handling will be reviewed as it is the foundation from where WER starts.

We'll take special interest into how the out-of-process manipulation of a process is accomplished with WER.

During this journey we'll also identify some undocumented behaviors and reveal some of the interesting internals of how the WER service operates with its client components.

Our example will assume a 64bit OS (Windows 10 1909) and 64bit application, there are some unique differences between 32bit and 64bit applications with regards to Exception handling which we will not discuss. Besides, 32bit is on its way out anyways :)

To conclude we'll go over what we did not get to deep dive into and offer some ideas for possible future research.

### An Overview of the WER architecture

Before we begin lets familiarize ourselves with an overview of how the WER process looks.


We'll discuss more in depth most of the steps involved in the above flow graph. Let's review the steps:

#### Step 1
An Application crashes and raises an Exception

#### Step 2
The Windows Exception Handling mechanism takes over and determines whether WER will be instantiated.

The Windows Exception Handling mechanism determines the fault,
and performs a series of logical checks to see if the process is critical, an AppContainer, needs to fail fast (exit immediately) and if WER needs to be invoked based on the exception. A lot more happens here, but we'll discuss only what applies to WER.

The WER Service is then initialized by the usermode native layer (ntdll).

#### Step 3
The WER Service is initialized. It identifies the faulting process and what kind of crash will be reported and spawns the appropriate Werfault.exe or wermgr.exe executables.

The WER Service starts the ETW Logging interfaces.

The WER Service also establishes the ALPC Port Communication Channel for the WER clients (werfault.exe).

#### Step 4
The WER Service then launches the client werfault.exe or wermgr.exe. Werfault.exe is used to perform a snapshot of the faulted application.

#### Step 5
The service will then continue to process its input and subsequently load and initialize the WER reporing facilities and load yet another Werfault.exe, this time with the appropriate security context for the faulting process.

#### Step 6
The Werfault.exe will now communicate accordingly with the Wer service and determine what needs to be reported and how based on configurations set by the user and application.

### Components of WER

There are several components involved with Windows Error Reporting. This section will provide a quick guide as to what component houses what functionality:

* **wersvc.dll**: The WER Service implementation. This is executed within an svchost executable and facilitates the reporting and dumping process.
* **werfault.exe**: The WER Client application that handles the usermode applications and is the main client for the WER process.
* **wermgr.exe**: The WER client involved with writing and uploading WER reports.
* **wer.dll**: Implements a subset of the WER reporting and dumping features.
* **faultrep.dll**: This DLL exposes functions for Reporting hangs in a user application, and setting exclusions and other Application specific settings.
* **werui.dll**: Handles Graphical User Interface application hangs and reporting.

The system DLLs `kernel32` and `ntdll` also play a role in initializing WER and we'll go over those in the coming paragraphs.

There are several other components such as the control panel items for WER settings and other Operating System utilities like the registry but they won't be discussed thoroughly here.

### Exception Handling Review

In order to understand WER we'll need to understand the basics of exception handling in Windows Applications; this will help us understand why and when WER gets involved. We'll start with the most basic example of an exception. The examples are specific to 64bit applications; 32bit exception handling is slightly different and not covered.

#include <windows.h>


int main(int argc, char** argv)
{

	std::print("Raising Error\n");

	RaiseException(EXCEPTION_FLT_DIVIDE_BY_ZERO, 0, 0, NULL);
	
	std::getchar();

	return 0;
}

> This application throws a division by zero exception allowing the exception to be continuable (second parameter of `RaiseException` API). It uses the `RaiseException` API, however, most exceptions are thrown due to some undesired behavior. Ultimately, `RaiseException` is called to transfer control to the Exception handling mechanisms.
>

If we debug this application and follow the execution of `RaiseException` this is what we understand:

`RaiseException` prepares a properly formatted `EXCEPTION_RECORD` structure and passes it to `RtlRaiseException`. The Exception Record contains the Exception code and location the exception occurred.

typedef struct _EXCEPTION_RECORD {
  DWORD                    ExceptionCode;
  DWORD                    ExceptionFlags;
  struct _EXCEPTION_RECORD *ExceptionRecord;
  PVOID                    ExceptionAddress;
  DWORD                    NumberParameters;
  ULONG_PTR                ExceptionInformation[EXCEPTION_MAXIMUM_PARAMETERS];
} EXCEPTION_RECORD;

> It's worth noting that you can use RaiseException to raise an exception with any information you'd like, from the exception code, flags, address and parameters, for example you can choose to raise an access violation on a variable g_data:
>

DWORD numArgs = 2;
const ULONG_PTR ulArgs[2] = {
	1, // WRITE violation
	(ULONG_PTR)&g_data,
};
RaiseException(EXCEPTION_ACCESS_VIOLATION,
	        0, numArgs, ulArgs );

> Args here identifies what kind of access violation ocurred, 0 for read and 1 for write and the location of the exception.
>

`RtlRaiseException` captures the current Context and saves it to return to later (if applicable).

Next, it attempts to locate the Exception Handler for the offending Instruction Pointer through a table-based lookup process. This process is specific to 64bit applications. Refer to [[osronline]](http://www.osronline.com/article.cfm%5Earticle=469.htm), [[Exception Handling]](https://revers.engineering/applied-re-exceptions/) and [[msdn]](https://docs.microsoft.com/en-us/cpp/build/exception-handling-x64?view=vs-2019) for more information on that process.

Then it checks for the presence of a debugger, if present the exception is passed to the debugger first via `ZwRaiseException`. *This is typically referred to as the first chance exception for debuggers and exception handlers.*

Finally, execution is then passed to `RtlDispatchException`.

The screenshot below shows the stack at the time `RtlDispatchException` is about to be invoked, showing that the first parameter contains the `EXCEPTION_RECORD` and the first member `ExceptionCode` being our DIVISION_BY_ZERO exception code `#define STATUS_FLOAT_DIVIDE_BY_ZERO ((DWORD )0xC000008EL)`.


`RtlDispatchException` is responsible for dispatching the exception to the appropriate handlers, this process includes:
* Invoke Vectored Exception Handlers
* Invoke second chance exception for a debugger to handle if applicable
* Determines the Exception Handler for the Exception (yes, again but more involved)
* Calls a series of default exception handlers if present via the `ntdll.RtlpExecuteHandlerForException` function:
* kernel32.GSHandler
* Applications C_default_handler
* ntdll's default exception dispatcher
* Unhandled Exception filter in kernel32 **

To put this is into a summarized form we can choose to say this:

*A first chance exception occurs, then a second chance exception occurs, then the exception is considered unhandled and passed to `kernel32.UnhandledExceptionFilter`*

For the usermode portion of exception handling this is pretty much all we need to know for now. WER is transferred control during the `kernel32!UnhandledExceptionFilter` invocation*.

*Note: If you intend to debug the exception handling mechanism through a debugger, the debugger must be hidden from the process or you must pass the exception to the application at every chance*

### Where Windows Error Reporting fits in
Now that we understand the basics of how an exception is dispatched in usermode, where does WER fit into all this?

WER is an optional (opt-in) feature that allows application developers and Microsoft to gather information when applications fail. WER itself can be disabled entirely as a feature, or fine tuned to contain as little or as much information about the Application and the Operating System as possible upon failure of either.

We can disable and enable WER altogether through the [registry directly](https://www.lifewire.com/how-do-i-disable-error-reporting-in-windows-2626074), via policy management consoles, or simply through [Powershell cmdlets](https://docs.microsoft.com/en-us/powershell/module/windowserrorreporting/?view=win10-ps).
We can also programamatically opt out of WER at runtime using the `SetLastError()` API call.

With that being said, WER is invoked when none of the aforementioned exception handlers have handled the exception/fault.
WER was specifically designed to report this exception to Microsoft (or other enterprise reporting server).

*WER is invoked at the end of exception unwinding when `kernel32.UnhandledExceptionFilter()` is called.*

#### Unhandled Exception Filter to WER
We now know that WER is transferred control through the `UnhandledExceptionFilter()` routine, let's see this process at a high level.


We've focused mainly on the decisions that may directly affect WER invocation. Inside of the UnhandledException there are several logical checks that determine whether WER will be invoked.

The first is a check to see if the current exception is equal to the fast fail exception error code. Applications can call `RaiseFailFastException` in order to bypass all exception handling mechanisms described earlier. It does so by preparing the proper EXCEPTION_RECORD, signaling the WER Service directly, then terminating the process.

If FastFail is not the case, it checks for a debugger and calls `Debugbreak()` if applicable.

The current threads Error Mode is then checked. An application can choose to omit itself from WER via a call to `SetErrorMode(SEM_NOGPFAULTERRORBOX)`, in which case the Exception is dispatched and the process is terminated.

If these conditions aren't met, then WER is invoked.

#### WER Initialization

The initializaion of WER begins in kernel32's `WerpReportFault` function. This function sends a signal to the kernel via `NtSetSystemInformation` with the `0xB1` (*SystemWin32WerStartCallout*) SystemInformation Parameter.
This internally calls out to an index (0x20) into the win32k Dispatch table. Presumably this is for the werui portion of WER to recover a hung GUI application but would need further research to confirm this.

Then `kernel32.WerpReportFaultInternal()` is invoked which takes an `EXCEPTION_RECORD` pointer as its argument.

This function is rather large and contains several key pieces of information that's later used in other WER facilities we'll see later. Whats important for initialization is that it prepares a table of information to pass to `ntdll!ReportExceptionInternal()`.
Among the data sent is the current Process ID, a handle to a Mapped Section Object and a Handle table containing the Thread handles of the current process.

`ReportExceptionInternal` is responsible for starting and querying the WER Service. A partially reverse engineered pseudocode is pasted below:

__int64 __fastcall ReportExceptionInternal(DWORD ProcessId, HANDLE FileMappingPtr, const void *HandleTableArray, unsigned int HandleTableArraySize, unsigned int a5, _QWORD *a6)
{
  __int64 size; // rbx
  __int64 result; // rax
  int nMessageId; // [rsp+20h] [rbp-E0h] BYREF
  _DWORD v12[349]; // [rsp+24h] [rbp-DCh] BYREF
  _QWORD v13[176]; // [rsp+5A0h] [rbp+4A0h] BYREF

  size = HandleTableArraySize;
  *a6 = 0i64;
  if ( HandleTableArraySize > 5 )
    return 0xC000042Bi64;
  memset(v13, 0, 0x578ui64);
  v13[6] = __PAIR64__(ProcessId, a5);
  LODWORD(v13[0]) = 0x5780550;
  LODWORD(v13[5]) = 0x20000000;
  v13[7] = FileMappingPtr;
  if ( HandleTableArray && (_DWORD)size )
    memmove(&v13[8], HandleTableArray, 8 * size);
  memset(v12, 0, sizeof(v12));
  nMessageId = 0x5780550;
  result = SendMessageToWERService((__int64)v13, (__int64)&nMessageId);
  if ( (int)result >= 0 )
  {
    if ( (_DWORD)result == 258 )
    {
      result = 0xC0000240i64;
    }
    else
    {
      *a6 = *(_QWORD *)&v12[11];
      result = 0i64;
    }
  }
  return result;
}

This function prepares a message to send to the WER Service via the `SendMessageToWERService()` function. `SendMessageToWERService()` takes two arguments, the first is a pointer to a user-supplied `WERSVC_MSG` buffer, and the second is the Message ID, which in this case is `0x5780550`. We can see that the `WERSVC_MSG` is a rather large data structure of 0x578 bytes.

## Initializing the WER Service

The WER Service (WERSvc) is a SYSTEM level service implemented in the `wersvc.dll` module and runs within the svchost.exe process context.

`SendMessageToWERService()` initiates first contact with the WERSvc through an undocumented procedure related to the Windows Notification Facility. Ntdll queries the state of the WERSvc through `ntdll.ZwQueryWnfStateNameInformation()`, then starts the service by sending it the WNF_WER_SERVICE_START UUID via `ntdll.NtUpdateWnfStateData()`. Along with starting the Service, it opens an Event Object named `SystemErrorPortReady`, which the WERSvc uses to send a notification that it has initialized successfully. Then WERSvc opens an IPC channel via an ALPC Port named `\\WindowsErrorReportingServicePort`. Through this port many of the WER clients will send messages to the WERSvc.

Now that the Windows Error Reporting Service has started, and it contains information about the faulted process through the `WERSVC_MSG` structure it was sent, it can continue the reporting process.

##### Processing IPC messages

Once the LPC Server is spawned it listens for messages through the ALPC Port and processes them inside of the `_ProcessRequest()` function. It's worth noting that this server supports a number of messages and this presents future research opportunities, Specifically the `CheckIfValidPortMessage()` and `DispatchPortRequestWorkItem()` functions which validates and dispatches the received messages respectively. A handwritten pesudocode version of `_ProcessRequest()` is shown below, removing alot of the non essential telemetry.

bool CWerService::_ProcessRequest(CWerService *this, WERSVC_MSG* WerMsg, int Arg3)
{
    bool result = true;

    if(!CWerService::CheckIfValidPortMessage(this, WerMsg))
    {
        EtwEvent();
        result = false;
    }

    WERSVC_MSG msg = new(0x580);
    this->WerMsgPtr = msg;
    memcpy(this->WerMsgPtr, WerMsg, sizeof(WERSVC_MSG)) // size = 0x578

    if(!TrySubmitThreadpoolCallback(CWerService::StaticDispatchPortRequestWorkItem,
    this->WerMsgPtr, *(this*)+3))
    {
        delete msg;
        result = false;
        //GetLastError and set result failure
    }

    return result;
}

Internally the messages are disaptched by `CWerService::DispatchPortRequestWorkItem()`. In order to stay on track let us focus now what occurs when an application faults.

##### WerFault.exe and Snapshots

As previously mentioned, the WerSvc dispatches work items via its `CWerService::DispatchPortRequestWorkItem()` function. When an application faults one of the attempts made is to create a Snapshot of the process.

Process Snapshots were introduced in Windows 8.1 as a means to provide developers a more robust and stable way to fork and query another processes' current state, it is an evolution and improvement on the old ToolHelp API's.

The WERSvc will attempt to acquire a snapshot of the faulted process by invoking `CWerService::TryCaptureSnapshot()`. It does so by spawning a WerFault.exe process with predefined arguments. In particular the argument `-pss` indicates to WerFault.exe to attempt a Process snapshot. A snippet is shown below:

   result = StringCchPrintfW(ApplicationName, 0x104ui64, L"%ws\\%ws", Buffer, L"WerFault.exe");
    if ( (int)result < 0 )
      return result;
    v14 = OpenProcess(0x1040u, 0, a3);
    if ( v14 )
    {
      TargetHandle = 0i64;
      v15 = GetCurrentProcess();
      if ( DuplicateHandle(v14, a4, v15, &TargetHandle, 6u, 1, 0) )
      {
        v17 = StringCchPrintfW(
                CommandLine,
                0x104ui64,
                L"%ws %ws %ws %llu %ws %u %ws %u",
                ApplicationName,
                L"-pss",
                L"-s",
                TargetHandle,
                L"-p",
                a2,
                L"-ip",
                a3);
        if ( v17 >= 0 )
        {
          memset_0(&StartupInfo, 0, 0x70ui64);
          StartupInfo.cb = 112;
          ProcessInformation.hProcess = 0i64;
          ProcessInformation.hThread = 0i64;
          *(_QWORD *)&ProcessInformation.dwProcessId = 0i64;
          StartupInfo.dwFlags = 1;
          StartupInfo.lpDesktop = (LPWSTR)&pActivityId;
          StartupInfo.wShowWindow = 0;
          v19 = CreateProcessW(
                  ApplicationName,
                  CommandLine,
                  0i64,
                  0i64,
                  1,
                  0x400u,
                  0i64,
                  0i64,
                  &StartupInfo,
                  &ProcessInformation);
          CloseHandle(v14);
          if ( v19 )
          {
            v21 = WaitForSingleObject(ProcessInformation.hProcess, 0xFFFFFFFF);

>Like many other aspects of WER, allowing snapshots can be configured via the registry or directly in code. For example you can disable Snapshots for certain applications under the `Software\\Microsoft\\Windows\\Windows Error Reporting\\Debug` Key, with a value of `DisableSnapshots` set to 1.
>

WerFault.exe creates the Snapshot, adjusts the snapshot tokens security, then saves the handle for later. These handles are later used to create a minidump of the faulted application if configuration allows.

##### WerFault and Security Context
After the snapshot is taken, the WERSvc will identify the current process' security context. It does this in order to launch a new WerFault.exe that follows the principle of least privilege and offers it just enough permissions to read and dump the memory of the faulted process.

This is implemented inside of `faultrep.dll`. After WERSvc calls `CWerService::TryCaptureSnapshot()` it invokes `CWerService::TryReportCrash()`. This function eventually calls to `faultrep.WerpInitiateCrashReporting()`. Here, WER identifies the currently logged on user and Impersonates it, then identifies the current process' Token level and appropriately spawns a new Werfault.exe, passing the `-u` parameter to signify this is a usermode crash. Other parameters include the process ID and a handle to a mapped view.

Werfault.exe invokes its `UserCrashMain()` function which will make another call to `faultrep.WerpInitiateCrashReporting()` to initiate the reporting of the faulted application. This process includes creating the directories that the reports go into and gathering the runtime information for those reports.

#### Reporting and Communicating with the Wer Service

We've gone over the main components involved with instantiating WER and subsequently preparing the faulted process for reporting. However, who is responsible for the actual dumping and reporting of the faulted process? This is done via the IPC mechanism we described earlier, the ALPC Port created by the WER Service.

The different components of WER all implement a similar function called `WersvcSendMessage()`. You'll find it in werfault.exe, wer.dll, and faultrep.dll. This function is used when some action that requires elevated permissions needs to be taken, for example, preparing the report directories for the application, querying regsitry keys for configurations, and other elevated tasks. The screenshot below shows the cross referenced functions for `WersvcSendMessage()` inside of the wer.dll module.


This function queries the state of the WER Service, starts it and sends messages using the `NtAlpcSendWaitReceivePort()` API.

The messages are then dispatched by the `wersvc.CWerService::DispatchPortRequestWorkItem()` function.

#### A note about Security Permissions Handling

WER manages to accomplish its task of transferring control to several components that requires different levels of permissions by utilizing a common practice in Windows. Services usually expose interfaces for user mode applications and (IPC mechanisms) utilize Thread impersonation and Discretionary Access Control Lists (DACLs) when creating file system artifacts. Of course, designs like these lend themselves to Privilege Escalation attacks and the attack surface is large with regards to WER. In fact, such an attack was publicized not too long ago regarding the reporting directories here, [Directory Deletion to SYSTEM shell](https://secret.club/2020/04/23/directory-deletion-shell.html).

### Pulling it all together

We've gone over how things work when a simple exception is raised and when and how WER is initialized. We've briefly discussed the IPC mechanism used to dispatch work items for elevated tasks and the different components involved during the Error Reporting process. Next we'll go over some of the ways applications can configure WER and how this affects the reporting process and introduces new attack surfaces and ideas.