Tuesday, January 25, 2011

How the OutputDebugString() API works

Tip - How the OutputDebugString() API works

Details - Hardcore Win32 developers are probably familiar with the OutputDebugString() API function that lets your program talk with a debugger.

When OutputDebugString() is called by an application, it takes the following steps.

1. Open DBWinMutex and wait until we have exclusive access to it.
2. Map the DBWIN_BUFFER segment into memory. If it’s not found, there is no debugger running so the entire request is ignored.The DBWIN_BUFFER when present, is organized like this structure. The process ID shows where the message came from, and string data fills out the remainder of the 4KB. By convention, a NULL byte is always included at the end of the message.

struct dbwin_buffer
  DWORD dwProcessId;
  // Total size must be 4KB ( dwProcessID + data )
  char data[4096-sizeof( DWORD )];

3. Open the DBWIN_BUFFER_READY and DBWIN_DATA_READY events. As with the shared memory segment, missing objects mean that no debugger is available.
4. Wait for the DBWIN_BUFFER_READY event to be signaled, this says that the memory buffer is no longer in use. Most of the time, this event will be signaled immediately when it’s examined, but it won’t wait longer than 10 seconds for the buffer to become ready (a timeout abandons the request).
5. Copy up to about 4KB of data to the memory buffer, and store the current process ID there as well. Always put a NULL byte at the end of the string.
6.Tell the debugger that the buffer is ready by setting the DBWIN_DATA_READY event. The debugger takes it from there.
7. Release the mutex
8. Close the Event and Section objects, though we keep the handle to the mutex around for later.

If we want to watch the strings outputted using OutputDebugString(), from the viewer application side it’s a bit simpler. The mutex is not used at all, and if the events or shared memory objects already exist, we presume that some other debugger is already running. Only one debugger can be in the system at a time.

1. Create the shared memory segment and the two events. If we can’t, exit.
2. Set the DBWIN_BUFFER_READY event so the applications know that the buffer is available.
3. Wait for the DBWIN_DATA_READY event to be signaled.
4. Extract the process ID and NULL-terminated string from the memory buffer.
5. Go to step #2

Unlike most of the Win32 API, the native version of OutputDebugString() is ASCII. So OutputDebugString() ultimately passes data to the debugger in the memory buffer strictly as ASCII. This suggests that for sending a quick message even in a Unicode program, it can be done by calling the OutputDebugStringA() version directly.

Reference - http://sumeshvv.wordpress.com/2010/12/02/how-the-outputdebugstring-api-works/

Posted By :Sumesh V V

No comments:

Post a Comment