Ring 0 debugging and Windbg – part 1

In this article I talk about debugging in ring 0 in the world of Windows(TM). I discuss some tricks which should help you with debugging. Kernel Debugging has become easier but a lot is not very clear and may require tricks to get it to work. Sigh!!!

 I start off illustrating how the basic transports used by Windbg work and then troubleshooting them. I discuss one topic on which I had to scrounge for information which has to debugging the boot environment.

It is not my intention to show you the basics of kernel debugging. Please refer to the Windbg documentation for that. Also, I am not looking at NET debugging (debugging over the network) which is probably going to be available in next version of the OS (Windows 8). There are few more interesting scenarios like accessing and changing registry values through the debugger which I shall deal with in the next article in this series and try and make a last ditch effort to save your machine from crashing components. I will also show how to debug Virtual machines and one of the lesser known facts of Virtual machine to virtual machine debugging. I had to set this up for a self paced class room where people could had access to VM’s only so that once the student was done both the VM’s for rolled back to the original state for the next one. And finally single machine debugging and what it can be used for.

You can use bcdedit (command line) to enable kernel debugging (vista and forward) or msconfig (GUI) Windows XP and forward to enable debugging. The following link shows how to enable debugging for various transports. I want to emphasize however there maybe other ways of doing the same things and please don’t hesitate to add that to comments if you know of other simpler ways.

Vista and forward — http://msdn.microsoft.com/en-us/library/aa906219.aspx

or

Pre-vista  — http://support.microsoft.com/kb/289022

1. Serial

 

Troubleshooting

If you can’t get serial debugging to work look for the COM port under device manager. If that doesn’t show up then either check your BIOS to see if the COM port is enabled. Sometimes legacy USB support in the BIOS causes this. Disable it in the BIOS.

In case you can’t  still get Windbg to work, check if you set the debugger machine as a debuggee in the past. Also, if you set the debugee as the debugger that can interfere with the claim to the COM Port. My test machines were being used to repro a problem so I used my second dev machine as the test machine and enabled debugging on it. But once I was done I wanted to use the dev machine as the host again.

I just couldn’t get Windbg to work. The solution in this case turned out to be to turn off debugging.  On Vista and forward you could do that by using the following command:

bcdedit /debug off or use msconfig to disable debugging. This requires a reboot to take effect.

2. Debugging with 1394

Lets start of by discussion some of the common questions one has to deal with once starting with 1394 debugging. Hardware!!

1394 Cards you can use – If you don’t have a 1394 port on your laptop, you can buy a 1394 Express card or a cardbus card which cost around $40. On a desktop you could buy a PCI 1394 card too which maybe cheaper. 1394 a/both b work fine. Express cards and some PCI cards come with multiple ports/slots which you could use to connect to multiple debuggee’s using different channels.

 

 

When you buy these cards make sure you order the right cable.

4-4 cable

 Ideal when you are debugging laptop to laptop with built in 1394 ports

 

 

 

 

 

 

6-4 cable

Especially if you have a 1394 express card or card bus card on one of the end.

 

6-6 cable

 Ideal if you have express /card bus cards on both ends

Multiple 1394 cards

If you have multiple 1394 cards on your target how do you pick which 1 to use for kernel debugging. It  is and maybe if the failure on the system as a target is intermittent

Steps to get dual 1394 controllers working on a single PC, where one controller is used to target the PC with KD, and the other controller is left free to target other systems, or to perform other non-KD 1394 tasks.

These steps explain how to dedicate one 1394 controller to be used to target the main PC. Once these steps are taken, any other 1394 controllers should be available to allow the main PC to target other systems for KD debugging.

 

1.  Open Device Manager.

2 .  Locate the IEEE 1394 node, and expand it. You should see two 1394 controllers. They should both be functioning correctly without any errors. If there are errors (i.e., yellow caution in devmgr or other errors), troubleshoot those first before proceeding.

1. For each controller, obtain its PCI location information as follows:

 a) Right-click and choose properties,

 b) Click on the Details node,

 c) Select “Location information”,

 d) Write down or note the information.

For example, if you see “PCI bus 21, device 0, function 1” write down 21.0.1. For this example, 21.0.1 will be used for one controller, and 49.0.0 will be used for the other. 21.0.1will be used to allow the main PC to be targeted by another host, and 49.0.0 will be used to allow the system with the two 1394 controllers to act as host, targeting other 1394 targets.

Note: The numbers for location should be in decimal, not hexadecimal.

2. Determine which controller you want the local system to use for its KD (as target), and issue something close to the following bcdedit commands:
a) bcdedit /dbgsettings 1394 channel:25
b) bcdedit /set {current} loadoptions busparams=21.0.1
This effectively makes it so the main PC with the two 1394 controllers capable of being targeted by a host, where that host will specify 1394 channel 25. The main PC controller used to handle this is 21.0.1.
After making the above bcdedit changes, reboot so they take effect.

3. You should be able to target other 1394 “targets” from the same PC.  In this case, 1394 controller 21.0.1 is used to target the PC within which it is installed, while 1394 controller 49.0.0 is available to target other 1394 targets.

Failure to open the 1394 channel on the  * host* error

Using 1394 for debugging

Failed to open 1394 channel 1.  Error0x80070002.

The 1394 debug driver may not be installed.

Attempting 1394 debug driver installation now.

One of the main cause is that 1394 debugging is already enabled on the box you are trying to use as a host i.e. it already acts as a target.

Solution:

• Run an elevated cmd prompt and type “bcdedit -debug off”. This should disable the machine from acting as a target and allow the host 1394 debug driver to take over control.
• Reboot and you should be good to go.

 Disabling the 1394 Host controller

For kernel debugging to work properly with a IEEE 1394 (FireWire) connection, you might have to disable the 1394 host controller on the target computer or disable the 1394 network adapter on the host computer. The required adjustments vary depending on the operating system on the target computer.

Role	OS	Action to make it work	Reason	Caution
Target	Srvr 2k3 (no ServicePack)XP Sp1	Disable the 1394 host controller	If the 1394 host controller remains enabled on computers that are running these operating systems, the core 1394 stack might compete with the 1394 debugging connection for resources. This competition might cause the debugging connection to fail or to be lost shortly after boot time	On Windows Vista (any version), Windows Server 2003 with Service Pack 1 (SP1), and Windows XP with Service Pack 2 (SP2), do not disable the host controller on the target computer. In fact, disabling the host controller while running these operating systems can prevent you from debugging the system during some power state changes.Do not disable the host controller on the host computer under any circumstances, regardless of the operating system.
Host	Pre-Windows Vista	Disable the 1394 network adapter	Disabling the 1394 network adapter can resolve issues that result from bus resets that are caused by loading the 1394 network adapter driver.	Do not disable the 1394 network adapter if you are using IP1394. This method does not apply to Windows Vista, because it does not support 1394 network adapters.

 

To disable the 1394 host controller on the target computer

1. On the target computer, open Device Manager (that is, click Start, click Run, and then enter Devmgmt.msc) and isable the 1394 contrller.
2. Restart the target computer.

NOTE: Be careful not to accidentally disable the 1394 network adapter.

To disable the 1394 network adapter on the host computer

1. On the target computer, open Device Manager (that is, click Start, click Run, and then enter Devmgmt.msc).
2. Expand Network Adapters, and locate 1394 Net Adapter.
3. Right-click 1394 Net Adapter, and then click Disable.
4. Restart the target computer.

NOTE: Before performing kernel debugging over a 1394 cable, you must also install the 1394 Virtual Driver on the host computer.

If you see Windbg returning errors about connection problems (Ctrl-Alt-D for verbose output), you can plug out and plug in the cable or the card. Even on a crashed target this does seem to resolve the problem a lot of the times.

Some myths about 1394 debugging.

MYTH: Only channel 44 can be used for debugging.

Not really!! This comes from the example shown in Windbg. Any port from channel 1 to channel 63 works including 13 J. Are you lizening David? J

MYTH: I can use Virtual COM over USB on a target machine for kernel debugging if it doesn’t have 1394 or serial ports

No it doesn’t work. Try USB debugging.

NOTE: Remember you need to start debugging on 1394 host with elevated privileges since it does involve installing the 1394 debug driver in the Windbg package on the host machine.

3.  Debugging Usb

Kernel debugging over USB works well if the target computer is running Windows Vista or later and the host is running Windows (XP+ preferred). However, it has some additional requirements on hardware.

Hardware/BIOS requirements

1. Cable

A USB 2.0 debug cable. This cable is not a standard USB 2.0 cable, because it has an extra hardware component that makes it compatible with the USB2.0 Debug Device Functional Specification. You can find these cables with an Internet search for “USB 2.0 debug cable”. These are some of the common ones:

• NET20DC cable by PLXTech, and it works well.
• GUN202 cable by Iogear.
• Ajays USB 2.0 Debug Cable and works well for me.

2. USB controller

• The host computer must have a USB 2.0 controller that is compatible with the Enhanced Host Controller Interface (EHCI) specification.
• The target computer must have a USB 2.0 controller that is compatible with the EHCI specification, and which supports kernel debugging. Not all EHCI-compatible controllers have this support, and there is no programmatic method to determine whether a given computer does have this support. The USB EHCI Specification states that this support was optional.

    Brands that I know off that support it include:

•  Intel ICH4 (and later)
•  VIA USB2.
• Etc.
• The USB debugging port must be exposed to the outside of the target computer. A few computers (mostly laptops) have a USB 2.0 controller that does support debugging through a certain physical port, but this port is not accessible from the outside of the computer. The tool usbview.exe which ships with Windbg package can help with this. Sigh!!

3. BIOS

     The BIOS must support hi-speed and talk to EHCI

Note : USB debugging does not work over a hub or docking station.

Now that we have looked at the hardware requirements let us look at the software support to enable USB debugging. USB debugging.

Software Setup

If your target computer is running Windows Vista, and your host computer is running Windows 2000 or later, you can perform kernel debugging with a USB 2.0 debugging cable.

To enable debugging with a USB cable in Windows Vista, use bcdedit and set the debugtype parameter to USB. You can set the global debug settings using the /dbgsettings option or set the parameters for specific boot entries using the bcdedit /set option.

To use bcdedit, open a Command Prompt window with elevated privileges (right click Command Prompt and click Run as administrator from the shortcut menu) since it does involve installing the USB  debug driver in the Windbg package on the host machine.

To set the debug settings for USB globally, use the following syntax:

bcdedit /dbgsettings usb [targetname:targetname] 

The following example shows how to specify USB as the global debug setting.

bcdedit /dbgsettings usb targetname:myusb

The following example shows how to specify the USB debug settings for a specific boot entry, and how to use the /debug command to enable kernel debugging for that boot entry.

bcdedit /set {18b123cd-2bf6-11db-bfae-00e018e2b8db } debugtype usb
bcdedit /set { 18b123cd-2bf6-11db-bfae-00e018e2b8db} targetname u2
bcdedit /debug {18b123cd-2bf6-11db-bfae-00e018e2b8db} on

If no {ID} is specified, the settings apply to the current({current}) boot entry. You can use the

bcdedit /enum command to view the current boot entries and their settings.

Please note that in windows 7 you can’t use USB for debugging boot applications (winload.exe/bootmgr.exe)

 

When multiple USB controllers exist which is mostly the case just like 1394 you need to specify the busparams as follows.

Bcdedit /set {current} loadoptions busparams=0.26.7

NOTE: For USB 2.0 debugging ,the bus parameters must be specified in hexadecimal on Windows Vista and Server 2008 and in decimal on Windows 7. Thanks to Doug Howe for this information. For 1394 debugging, the bus parameters must be specified in decimal. Device manager shows the busparams  in Decimal.

 

 

 

 

 

 

NOTE: USB debugging does not work over a hub or docking station.

 4. Boot debugging

Now let us look at a topic which requires a lot of spelunking when it comes to finding information. Bootdebuging referers to debugging components /applications which are loaded much before ntoskrnl.exe. I am going to focus on Windows Vista and forward since the design of boot components has had a major overhaul from Vista. . If you are looking at boot debugging for pre-Vista you could also refer to this well written blog here;: http://kobyk.wordpress.com/2008/07/04/replacing-boot-load-drivers-with-the-windows-boot-debugger/

 The boot components/applications include bootmgr, winload, winresume etc

The way of enabling boot debugging is using the bootdebug parameter. This enables or disables the boot debugger for a specified boot entry. Although this command works for any boot entry, it is effective only for boot applications. It is not sufficient to turn it on using “dbgsettings”.  

The bcdedit /set command sets a boot entry option value in the boot configuration data store. 

You can use this command to configure specific boot entry elements such as kernel debugger settings which is what we will be using it for. In addition to that it can be  data execution protection (DEP) , processor address extension (PAE) options, and to load alternate hardware abstraction layer (HAL) and kernel files. For more details refer to http://msdn.microsoft.com/en-us/library/aa906211.aspx

You can look at the bcdedit elements you specified also from the registry also. This values used to store the  bootoptions specified in boot.ini pre-vista.

Path — “HKLM\System\CurrentControlSet\Control”

Value – “SystemStartOptions”

The following commands allow you to debug the bootmgr and break in very early pretty much as early as the hand off from the BIOS is done to the bootmgr.

For serial the settings should look like:

bcdedit /bootdebug {bootmgr} on

bcdedit /debug {bootmgr} on //this maybe optional

bcdedit /set {bootmgr} debugtype serial
bcdedit /set {bootmgr} baudrate <baudrate, typically 115200>
bcdedit /set {bootmgr} debugport <1>

For 1394 the settings look like

bcdedit /bootdebug {bootmgr} on

bcdedit /debug {bootmgr} on //this maybe optional

bcdedit /set {bootmgr} debugtype 1394

bcdedit /set {bootmgr} channel <channel #>
If you have multiple 1394 controllers then you can also specify:

bcdedit /set {bootmgr} loadoptions busparams= Bus.Device.Function

After you enable the above settings in BCDEdit and reboot the machine, and the appropriate boot binary(in our case bootmgr) should break into the debugger.

 There is one very important step to remember and that is to make sure you have set initial breakpoint using the Windbg before you restart

Kd> sxe ibp

I have also found it helps to do this in case it is the bootmgr you are interested in:

Kd> sxe ld:bootmgr

 

For example – Here’s how the bootmgr will break into the debugger upon load. Now you can go ahead step through code, set your own breakpoints etc.

 

Break instruction exception – code 80000003 (first chance)
bootmgr!RtlpBreakWithStatusInstruction:
00443100 cc              int     3
kd> k
ChildEBP RetAddr 
bootmgr!RtlpBreakWithStatusInstruction 00061e94  bootmgr!vDbgPrintExWithPrefixInternal+0xfe 00061ea4 0043e728 bootmgr!DbgPrint+0x11 00061ec0 0043ea30 bootmgr!BlBdStart+0xc3 00061f58 bootmgr!BlBdInitialize+0x17d 00061f68 0041e6a0 bootmgr!ReinitializeLibrary+0x24 00061f74 00401174 bootmgr!BlInitializeLibrary+0x10 00061ff0 00020a9a bootmgr!BmMain+0x174

After you are done you can clear the settings as follows:

Kd> Sxi ibp

Kd> Sxe ld:””

Thanks to David Rocca for proof reading this. Till next time, Happy debugging.