http://www.oracle.com/technetwork/server-storage/solarisstudio/documentation/amd64-dbx-364568.html AMD64 Instruction-Level Debugging With dbxBy Nasser Nouri, May 2008, Revised April 2011Debugging at the machine-instruction level in the Solaris Studio dbx command-line debugger environment becomes very handy when a software bug cannot be found easily. Usually, programs are written in high-level languages such as C, C++, or Fortran, and most of the software defects can be debugged in the dbx environment at the same high level.However, having some knowledge of the machine-instruction level of the system on which the program is running, and using the right tool, such as dbx, can shorten the time to identify the culprit and come up with the optimal solution to fix the defect.This article describes how to use the dbx debugger efficiently on the AMD64 architecture. It describes how to display the contents of memory at specified addresses, and how to display machine instructions. Use the regs command to print out the contents of machine registers or the print command to print out individual registers. Use the nexti, stepi, stopi, and tracei commands to debug at AMD64 machine-instruction level.AMD64 Architecturedbx CommandsThe Problem StatementThe dbx FailureThe Debugging SessionIn ConclusionAMD64 ArchitectureFirst let's review briefly the AMD64 architecture and see how it is different from the 32-bit x86 architecture. I describe only the materials that are relevant to this article. For an in-depth understanding of AMD64 architecture, please refer to AMD64 manuals (http://developer.amd.com) and AMD64 Application Binary Interface (ABI) (http://www.x86-64.org).The AMD64 architecture has sixteen 64-bit general purpose registers (GPRs): RAX, RBX, RCX, RDX, RBP, RSI, RDI, RSP, R8, R9, R10, R11, R12, R13, R14, and R15. Compared to the x86 architecture, the AMD64 architecture has eight new GPRs.The RAX, RBX, RCX, RDX, RBP, RSI, RDI, and RSP registers are used by both 32-bit and 64-bit binaries. However, in 32-bit mode, only the low 32 bits of these registers are accessible by 32-bit binaries. In the x86 architecture, these registers are EAX, EBX, ECX, EDX, EBP, ESI, EDI, and ESP, respectively.The AMD64 architecture provides sixteen 128-bit XMM registers XMM0 through XMM15. Registers XMM0 through XMM7 are used for passing float and double parameters. The long double type is passed in memory. A long double in AMD64 architecture is 16 bytes long compared to 12 bytes in the x86 architecture. The long double type is implemented based on 80-bit extended (IEEE) standard.The AMD64 architecture also provides eight x87 floating point registers, MMX0/FPR0 through MMX7/FPR7, each 80 bits wide.In contrast to the 32-bit architecture in which the function parameters are passed on the stack, the 64-bit architecture has six registers available for integer parameter passing. If the number of integer parameters is more than six, the remaining parameters are passed on the stack.The bool, char, short, int, long, long long, and pointer types are classified as integer class. For passing parameters of the integer class, the next available register of the sequence RDI, RSI, RDX, RCX, R8, and R9 is used.Registers RBP, RBX, and R12 through R15 belong to the calling function, and the called function is required to preserve their values.The RIP register is the instruction pointer register. In 64-bit mode, the RIP register is extended to 64 bits to support 64-bit offsets. In 32-bit x86 architecture, the instruction pointer register is the EIP register.The return value of a function is classified based on the rules that are specified in AMD64 ABI. For instance, if the return value needs to be passed in memory, then the caller provides space for the return value and passes the address of this storage in the RDI register as if it were the first argument to the function. On return, the RAX register contains the address that has been passed by the caller in the RDI register.Similarly, if the return type is integer, the next available register of the sequence RAX, RDX is used.In addition to registers, each function has a frame on the run-time stack. The run-time stack grows downwards from a high address. Table 1 shows the stack organization.Table 1 Stack Frame With Base PointerPosition Contents Frame 8n+16 (%rbp)... 32 (%rbp)24 (%rbp)16 (%rbp) argument #n... argument #2argument #1argument #0 High address Previous frame 8 (%rbp) return address Current frame 0 (%rbp) previous %rbp value Current frame -8 (%rbp)-16 (%rbp)... 0 (%rsp) local variable #1local variable #2... local variable #n Current frame   Low address -128 (%rsp) red zone   The RSP register is the stack pointer register and the RBP register is the frame pointer register. Stack operations make implicit use of the RSP register, and in some cases, the RBP register. The RSP register is decremented when items are pushed onto the stack, and incremented when they are popped off the stack. The RBP register points to the lowest address of the data structure that is passed from one function to another.The 128-byte area beyond the location pointed to by the RSP register is known as red zone and is considered to be reserved. Functions can use this area for temporary data that is not needed across function calls. In particular, leaf functions can use this area for their entire stack frames, rather than adjusting the stack pointer in the prologue and the epilogue.prologue:pushq %rbp          / save frame pointermovq %rsp,%rbp      / set new frame pointersubq $48,%rsp       / allocate stack spacemovq %rbx,-16(%rbp) / save callee-saved registersmovq %r12,-24(%rbp)movq %r13,-32(%rbp)movq %r14,-40(%rbp)movq %r15,-48(%rbp)There is no need to adjust the RSP stack pointer register if the red zone area is used. In other words, the subq $48,%rsp instruction is not needed in function prologue if the red zone area is used.epilogue:movl -4(%rbp), %eax / set up return valuemovq -16(%rbp),%rbx / restore callee-saved registersmovq -24(%rbp),%r12movq -32(%rbp),%r13movq -40(%rbp),%r14movq -48(%rbp),%r15leaveretThe C++ language has its own Application Binary Interface (ABI). The C++ ABI has well-defined rules for function parameter passing and return values. The C++ ABI rules supplement the AMD64 ABI rules; the C++ compiler has to use the C++ ABI rules for function parameter passing in addition to the AMD64 ABI rules.dbx CommandsThe following commands are documented in Solaris Studio 12.2: Debugging a Program With dbx for machine-instruction level debugging.examine [ address ] [ / [ count ] [format ] ] Display the contents of memory starting at address for count items in format format stepi Single step one machine instruction (step into calls) nexti Step one machine instruction (step over calls) listi Intermix source lines and assembly code tracei Tracing at the machine-instruction level stopi Set breakpoints at the machine-instruction level dis Disassemble 10 instructions, starting at the value of `+' print expression, ... Print the value of one or more expressions expression, ... regs [-f] [F] Print value of registers-f: include floating-point registers (single precision)-F: Include floating-point registers (double precision) The Problem StatementTo demonstrate machine-instruction level debugging, let's use a real bug report that was filed against the 64-bit dbx on the AMD64 platform, including a test case. The bug report says “On AMD64 dbx prints hex values instead of letters after strchr call:(dbx) print strchr("hello", 'l') = 0xfffffd7fffdff742 "\xdf\xff^?\xfd\xff\xff^D"Here is the test case:  char *b = "hello";  printf("%s\n", b);  printf("%s\n", strchr("hello", 'l'));}There is nothing wrong with the program. The bug is in dbx.The dbx FailureFirst let's observe the normal flow of the program in the dbx environment by just stepping through the test case code.% dbx a.outReading a.outReading ld.so.1Reading libc.so.1(dbx) stop in main(2) stop in main(dbx) runRunning: a.out(process id 16245)stopped in main at line 3 in file "1.c"  3 char *b = "hello";(dbx) nextstopped in main at line 4 in file "1.c"  4 printf("%s\n", b);(dbx) nexthellostopped in main at line 5 in file "1.c"  5 printf("%s\n", strchr("hello", 'l'));(dbx) nextllostopped in main at line 6 in file "1.c"  6 }(dbx) nextexecution completed, exit code is 4The print statement at line 5 calls the strchr() function with two parameters. The strchr() function searches through the first parameter hello and returns a pointer to the first occurrence of the l character. Hence, the llo character string is displayed correctly by the printf statement.Now let's reproduce the failure by calling the strchr() function directly from the dbx command line using the print command. The call command in dbx can also be used to call the strchr() function from the command line.% dbx a.outReading a.outReading ld.so.1Reading libc.so.1(dbx) stop in main(2) stop in main(dbx) runRunning: a.out (process id 14772)stopped in main at line 3 in file "1.c"<  3 char *b = "hello";(dbx) next stopped in main at line 4 in file "1.c"  4 printf("%s\n", b);(dbx) next hellostopped in main at line 5 in file "1.c"  5 printf("%s\n", strchr("hello", 'l'));(dbx) print strchr("hello", 'l')strchr("hello", 'l') = 0xfffffd7fffdff742 "\xdf\xff^?\xfd\xff\xff^D"dbx prints incorrect output when the strchr() function is called by the print command. dbx should display the llo string instead of hex characters, since the call to the strchr() function is supposed to return a pointer to the first occurrence of the l character in the string hello.The Debugging SessionLet's run the debugger with the a.out executable and stop right before the printf statement. The strchr() function is defined in the libc library and most likely is not compiled with the -g option. So there is no debugging information and we have to rely on the assembly code only.The stopi command is used to set a breakpoint at the first machine instruction of the strchr() function.% dbx a.outReading a.outReading ld.so.1Reading libc.so.1(dbx) stop in main(2) stop in main(dbx) runRunning: a.out (process id 15045)stopped in main at line 3 in file "1.c"  3 char *b = "hello";(dbx) nextstopped in main at line 4 in file "1.c"  4 printf("%s\n", b);(dbx) nexthellostopped in main at line 5 in file "1.c"  5 printf("%s\n", strchr("hello", 'l'));(dbx) stopi at strchr(3) stopi at &strchr(dbx) print strchr("hello", 'l')stopped in strchr at 0xfffffd7fff3079100xfffffd7fff307910: strchr : movb& (%rdi),%dldbx stops at the first instruction of the strchr() function after the strchr() function is called from the dbx command line using the print command.The dis command can be used to display the first portion of machine instructions for the strchr() function.(dbx) dis strchr 0xfffffd7fff307910: strchr       : movb (%rdi),%dl 0xfffffd7fff307912: strchr+0x0002: cmpb %dh,%dl0xfffffd7fff307915: strchr+0x0005: je strchr+0x3f [0xfffffd7fff30794f, .+0x3a ]0xfffffd7fff307917: strchr+0x0007: testb %dl,%dl0xfffffd7fff307919: strchr+0x0009: je strchr+0x33 [0xfffffd7fff307943, .+0x2a ]0xfffffd7fff30791b: strchr+0x000b: movb 0x0000000000000001(%rdi),%dl0xfffffd7fff30791e: strchr+0x000e: mpb %dh,%dl0xfffffd7fff307921: strchr+0x0011: je strchr+0x3c [0xfffffd7fff30794c, .+0x2b ]0xfffffd7fff307923: strchr+0x0013: testb %dl,%dl 0xfffffd7fff307925: strchr+0x0015: je strchr+0x33 [0xfffffd7fff307943, .+0x1e ]The first instruction of the strchr() function is movb (%rdi),%dl, which moves the contents of the memory location pointed to by the %rdi register to the low eight bits of the %rdi register itself. The first instruction is not the pushq %rbp instruction, which means the strchr() function has no prologue. It is not a defect that the function does not have a prologue.The debugger is stopped at the first instruction, which is the right place in the program to verify whether the input parameters are being passed correctly to the strchr() function. The strchr() function has two parameters. The first parameter is a pointer to the memory location that contains the hello character string and the second parameter is the character l. Based on the AMD64 ABI, the first and second parameters are assigned to the %rdi and %rsi registers in sequence. There are two ways to display the content of the %rdi and %rsi registers.You can use the print command to print the contents of the individual registers. The -flx options force dbx to display the contents of the %rdi and %rsi registers in long-hex format.(dbx) print -flx $rdi $rdi = 0xfffffd7fffdff740(dbx) print -flx $rsi$rsi = 0x6cYou can use the regs command to display the contents of all of the AMD64 registers.(dbx) regscurrent frame: [1]r15         0x0000000000000000r14         0x0000000000000000r13         0x0000000000000000r12         0x0000000000000000r11         0xfffffd7fff307910r10         0x0000000000000000r9          0x0000000000010000r8          0xfefeff6e6b6b6467rdi         0xfffffd7fffdff740rsi         0x000000000000006crbp         0xfffffd7fffdff810rbx         0xfffffd7fff3fb190rdx         0x0000000000000000rcx         0x000000003f570d87rax         0x0000000000000000trapno      0x0000000000000003err         0x0000000000000000rip         0xfffffd7fff307910:strchr movb (%rdi),%dlcs          0x000000000000004beflags      0x0000000000000282rsp         0xfffffd7fffdff738ss          0x0000000000000043fs          0x00000000000001bbgs          0x0000000000000000es          0x0000000000000000ds          0x0000000000000000fsbase      0xfffffd7fff3a2000gsbase&     0xffffffff80000000The %rdi register contains a pointer to the memory location 0xfffffd7fffdff740, which is allocated on the stack. In the normal program flow, the %rdi register contains a pointer to the memory location in the data segment. However, when dbx is asked to call a function (strchr()), dbx copies the memory location in the data segment onto the stack and passes the stack address to the %rdi register.The contents of the memory location 0xfffffd7fffdff740 can be verified by using the examine command. The memory location should contain the hello character string.(dbx) examine 0xfffffd7fffdff740 / 20xfffffd7fffdff740: 0x6c6c6568 0x0000006fBy looking up the ASCII table, we can verify that indeed the memory location 0xfffffd7fffdff740 contains the hello character string. The hex number 68 stands for the character h, 65 stands for the character e, 6c stands for the character l, and 6f stands for the character o.You can use the examine command directly to display the contents of the memory location 0xfffffd7fffdff740 as a character string without referring to the ASCII table .(dbx) examine 0xfffffd7fffdff740 / 6c0xfffffd7fffdff740: 'h' 'e' 'l' 'l' 'o' '\0'The %rsi register contains the hex number 6c, which stands for the l character.The other two important registers are the %rsp (the stack pointer) and %rbp (the frame pointer). The %rsp register is pointing to the top of the stack and its value is 0xfffffd7fffdff738. As you can see, this value is very close to the contents of the %rdi register, which is pointing to the memory location on the stack that contains the hello character string.The %rbp register is the frame pointer and contains 0xfffffd7fffdff81 value. The %rbp register is not used in the strchr() function.The contents of the run-time stack can be displayed using the examine command.(dbx) examine 0xfffffd7fffdff738 / 32 lx0xfffffd7fffdff738: 0xfffffd7fff220004 0x0000006f6c6c65680xfffffd7fffdff748: 0x0000000000000000 0x00000000000000000xfffffd7fffdff758: 0x0000000000000000 0xfffffd7fffdff7b00xfffffd7fffdff768: 0xfffffd7fff3c7e50 0x00000000000100000xfffffd7fffdff778: 0x0000000000000000 0x0000000000410c500xfffffd7fffdff788: 0x0000000000000000 0xfffffd7fffdff8480xfffffd7fffdff798: 0x0000000000410c50 0x0000000000410c580xfffffd7fffdff7a8: 0xfffffd7fff3fb190 0x00000000000000000xfffffd7fffdff7b8: 0x0000000000000000 0xfffffd7fffdff8100xfffffd7fffdff7c8: 0x000000000040099d 0x00000000000000000xfffffd7fffdff7d8: 0x0000000000000000 0x00000000000000000xfffffd7fffdff7e8: 0x0000000000000000 0x00000000000000000xfffffd7fffdff7f8: 0xfffffd7fff3fb190 0x0000000000410c500xfffffd7fffdff808: 0xfffffd7fffdff838 0xfffffd7fffdff8200xfffffd7fffdff818: 0x000000000040080c 0x00000000000000000xfffffd7fffdff828: 0x0000000000000000 0x0000000000000001 In fact, we can unwind the run-time stack by following the principles that we learned in the previous section (see Table 4) about the stack frame with the base pointer. For instance, the hex number 0x40080c is the address of next instruction after the callq instruction. The main function is called from the _start() function using the callq instruction.The hex number 0x40080c is the return address that is pushed onto the stack before the call to the main() function. The instruction at address 0x40080c, push %rax, will be executed upon the completion of the main() function. In other words, the address 0x40080c will be loaded into the program counter, the %rip register, once the main function returns.You can use the objdump utility program to dump the text section of an executable.objdump -S a.out00000000004007a0 <_start>:  4007a0:    6a 00                   pushq  $0x0  4007a2:    6a 00                   pushq  $0x0  4007a4:    48 8b ec                mov    %rsp,%rbp  4007a7:    48 8b fa                mov    %rdx,%rdi  4007aa:    48 c7 c0 80 0a 41 00    mov    $0x410a80,%rax  ...  400806:    59                      pop    %rcx  400807:    e8 54 01 00 00          callq  400960 
     
        40080c:    50                      push   %rax  40080d:    50                      push   %rax   ...The first instruction of the main function is push %rbp. Hence, the previous frame pointer (0xfffffd7fffdff820) is pushed onto the stack right after the return address. Similarly, the return address (0x40099d) is pushed onto the stack when the strchr() function is called from the command line.0000000000400960 
      
       400960:    55                      push  400961:    48 8b                   mov    %rsp,%rbp  400964:    48 83 ec 40             sub    $0x40,%rsp  ...  40099d:    b8 6c 00 00 00          mov    $0x6c,%eax  4009a2:    0f be f0                movsbl %al,%esi  4009a5:    48 c7 c7 68 0c 41 00    mov    $0x410c68,%rdi  4009ac:    b8 00 00 00 00          mov    $0x0,%eaxHowever, the strchr() function does not have a function prologue, so the content of %rbp register stays the same when the strchr() function is called from the main() function. The content of %rbp register is the hex value 0xfffffd7fffdff810 and in turn the content of the 0xfffffd7fffdff810 address points to the previous frame pointer 0xfffffd7fffdff820.(dbx) examine 0xfffffd7fffdff8100xfffffd7fffdff810: 0xfffffd7fffdff820Going forward, we single step through the machine instructions using the nexti command until we get to the instruction that returns the return value in the %rax register. We can use the dis command to display the last portion of machine instructions for the strchr() function.(dbx) dis0xfffffd7fff307941: strchr+0x0031: jne strchr [0xfffffd7fff307910, .-0x31 ]0xfffffd7fff307943: strchr+0x0033: xorl %eax,%eax0xfffffd7fff307945: strchr+0x0035: ret0xfffffd7fff307946: strchr+0x0036: incq %rdi0xfffffd7fff307949: strchr+0x0039: incq %rdi0xfffffd7fff30794c: strchr+0x003c: incq %rdi0xfffffd7fff30794f: strchr+0x003f: movq %rdi,%rax0xfffffd7fff307952: strchr+0x0042: ret0xfffffd7fff307953: strchr+0x0043: addb %al,(%rax)(dbx) nextistopped in strchr at 0xfffffd7fff3079490xfffffd7fff307949:strchr+0x0039:incq %rdi(dbx) nextistopped in strchr at 0xfffffd7fff30794c0xfffffd7fff30794c: strchr+0x003c: incq %rdi(dbx) nextistopped in strchr at 0xfffffd7fff30794f0xfffffd7fff30794f: strchr+0x003f: movq %rdi,%rax(dbx) nextistopped in strchr at 0xfffffd7fff3079520xfffffd7fff307952: strchr+0x0042: retBased on the description of the strchr() function, at the end it is supposed to return a pointer to the first occurrence of the l character in the string hello. We can verify the correctness of the strchr() function by examining the contents of the %rax register.(dbx) examine $rax / 4c0xfffffd7fffdff742:     'l' 'l' 'o' '\0'Indeed, the value of the %rax register is a pointer to the memory location 0xfffffd7fffdff742, which is allocated on the stack and contains the llo character string.We have verified that the strchr() function works correctly and returns a pointer to the llo character string in the %rax register. So the problem must be with what dbx does internally after it finishes calling the strchr() function. Fast forward, after calling a user function, dbx always calls the fflush() function to flush the output stream. The fflush() function takes one parameter, which is a pointer to the FILE data structure.fflush - flush a stream#include 
       
        int fflush(FILE *stream);You can use the dis command to display the machine instructions for the fflush function.(dbx) dis fflush0xfffffd7fff33dca0: fflush:        pushq %rbp0xfffffd7fff33dca1: fflush+0x0001: movq %rsp,%rbp0xfffffd7fff33dca4: fflush+0x0004: movq %rbx,0xfffffffffffffff0(%rbp)0xfffffd7fff33dca8: fflush+0x0008: movq %r12,0xfffffffffffffff8(%rbp)0xfffffd7fff33dcac: fflush+0x000c: subq $0x0000000000000010,%rsp0xfffffd7fff33dcb0: fflush+0x0010: testq %rdi,%rdi0xfffffd7fff33dcb3: fflush+0x0013: movq %rdi,%rbxLet's go over the fflush function prologue:pushq %rbpStore the previous frame pointer on the stack.movq %rsp, %rbpStore the value of the %rsp register or the previous stack pointer into the %rbp register. This value is the new frame pointer for the fflush() function.movq %rbx,0xfffffffffffffff0(%rbp)movq %r12,0xfffffffffffffff8(%rbp)The %rbx register and the %r12 register are callee-saved registers. The fflush() function must preserve the contents of these registers on the stack for the caller function so they can be restored later in the function epilogue just before exiting the function.sub $0x0000000000000010,%rspAdjust the stack pointer for the fflush() function.The stopi command is used to stop at the first instruction of fflush() function.(dbx) stopi at fflush(dbx) contdbx: Call to 'strchr' completed. Going back to previous commandinterpreterstopped in fflush at 0xfffffd7fff33dca00xfffffd7fff33dca0: fflush: pushq    %rbpdbx stops at the first instruction of the fflushLet's display the %rdi, %rbp, and %rspregisters. The %rdi register contains a pointer to the FILE data structure.(dbx) print -flx $rdi$rdi = 0xfffffd7fff37f0a0(dbx) print -flx $rbp$rbp = 0xfffffd7fffdff810(dbx) print -flx $rsp$rsp = 0xfffffd7fffdff748We step through the function prologue and print the %rsp and %rbp registers again.(dbx) stepistopped in fflush at 0xfffffd7fff33dca10xfffffd7fff33dca1: fflush+0x0001: movq %rsp,%rbp(dbx) stepistopped in fflush at 0xfffffd7fff33dca40xfffffd7fff33dca4: fflush+0x0004: movq %rbx,0xfffffffffffffff0(%rbp)(dbx) stepistopped in fflush at 0xfffffd7fff33dca80xfffffd7fff33dca8: fflush+0x0008: movq %r12,0xfffffffffffffff8(%rbp)(dbx) stepistopped in fflush at 0xfffffd7fff33dcac0xfffffd7fff33dcac: fflush+0x000c: subq $0x0000000000000010,%rsp(dbx) stepistopped in fflush at 0xfffffd7fff33dcb00xfffffd7fff33dcb0: fflush+0x0010: testq %rdi,%rdi(dbx) print -flx $rbp$rbp = 0xfffffd7fffdff740(dbx) print -flx $rsp$rsp = 0xfffffd7fffdff730If you can recall from previous section, the run-time stack grows downwards from high address. By careful examination of the %rsp register and comparing its value (0xfffffd7fffdff730) with the last value of the %rsp register (0xfffffd7fffdff738) in the strchr() function, it becomes obvious that the space that is allocated on the stack for the fflush() function overlaps with the space for the strchr() function.The 0xfffffd7fffdff738 value is right between the value of the %rbp register (0xfffffd7fffdff740) and the value of the %rsp register (0xfffffd7fffdff730) of the fflush() function. Therefore, the fflush() function overwrites the contents of the run-time stack for the strchr() function, which explains why the print strchr("hello", 'l') command displays garbage instead of the llo character string.The fix for the dbx debugger is to preserve the contents of the run-time stack just before the call to the fflush() function and restore it just before returning to the print command.In ConclusionIn general, low-level debugging requires the user to have some kind of knowledge about the system on which the program is executing. But once necessary knowledge is learned, even the most difficult bugs can be detected using the low-level debugging techniques and using the right tool, such as dbxYou can learn more about the x86 assembly language by referring to the article Assembly Language Techniques for Oracle Solaris x86 Platforms