Debug Breakpoint Handling

  COMPILATION LISTING OF SEGMENT hello
  Compiled by: Multics PL/I Compiler,
  	       Release 33e, of October 6, 1992
  Compiled at: Arrakis
  Compiled on: 01/28/17  1126.7 mst Sat
      Options: table list

   1 hello:    proc;
   2
   3   dcl  ioa_$nnl entry() options(variable);
   4
   5           call ioa_$nnl("Hello,");
   6
   7           call ioa_$nnl(" world!^/");
   8
   9           end hello;
  

A Simple Program

This analysis begins as a thought experiment, with creation of a user program in which a debug breakpoint can be set. To keep things simple, we'll use a "Hello, world!" program. The PL/I compiler listing shows source line numbers for this program.

The program is modified from its simplest form to permit setting a breakpoint between output of "Hello," and " world!". This will give visual confirmation of: where the breakpoint is set; when it is actually reached; and when the program ends.

Debugging the Program

The analysis continues by using debug to set a breakpoint in the hello program.

Setting a Breakpoint

The following debug command and debugging requests:

• display the source at line 7 of the hello.pl1 program file; and
• set a breakpoint at the second instruction of line 7 in the hello object segment.

  debug
  /hello/&a7,s
  call ioa_$nnl(" world!^/");
  ,i3
  LINE NUMBER 7
      27    777755 2370 04     ldaq      -23,ic      000004 = 040167157162 154144041136
      30   6 00102 7571 00     staq      pr6|102
      31    057000 2350 03     lda       57000,du
  +1<
  Break   1  set at  hello|30 (line 7)  At instruction:
      30   6 00102 7571 00     staq      pr6|102
 

After a breakpoint is set, the program location now contains a Master Mode Entry 2 mme2 instruction, with an operand set to the break number. When executed, this instruction causes an event that results in signalling the mme2 condition.

  -1,i3
  LINE NUMBER 7
      27    777755 2370 04     ldaq      -23,ic      000004 = 040167157162 154144041136
      30       000001 0040 00  mme2      1
      31    057000 2350 03     lda       57000,du
  .q
  r 07:53 0.149 0
 

Execution at a Breakpoint

  hello
  Hello,Break  1  at line 7  of  hello  -  at  367|30
  

When a breakpoint is set, running the hello program encounters the breakpoint. The debug on-unit for the mme2 condition accepts debugging requests.

Ring 0: Breakpoint Instruction Generates a Fault

To learn how a breakpoint interrupts execution of its containing program, start with the components introduced above.

• Object segment for the program, compiled with the -table option, in which a debug breakpoint (the mme2 instruction) has been set. The Procedure Pointer Register PPR points to the currently-running instruction, which is the breakpoint instruction. It is shown in bold green text.
• Stack frame for the running program. The stack_4 segment begins with a stack_header structure; and grows as frames for new programs are pushed onto the bottom of the stack segment. Stack Pointer Register pr6 points to the frame for the running program. This running frame may grow or shrink as the program changes its set of variables and temporary storage. The running frame is highlighted in green.
• The user's breakmap segment contains information about breakpoints set in programs, including the instruction originally at each breakpoint location.

Executing a Breakpoint Instruction

When the running program executes the mme2 instruction, it causes a MME2 fault event to occur. The processor suspends execution of the program running in the user ring, and begins a Fault Cycle in ring 0.

Fault Cycle

The Control Unit is the section of the processor that oversees the flow of instruction executions. Two sorts of events can temporarily suspend program execution and redirect the flow to other instructions: faults and interrupts. When such events occur, the processor begins a Fault Cycle to redirect instruction execution. For a MME2 fault at hello|30, or for other faults and interrupts, the Fault Cycle includes the following:

Fault and Interrupt Vectors

The Fault Cycle references a fault or interrupt vector entry specific to the particular fault or interrupt that has occurred. Such entries are defined in the Multics fault vector structure¹, which holds several arrays containing:

• interrupt vector entries for 32 interrupt types;
• fault vector entries for 32 fault types;
• 32 pointers: the handlers for each type of interrupt;
• 32 pointers: the control unit storage location used for each interrupt type;
• 32 pointers: the handlers for each type of fault; and
• 32 pointers: the control unit storage location used for each fault type.

1. The array of interrupt vector entries always begins at absolute memory address 0. The architecture allows the array of fault vector entries to begin at any even location elsewhere in the first page of memory. (Location of fault vector entries is specified in the Fault Vector switches on the processor configuration panel.) For Multics, the fault entries and other pointer arrays immediately follow the interrupt vector entries, as shown in fault_vector.incl.pl1.

Fault Cycle for MME2

When the hello program executes the breakpoint mme2 instruction, that instruction starts a MME2 fault cycle. The MME2 fault has a fault number of 21. Its fault vector entry is at address 152¹. The xed instruction executes the pair of instructions in that fault vector.

1. By definition, the Master Mode Entry 2 (mme2) instruction uses the vector entry located at: FAULT_VECTOR|52. This is an absolute address in the first page of memory. The FAULT_VECTOR switches in the processor configuration panel specify an address of 100₈. Thus, the MME2 fault vector entry is located at absolute address: 152₈.

The fault vector entry begins with an scu instruction, which copies the Safe-Stored Control Unit Data to the location given in the scu data ptr for the MME2 fault: pds$signal_data.scu². The Process Data Segment pds is a per-process, ring 0 segment created in the process directory. Its pages are wired in memory while its owning process is running on the processor.

2. Since the MME2 fault was started by the process running on this processor, it is appropriate to store data about the fault in the process's pds segment. The signal_data entry is structured as a standard set of machine conditions (mc), as shown in mc.incl.pl1. Thus, pds$signal_data.scu is a reference to the mc.scu structure element at the pds$signal_data location.

At this point in handling the MME2 fault, only three instructions have been executed:

1. The xed instruction generated by the processor at the end of the Fault Cycle.
2. The two instructions it executes from the fault vector entry:
   1. The scu instruction to save the safe-stored Control Unit Data in memory.
   2. The tra instruction to entry the MME2 fault handler.

None of these instructions has changed, nor depends upon, any pointer register or register of any kind. All program-visible registers still contain data they held when the fault occurred.

Ring 0 Fault Handler: fim$signal_entry

The ring 0 Fault Interceptor Module: fim.alm¹ is a ring 0, kernel segment whose pages are wired in memory. This prevents taking a page fault while processing another type of fault. The fim operates in privileged instruction mode, with interrupts inhibited² to prevent I/O interrupts and other low-priority faults from interrupting its handling of the fault or interrupt.

1. To review the code in fim.alm, you will also need the ALM version of the machine conditions structure, mc.incl.alm.
2. Interrupts and certain low-priority faults are recognized only at specific times during the execution of an instruction pair. If, at these times, bit 28 in the instruction word is set ON, the trap is inhibited and program execution continues. The interrupt or fault signal is saved for future recognition and is reset only when the trap occurs.

The fim defines several fault handler³ entrypoints. Most of these provide similar services, but store data for different groups of faults in distinct locations. The MME2 fault is one of a group of faults that are handled directly in the faulting ring of execution. The fault event is converted to a condition that is signaled in this faulting ring⁴. The fim$signal_entry handles such faults.

3. Modules other than fim.alm manage expected kernel faults (e.g., page faults) needing special handling; and unexpected faults (e.g., execute faults) that may stop system operation. No further mention is made of these other fault handlers.
4. Most often, the faulting ring of execution is the user ring. But it might also be one the of rings between the user ring and ring 0, if code in such ring experiences a fault or interrupt.

Ring 0 Exit Routine: signaller

The fim handler captured machine conditions at the time of the MME2 fault, and chose a condition name to be signaled when reporting the fault event in the outer ring. The next step is to actually signal that condition on the stack of the faulting program, which is usually an outer ring stack. The ring 0 signaller.alm segment¹ performs this operation.

1. The signaller.alm source code references include files:
   • signaller_stack.incl.alm: describes the layout of the stack frame which signaller pushes onto the outer ring stack.
   • stack_frame.incl.alm and stack_frame.incl.pl1: describes standard information included at the start of every stack frame.
   • stack_header.incl.alm: describes the standard information placed at the beginning of each stack. signaller uses several header items to invoke the outer ring signal mechanism (which can differ for each ring).

signaller is a ring 0 segment that operates in privileged instruction mode, with interrupts inhibited to prevent I/O interrupts and other low-priority faults from interrupting its extended handling of the fault.

signaller$signaller performs checks for different user environments when selecting the stack on which to signal the condition.

• Did the fault occur in the signaller or restart_fault programs? If so, terminate the process because the signaller's code is unexpectedly causing a fault.
• Was the user program running in BAR mode when the fault occurred? If so, construct a stack frame pointer using stack_header.bar_mode_sp (stack frame which called the program running in BAR mode). If the BAR mode stack frame is not the last stack frame, then the stack is corrupt; terminate the process. Otherwise, signal on the stack given in the fault's machine conditions (mc.prs(6)).
• Does Control Unit Data show a ring number of the faulting program equal to the ring number in the machine conditions stack pointer? If not, signal the condition on the stack of the faulting program's ring number: pds$stacks(program_ring). Normally, signal on the stack given in the fault's machine conditions (mc.prs(6)).

For a MME2 fault caused by a debug breakpoint, the selected stack is most likely the stack for the user's login ring. That is the ring containing object segments in which breakpoints can be set.

Preparing to Call the Outer Ring Signal Mechanism

Having decided on which stack to signal the condition, the signaller creates a new stack frame at the end of this stack¹.

1. I mistakenly thought the signaller creating a new frame at the end of an outer ring stack was called a crawlout operation. It differs from a crawlout in several ways. But its similarity to a crawlout led Bernie Greenberg to christen this ring-0-signalling-in-outer-ring mechanism as a walkout from ring 0.

User Ring: Breakpoint as a Condition

Returning to the original ring of execution, a mme2 condition is signaled to indicate that execution has reached the program breakpoint.

Code in return_to_ring_0_¹ is now running in the outer ring. The current Stack Frame Pointer (pr6) points to its stack frame, just created by signaller. Therefore, that stack frame is shown in green.

2. return_to_ring_0_.alm is a small ALM program available for your review.

Interrupts and low priority faults are no longer inhibited. So any pending interrupts or faults (e.g., ring alarm faults) that arose while running in ring 0 code can now occur.

Code at the return_to_ring_n_ label consists of only four instructions:

     nop
     nop
     nop
     callsp bb|0
 

The three nop instructions give the processor an opportunity to process any pending interrupts or low-priority faults delayed while running in ring 0 with interrupts inhibited.

Officially, the callsp instruction is known as a call6 instruction. Its operand bb|0 is an ALM synonym for pr3|0, which contains the outer ring signal_ptr value. call6 references pr6, which correctly points to the return_to_ring_0_ stack frame. And signaller loaded the Argument List Pointer Register (pr0) for this call with the address of the arg_list constructed in this stack frame (shown in the diagram above).

This call invokes the outer ring signal mechanism.

Support Routine: signal_

In the user ring, stack_header.signal_ptr points to the signal_¹ program. This program supports two kinds of condition handlers. The normal handler is an on-unit attached to the stack frame of a running program. This on-unit is active only while the stack frame appears on the stack.

1. The signal_.pl1 program source is available for review. Only the first 23 lines following the ARG(0): label are pertinent to this analysis. They show invocation of the sct_manager_$call_handler.

However, debug must implement breakpoints set in a program when debug is no longer running. For example, breakpoints set in a program should be handled if some program failure triggers creation of a new process. The user reinvokes the program being debugged from a command, without restarting debug.

For such cases, signal_ supports a less-used type of condition handler called a static handler. Before searching for a normal condition on-unit, signal_ calls sct_manager_$call_handler, asking it to invoke any static handler for the condition being signaled. It passes the following information:

       call sct_manager_$call_handler( mc_ptr, condition_name, ..., continue);
 

If sct_manager_$call_handler can handle the condition, it sets continue="0"b to stop further signalling.

Support Routine: sct_manager_

The sct_manager_¹ looks in the System Condition Table (SCT) for static handlers. The SCT for a ring is located just after the declared stack_header structure in the Stack Header region (first page) of the stack segment. The stack_header.sct_ptr points to the SCT.

1. The sct_manager_.pl1 is available for review. It matches mc.fcode values against SCT index constants defined in static_handlers.incl.pl1.

For each possible value of mc.fcode (the machine conditions fault code), a handler slot is reserved in the SCT. For MME2, the fault code is 21. sct_handler(21) points to process_overseer_$mme2_fault_handler_. This entry was added to the SCT by initialize_process_ when the process was created.

sct_manager_ calls the static handler, with all of its arguments. It permits the handler to decide how to set the continue argument.²

2. If no static handler is found for a given fault code, then sct_manager_ sets the continue flag to "1"b, and returns to let signal_ continue searching for an on-unit.

Handler for the Breakpoint

The debug break handling entrypoint is invoked to accept debugging requests.

Support Routine: debug$mme2_fault

process_overseer_$mme2_fault_handler_ is a proxy handler that actually calls the debug$mme2_fault static handler. It passes all incoming arguments to the debug handler.

debug$mme2_fault¹ is called with a pointer to the machine conditions captured for the MME2 fault. These are the machine conditions placed in the return_to_ring_0_ stack frame by signaller. debug uses them to locate the breakpoint that executed the mme2 instruction, and to find that breakpoint entry in the breakmap.

1. Source for the debug$mme2_fault entrypoint may be reviewed in debug.pl1, starting at the mme2_fault: entrypoint label.

It calls db_parse² to parse incoming debug request lines from the user: requests to trace the stack, set or reset breakpoints, continue execution after a breakpoint is hit, etc. For a .c request, it calls db_break$restart to continue execution from the point at which it was suspended by the current breakpoint.

2. Operations of db_parse are not pertinent to this analysis, so source for this module has not been provided for review.

Restarting Execution at the Breakpoint

db_break$restart¹ handles restarting of execution when continuing from a debug breakpoint.

1. Source for the db_break$restart entrypoint is available for review in db_break.pl1, at the restart: label.
   • db_break.pl1 makes extensive references to the machine conditions and Control Unit Data in mc.incl.pl1.

After Debug Handles the Breakpoint

Upon return from db_break$restart, the debug$mme2_fault static handler returns to its caller, the proxy static handler.

The proxy handler, process_overseer_$mme2_fault_handler_, sets its continue parameter to "0"b, telling signal_ the mme2 condition has been handled. It then returns to its caller.

The sct_manager_$call_handler that invoked the static handler now returns to its caller.

signal_, seeing continue="0"b, returns to its caller.

Support Routine: return_to_ring_0_ - Primary Entrypoint

In Support Routine: return_to_ring_0_ - return_to_ring_n_ Label (see above), signaller exited ring 0 by doing a rtcd instruction to the return_to_ring_0_$return_to_ring_n_¹ label. Code at that external label called the outer ring signal_ program.

1. return_to_ring_0_.alm is a small ALM program available for your review.

But signaller set the owner of the stack frame to the primary entrypoint: return_to_ring_0_$return_to_ring_0_. It also set the stack_frame.return_ptr to this same code location. Thus, when signal_ returns, the return operator transfers to this code.

Primary entrypoint code is:

  return_to_ring_0_:
	  callsp	restart_fault_ptr,*	call into ring zero

	  even
  restart_fault_ptr:
	  its	75|0
 

restart_fault_ptr² points to the restart_fault|0 entrypoint in segment 75₈.

2. Source for return_to_ring_0_.alm shows restart_fault_ptr: as the label of an ITS pointer defined in the text section of program, initialized to a null pointer value (-1|1). However, initialize_faults sets this pointer to addr(restart_fault$) (the PL/I equivalent of restart_fault|0) when the kernel is loaded during system initialization. Since the loaded kernel segments are part of every process, all processes use the restart_fault|0 value.

Return to Ring 0

The kernel must validate the machine conditions to be restarted. If acceptable, it restarts execution based upon the registers and instructions in those machine conditions.

Ring 0 Entrance Routine: restart_fault

restart_fault¹ is a gate into ring 0,² with ring brackets: 0,0,7. Like all gate segments, the first section of code is a transfer vector with an entry for each of the permitted gate entrypoints. The two entrypoints are:

  000000  tra restart_entry
  000001  tra cleanup_entry
 

1. Source for restart_fault is available for review in restart_fault.alm. Also needed are:
   • mc.incl.alm: the ALM version of the machine conditions structure.
   • mc.incl.pl1: the PL/I version of the machine conditions structure.
2. While restart_fault code is structured like a gate, and stored in a hardcore segment with gate ring brackets, it differs from normal gates (such as hcs_) in one important aspect. restart_fault has no entry in the file system. It is part of the Multics kernel, loaded by system initialization, and activated as a segment in every process. Since it has no file system entry, it cannot be called symbolically from a source program; without a file system entry, there is no way to snap (convert) the symbolic entrypoint name referenced in a program to an entry variable.

restart_fault operates with interrupts inhibited to prevent I/O interrupts and other low-priority faults from interrupting its restarting of faulting code.

restart_fault is co-framer (along with signaller) of storage in return_to_ring_0_'s stack frame. It knows the layout of the automatic variables in that stack frame; in particular, it uses the machine conditions (mc) structure located in this stack frame. And while signaller creates the stack frame, restart_fault removes it if the fault is restarted.³

3. If the faulting program is aborted using the release command, then the stack unwinder program removes the return_to_ring_0_ frame from the stack, after invoking its associated cleanup on-unit. Earlier, it was noted that signaller defined this on-unit as restart_fault|1, which invokes the cleanup_entry in this gate.

Validating User Ring Control Unit Data

restart_fault's restart_entry code prepares for a possible restart as follows:

• Copy the machine conditions from the return_to_ring_0_ stack frame to pds$signal_data. This storage is protected in ring 0, and wired in memory to avoid page faults while restarting from the fault.
  • The Pointer Registers are copied individually, to ensure that Addressing Unit pointer segment and ring number validation gets applied when copying the pointers between rings.
• Find the machine conditions in pds$mc_save_area matching the unique SCU index from the copied machine conditions.
• Compare the mc.scu data copied from the outer ring with data saved by signaller at the time of the fault.

The outer ring can make changes to the Control Unit Data to correct the cause of the fault, and request ring 0 to then restart using this corrected instruction data. restart_fault must ensure these changes:

• Request restart for fault conditions that are possible to restart. (Some fault conditions are not restartable.)
• Do not change from the original ring of execution, or segment containing the faulting execution point. The location within the executing segment (mc.scu.ic, the Instruction Counter field of Control Unit Data, word 4) may change.
• Do not change the segment or ring number of any address referenced in the Temporary Pointer Register (tpr). The offset value in this register (mc.scu.ca, the Computed Address field of the Control Unit Data, word 5) may change.

Therefore, restart_fault imposes the following restrictions on the Control Unit Data to be restarted.

• Is the copied Control Unit Data exactly equal to the saved data?
  • They may be totally equal unless the mid-instruction fault (MIF) flag is on, and the fault is an Illegal Procedure (IPR) fault.
  (For MME2 fault, debug has changed some of the data.)
• If the two Control Unit Data images differ:
  • Is this a restartable mid-instruction fault?
    • scu.ir.mif (mid-instruction fault) must be "0"b; or
    • both scu.cu.IF (fault occurred during instruction fetch) and scu.cu.RFI (refetch instruction) must be "1"b.
    (For MME2 fault, all of these flags = "0"b.)
  • Is this a non-recoverable fault?
    • Verify that scu.if.parm (parity mode) is "0"b.
    • Verify that scu.if.abs (running in absolute mode) is "0"b.
    (For MME2 fault, these flags are both clear.)
  • Is this an attempt to change the ring or segment of execution?
    • Verify there are no changes in the words 0-3 of Control Unit Data. (Words 4-7 may differ.)
    (For MME2 fault, there are none. debug changes were made in words 6 and 7.)
  The process is terminated if any of these tests fail.

Restarting the Verified Machine Conditions

After verification, restart_fault continues:

• If an overflow fault occurred, clear the affected mc.scu.indicators, in case the restart encounters another overflow situation.
• Copy the EIS restart data saved from the fault into pds$signal_data.eis_info. The user ring may not change this data.
• Pop (remove) the return_to_ring_0_ stack frame from the outer ring stack.

Execute a Restore Control Unit rcu instruction to load the verified, possibly modified Control Unit Data from mc.scu to the processor Control Unit internal registers. This restarts execution of the faulting program in its original ring of execution.

It is important to note that the rcu instruction, may only be used when running in privileged mode. Attempting to restore damaged or malicious Control Unit Data could:

• Change addressing and privilege modes of operation.
• Cause the CPU to stop all processing (hang or lockup).
• Transfer to arbitrary locations in arbitrary rings of execution.
• Read or store data at any location in memory.

Therefore, its use is restricted to the ring 0 kernel software.

debug does modify the machine conditions to run the original instruction at the breakpoint, which it copied from the breakmap into mc.scu.even_inst.

When the machine conditions are restarted, the hello program continues execution with the original instruction that was at the breakpoint, followed by the remaining instructions in the program.¹

1. Note that the Control Unit reads instructions from memory 2-words at a time, starting with an even location. Since most instructions fill one word, this means two instructions may be present in the Control Unit Data. The breakpoint examined here was at an even location. So the mme2 instruction and the lda that followed it were in the Control Unit when the fault occurred.

   When restarting from the machine conditions altered by debug, debug replaced the mme2 instruction causing the fault with the original staq instruction in the mc.scu.even_inst. It left the lda untouched in mc.scu.odd_inst. Thus, execution continued with: the two instructions loaded from machine conditions into the Control Unit; followed by remaining instructions starting at hello|32.

User Ring: Program Continues Execution

As the hello program continues execution, note that the breakpoint remains set in the object segment. The next traversal of this code path will also stop at the breakpoint.

The breakpoint remains in the object segment until: it is reset via the debug .br request; or until the source is recompiled.

Original 08 Feb 2017 GCD