crash —
UNIX system failures
This section explains what happens when the system crashes and (very briefly)
  how to analyze crash dumps.
When the system crashes voluntarily it prints a message of the
    form
panic: why i gave up the
  ghost
on the console, takes a dump on a mass storage peripheral, and
    then invokes an automatic reboot procedure as described in
    reboot(8). (If auto-reboot
    is disabled on the front panel of the machine the system will simply halt at
    this point.) Unless some unexpected inconsistency is encountered in the
    state of the file systems due to hardware or software failure, the system
    will then resume multi-user operations.
The system has a large number of internal consistency checks; if
    one of these fails, then it will panic with a very short message indicating
    which one failed. In many instances, this will be the name of the routine
    which detected the error, or a two-word description of the inconsistency. A
    full understanding of most panic messages requires perusal of the source
    code for the system.
The most common cause of system failures is hardware failure,
    which can reflect itself in different ways. Here are the messages which are
    most likely, with some hints as to causes. Left unstated in all cases is the
    possibility that hardware or software error produced the message in some
    unexpected way.
  - iinit
- This cryptic panic message results from a failure to mount the root
      filesystem during the bootstrap process. Either the root filesystem has
      been corrupted, or the system is attempting to use the wrong device as
      root filesystem. Usually, an alternative copy of the system binary or an
      alternative root filesystem can be used to bring up the system to
      investigate.
- Can't exec /sbin/init
- This is not a panic message, as reboots are likely to be futile. Late in
      the bootstrap procedure, the system was unable to locate and execute the
      initialization process,
      init(8). The root
      filesystem is incorrect or has been corrupted, or the mode or type of
      /sbin/init forbids execution.
- IO err in push
-  
- hard IO err in swap
- The system encountered an error trying to write to the paging device or an
      error in reading critical information from a disk drive. The offending
      disk should be fixed if it is broken or unreliable.
- realloccg: bad optim
-  
- ialloc: dup alloc
-  
- alloccgblk: cyl groups corrupted
-  
- ialloccg: map corrupted
-  
- free: freeing free block
-  
- free: freeing free frag
-  
- ifree: freeing free inode
-  
- alloccg: map corrupted
- These panic messages are among those that may be produced when filesystem
      inconsistencies are detected. The problem generally results from a failure
      to repair damaged filesystems after a crash, hardware failures, or other
      condition that should not normally occur. A filesystem check will normally
      correct the problem.
- timeout table overflow
- This really shouldn't be a panic, but until the data structure involved is
      made to be extensible, running out of entries causes a crash. If this
      happens, make the timeout table bigger.
- KSP not valid
-  
- SBI fault
-  
- CHM? in kernel
- These indicate either a serious bug in the system or, more often, a glitch
      or failing hardware. If SBI faults recur, check out the hardware or call
      field service. If the other faults recur, there is likely a bug somewhere
      in the system, although these can be caused by a flakey processor. Run
      processor microdiagnostics.
- machine check %x: description
-  
-    machine dependent machine-check information
- Machine checks are different on each type of CPU. Most of the internal
      processor registers are saved at the time of the fault and are printed on
      the console. For most processors, there is one line that summarizes the
      type of machine check. Often, the nature of the problem is apparent from
      this message and/or the contents of key registers. The VAX Hardware
      Handbook should be consulted, and, if necessary, your friendly field
      service people should be informed of the problem.
- trap type %d, code=%x, pc=%x
- A unexpected trap has occurred within the system; the trap types are:
    
    
0	reserved addressing fault
1	privileged instruction fault
2	reserved operand fault
3	bpt instruction fault
4	xfc instruction fault
5	system call trap
6	arithmetic trap
7	ast delivery trap
8	segmentation fault
9	protection fault
10	trace trap
11	compatibility mode fault
12	page fault
13	page table fault
    
 The favorite trap types in system crashes are trap types 8 and
        9, indicating a wild reference. The code is the referenced address, and
        the pc at the time of the fault is printed. These problems tend to be
        easy to track down if they are kernel bugs since the processor stops
        cold, but random flakiness seems to cause this sometimes. The debugger
        can be used to locate the instruction and subroutine corresponding to
        the PC value. If that is insufficient to suggest the nature of the
        problem, more detailed examination of the system status at the time of
        the trap usually can produce an explanation. 
- init died
- The system initialization process has exited. This is bad news, as no new
      users will then be able to log in. Rebooting is the only fix, so the
      system just does it right away.
- out of mbufs: map full
- The network has exhausted its private page map for network buffers. This
      usually indicates that buffers are being lost, and rather than allow the
      system to slowly degrade, it reboots immediately. The map may be made
      larger if necessary.
That completes the list of panic types you are likely to see.
When the system crashes it writes (or at least attempts to write)
    an image of memory into the back end of the dump device, usually the same as
    the primary swap area. After the system is rebooted, the program
    savecore(8) runs and
    preserves a copy of this core image and the current system in a specified
    directory for later perusal. See
    savecore(8) for
  details.
To analyze a dump you should begin by running
    adb with the -k flag on the
    system load image and core dump. If the core image is the result of a panic,
    the panic message is printed. Normally the command “$c” will
    provide a stack trace from the point of the crash and this will provide a
    clue as to what went wrong. For more detail see “Using ADB to Debug
    the UNIX Kernel”.
gdb(1),
  reboot(8)
“VAX 11/780 System Maintenance Guide” and “VAX Hardware
  Handbook” for more information about machine checks.
“Using ADB to Debug the UNIX Kernel”