Network Black Start

Black Start is a term used in the UK electricity system as the name of the plan to restart the grid in the event of total failure, this process requires power to restart some station so the procedure requires pre-planning to work. We can take this idea and apply this strategy to re-starting a network. The assumption is that the configuration and the code on every device is potentially compromised, and all the passwords have been changed or deleted. Therefore, every device will need to be recovered, new code applied and configurations restored. This technique can also be applied to the rest of your datacentre infrastructure.

Prerequisites for Recovery

Device Inventory - To know all the recovery process required
Password Recovery Process - For each of the device types
Software Recovery Process - For each of the device types
Configuration Archive - For each of the devices
Software Archive - For each of the devices
Device Recovery Tools - Serial access, management network with Dynamic Host Configuration Protocol (DHCP), Trivial File Transfer Protocol (TFTP) access to configurations and software

Recovery Process

For each device the recovery process will need to be followed. In the case of a Cisco router, the device needs to be power cycled and at the correct time the break signal needs to be sent to interrupt the normal boot process to allow the startup configuration to be bypassed and allow access to the device. Then the password can be reset and old configuration reloaded. This can be a laborious process even with remove serial access, there still need to be coordination between a technician in the datacentre and the network engineer performing the reset.

Automated Recovery Process

The answer to this is automation, using Python and pySerial it is possible to create scripts that can wait listening to a serial port for a device to be rebooted and when the correct point in the boot process is reached to send the commands to interrupt the boot sequence and run the commands to bypass the loading of the startup configuration ready for recovery. With more work it would be possible to identify the device and put the correct code and configuration back on.

Example Code

# recover_cisco_routre.py
import serial
from time import sleep
def send_to_console(ser: serial.Serial, command: str, wait_time: float = 0.5):
    command_to_send = command + "\r"
    ser.write(command_to_send.encode('utf-8'))
    sleep(wait_time)
    print(ser.read(ser.inWaiting()).decode('utf-8'), end="")

with serial.Serial("/dev/ttyUSB0", timeout=100) as ser:
    print(f"Connecting to {ser.name}...")
    inp_data = ser.read_until(b"ROMMON initialized")
    ser.send_break()
    print(inp_data)
    send_to_console(ser, "")
    send_to_console(ser, "confreg 0x2142")
    send_to_console(ser, "reset")
    inp_data = ser.read_until(b"initial configuration dialog? [yes/no]")
    print(inp_data)
    send_to_console(ser, "no", wait_time=2)
    send_to_console(ser, "")
    send_to_console(ser, "enable")
#    send_to_console(ser, "copy startup-config running-config")
#    send_to_console(ser, "config term")
#    send_to_console(ser, "enable secret ")
#    send_to_console(ser, "line vty 0 4")
#    send_to_console(ser, "password ")
#    send_to_console(ser, "exit\nexit")
#    send_to_console(ser, "copy running-config startup-config")
    send_to_console(ser, "config term\nconfig-register 0x2102\nexit\nno")
    print(f"Connection to {ser.name} closed.")

Summary

The process or network recovery requires a large amount of planning before it can be enacted. Having out of band network and serial access to all devices with speed thing up but it is essential to have an archive of both configuration and software to be able to restore all of your devices. This does not have to be limited to network devices, severs and Virtual Machine (VM) platforms can be included too. The software archive is important in case support contracts have lapsed or older devices are no longer supported.