I had to recover access to an ASR9000 recently, and it didn’t go as smoothly as password recovery on an IOS box. The procedure is different, the answer depends on which flavor of IOS XR the router runs, and the useful details are spread across a Cisco ROMMON guide, a long-running Cisco Community article and a handful of forum threads. These are the notes I wish I’d had in one place. I keep them here as my own reference.
Standard disclaimer: this needs physical console access and should only ever be done on equipment you’re responsible for. Schedule a maintenance window, because every method below involves reloading the router.
First: what are you actually locked out of?
There are two different problems that look the same from the login prompt, and they have different fixes:
- You’ve lost the local password. Nobody knows the local admin credentials. The config register method (0x142) handles this on classic XR.
- AAA has locked you out. Login is pointed at TACACS+ or RADIUS with no local fallback, and the server is unreachable or rejecting you. 0x142 will not fix this, because it doesn’t skip the running configuration. You need one of the config-bypass methods further down.
Second: classic (32-bit) or 64-bit IOS XR?
The ASR9000 family has run two very different IOS XR architectures:
| Classic XR (32-bit, “cXR”) | 64-bit XR (“eXR”) | |
|---|---|---|
| Underlying OS | QNX | Linux, with separate System Admin and XR VMs |
| Boot interrupt lands in | ROMMON (rommon B1 >) |
GRUB |
| Config register trick | Works (0x142) | Does not work |
| Recovery options | 0x142, ROMMON variables, ksh | Via the sysadmin VM, the system recovery feature where supported, or a reimage |
If the console shows a sysadmin-vm prompt anywhere during boot, or you land in a GRUB menu instead of ROMMON, you’re on 64-bit XR. Skip to that section.
Before you start
- Console cable on the active RSP. The RSP console defaults to 9600 baud, 8N1, no flow control. If you see garbage, check the baud rate before anything else.
- Know how to send a break from your terminal program. In PuTTY it’s Ctrl+Break, or right-click the title bar, then Special Command, then Break. In SecureCRT it’s Ctrl+Break or the Send Break option. On a laptop without a Break key, use the menu option.
- One RSP or two? With redundant RSPs you need a console on both, or the standby will take over while you’re working on the active one.
- Have the config backed up off-box if at all possible. Some of the methods below boot the router with an empty configuration.
Classic XR: lost local password (config register 0x142)
This is the procedure for when the problem is just a forgotten local password.
Single RSP
- Power-cycle or reload the router and send a break during the first part of the boot to land in ROMMON.
- Set the config register and reset:
rommon B1 > confreg 0x142 rommon B2 > reset - Let it boot. Instead of the normal login, you get the administrative user dialog:
Enter root-system username: newadmin Enter secret: Enter secret again: - Log in with the new account. The config register goes back to its previous value (normally 0x102) once the new user is saved. Confirm it in
show version, which lists the configuration register for each RSP:RP/0/RSP0/CPU0:router# show version | include register
The important difference from IOS: 0x142 is not 0x2142. On IOS, 0x2142 boots without the startup config. On IOS XR, 0x142 keeps the full configuration and only adds the “create a root-system user” prompt. Your existing config, including any broken AAA, is still there.
Redundant RSPs
With two RSPs, the standby will happily take over while the active is sitting in ROMMON. So:
- Get both RSPs into ROMMON (console and break on each).
- On the RSP you want to come up active, set
confreg 0x142andreset. Complete the root-system user prompt as above. - Once that RSP is up and you’re logged in, boot the standby RSP normally (config register 0x102) so it rejoins as standby.
Classic XR: locked out by AAA (TACACS+/RADIUS)
If the problem is an AAA config like this one, which has no local fallback:
aaa authentication login default group tacacs+
then the router needs to boot without, or around, that configuration. There are three approaches, from least to most destructive.
Option 1: ksh from the standby RSP console
On the standby RSP’s console, press ESC and type ksh. If you can log in with any local account, you can save the running config to disk, edit out the bad AAA lines, and either roll back:
# nvgen -c -l 1 -t 1 -o 1 > /harddisk:/backupconfig.txt
# config_rollback -n 0x1
or bypass AAA and get an exec prompt directly:
# /pkg/bin/exec -a
RP/0/RSP0/CPU0:router#
Option 2: AUX_AUTHEN_LEVEL=0
If ksh also wants credentials you don’t have, set this in ROMMON on the RSP. It lets the AUX port drop straight to ksh on boot with no login prompt:
rommon B1 > AUX_AUTHEN_LEVEL=0
rommon B2 > sync
rommon B3 > reset
Then, on the AUX port, /pkg/bin/exec -a gives you an exec prompt, and plain config drops you into config mode, where you can fix the AAA lines.
Option 3: boot with no configuration
Point the router at config files that don’t exist. The running (exec) configuration:
rommon B1 > IOX_CONFIG_FILE=/harddisk:/no-config
rommon B2 > sync
rommon B3 > reset
If you also need to clear the local users, which live in the admin configuration:
rommon B1 > IOX_ADMIN_CONFIG_FILE=/disk0:/none
rommon B2 > sync
rommon B3 > reset
With both set, the router comes up with a blank config and prompts for a new root-system user. Then load your saved configuration back in and fix whatever caused the lockout.
Clean up the ROMMON variables (don’t skip this)
Variables set with sync are persistent. If you leave them, the next reload boots without your config again. Remove them once you’re back in:
rommon B1 > unset IOX_CONFIG_FILE
rommon B2 > unset IOX_ADMIN_CONFIG_FILE
rommon B3 > unset AUX_AUTHEN_LEVEL
rommon B4 > sync
A leftover variable shows itself with errors like this after the next boot:
%SECURITY-LOCALD-3-LWA_ADD_FAIL : Failed to add the username admin to
lightweight authentication password database: No such file or directory
Quirk: an unexpected “Enter root-system username” prompt
Some routers drop into the root-system user prompt on their own (Cisco bug CSCth03923), and then a username and password you know are correct fail. The workaround is to enter a new username that definitely isn’t already configured.
64-bit IOS XR (eXR)
None of the ROMMON tricks apply here, and there’s no config-register bypass. Your options, in order:
- Go in through the System Admin VM. If you still know the sysadmin credentials (often different from the XR ones), you can reach the XR VM from there and bypass XR authentication. On the console, press Ctrl+O to toggle to the sysadmin VM and log in. Then:
sysadmin-vm:0_RP0# show sdr sysadmin-vm:0_RP0# run [sysadmin-vm:0_RP0:~]$ ssh <XR VM IP from show sdr> [xr-vm_node0_RP0_CPU0:~]$ exec -a RP/0/RP0/CPU0:router#From that prompt you can add or fix users and AAA config, then
commit. - The
system recoveryfeature. Newer XR releases have a “Cisco IOS XR-recovery” GRUB entry that resets to a blank config with new root credentials. It only works ifsystem recoverywas configured before you got locked out. It erases the configuration and user files, and Cisco documents it for the 8000 series and NCS 5500. Check your ASR9000 release notes before counting on it. - Reimage. If neither works, the fallback is a USB or iPXE boot and a fresh install, then restoring your config from backup.
Afterwards
- Save it the XR way. Changes take effect with
commit. There’s nocopy run startstep like on IOS. Take an off-box copy too:RP/0/RSP0/CPU0:router# copy running-config tftp://<server>/router-backup.cfg - Add a local fallback to AAA so a dead TACACS+ server can’t lock you out again:
aaa authentication login default group tacacs+ local - Store the root-system credentials somewhere you’ll find them, such as a password manager, not a sticky note.
- Double-check the ROMMON variables and config register on both RSPs before you close the change.
For telling the IOS variants apart in general, see Cisco IOS vs IOS-XE vs IOS-XR. To keep management access locked down properly once you’re back in, see securing the management plane (written for IOS-XE, but most of it applies).
References
- ASR9000/XR: How to reset a lost password (Cisco Community, Xander Thuijs)
- New password recovery procedure on Cisco IOS XR (Cisco,
system recovery)
Leave a Reply