Why Cat9K Upgrades Break — and How to Get Them Right
The Cisco Catalyst 9000 series is the backbone of countless enterprise networks. Catalyst 9300s stacking in IDF closets, 9400s serving as distribution switches, 9500s anchoring the core. When it’s time to upgrade IOS-XE on these platforms, the process looks deceptively simple: copy the image, set the boot variable, reload. In practice, experienced engineers know better. Cat9K upgrades have a specific set of failure modes that bite the unprepared — corrupted ROMMON variables, failed ISSU pre-checks, stacked switches that partially upgrade, and the dreaded situation where a switch boots into an unexpected install mode. This guide walks through the entire upgrade workflow, the mistakes that cause outages, and how to verify every step of the way.
Understanding Install Mode vs. Bundle Mode
Before touching an upgrade, you need to know which boot mode the switch is in. Cat9K platforms support two modes:
- Bundle mode: The switch boots from a monolithic .bin file. Simple, but slower to boot and uses more flash space. This is legacy behavior.
- Install mode: The image is expanded into packages stored on flash. Faster boot times, supports In-Service Software Upgrade (ISSU), and is the Cisco-recommended method for 16.12+.
To check which mode you’re in:
SW1# show version | include Install Mode
Switch 1 : V01 : Provisioning File : cat9k_iosxe.17.09.04a.SPA.confIf you see a .conf file, you’re in install mode. If you see a .bin path, you’re in bundle mode.
You can also verify with:
SW1# show boot
BOOT variable = flash:packages.conf;
Configuration Register is 0x102Install mode uses packages.conf as the boot variable. If the boot variable points directly to a .bin file, you’re in bundle mode.
Pre-Upgrade Checklist
Never start an upgrade without running through this checklist. Most outages happen because someone skipped a step.
1. Verify Available Flash Space
SW1# dir flash: | include bytes
21217476 bytes total (8234567 bytes free)A full Cat9K IOS-XE 17.x image is roughly 900 MB to 1.1 GB. If you’re in install mode, the expanded packages take even more space. Rule of thumb: you need at least 1.5x the image size free before you begin. If flash is tight, clean up old packages first:
SW1# request platform software package clean switch all file flash:This removes inactive package files. Run show platform software package first to see what’s active vs. inactive before cleaning.
2. Verify Stack Health (if applicable)
SW1# show switch
Switch/Stack Mac Address : 000c.29ab.1234 - Local Mac Address
Mac persistency wait time: Indefinite
H/W Current
Switch# Role Mac Address Priority Version State
------------------------------------------------------------
*1 Active 000c.29ab.1234 15 V02 Ready
2 Member 000c.29ab.5678 1 V02 Ready
3 Member 000c.29ab.9abc 1 V02 ReadyAll stack members must be in Ready state. Any member showing Provisioning, Waiting, or Error will cause the upgrade to fail or partially complete — the worst possible outcome for a stack.
3. Check for Active ISSU Compatibility
SW1# show issu state detailISSU (In-Service Software Upgrade) allows you to upgrade a stacked switch without a full reload, minimizing downtime to sub-second for most traffic. But ISSU only works between compatible versions. Before attempting ISSU, verify on Cisco’s compatibility matrix that your current version and target version support it. As a general rule, ISSU is supported for maintenance releases within the same train (e.g., 17.06.x to 17.06.x+1) but NOT across major trains (e.g., 16.12 to 17.09).
4. Back Up Your Configuration
This seems obvious, but it still gets skipped:
SW1# copy running-config tftp://10.0.0.5/SW1-backup-20261021.cfg
SW1# copy startup-config flash:startup-backup-20261021.cfgKeep a copy on the switch itself in case TFTP isn’t reachable during recovery.
The Install Mode Upgrade Workflow
Assuming you’re already in install mode (which you should be on any Cat9K running 17.x), here’s the complete upgrade procedure.
Step 1: Copy the Image to Flash
SW1# copy tftp://10.0.0.5/cat9k_iosxe.17.12.01.SPA.bin flash:
Destination filename [cat9k_iosxe.17.12.01.SPA.bin]?
Accessing tftp://10.0.0.5/cat9k_iosxe.17.12.01.SPA.bin...
Loading cat9k_iosxe.17.12.01.SPA.bin from 10.0.0.5:
!!!!!!!!!!!!!!!!!!!!!!!!!
[OK - 1073741824 bytes]After copying, verify the MD5 hash against Cisco’s software portal:
SW1# verify /md5 flash:cat9k_iosxe.17.12.01.SPA.bin
..................................................
verify /md5 (flash:cat9k_iosxe.17.12.01.SPA.bin) = a1b2c3d4e5f6...Do not skip the MD5 check. A corrupted image that passes the install step but fails at boot leaves your switch in ROMMON — not a place you want to be at 2 AM.
Step 2: Install the Image (Add + Activate + Commit)
The install workflow has three sub-commands. Understanding each is critical:
SW1# install add file flash:cat9k_iosxe.17.12.01.SPA.bin activate commit prompt-level noneYou can also run them separately for more control:
! Step 1: Expand and validate the image
SW1# install add file flash:cat9k_iosxe.17.12.01.SPA.bin
! Step 2: Stage the new image (will prompt for reload)
SW1# install activate
! Step 3: After reboot, commit (makes it persistent)
SW1# install commitCritical gotcha: If you reload without running install commit, the switch will roll back to the previous version on the next reload. This is actually a safety feature — but it catches engineers off guard. Always run install commit after verifying the switch came up correctly.
Step 3: Monitor the Reload
During a stack reload, watch the console carefully. Each stack member will reload sequentially (in ISSU mode) or simultaneously (in standard reload). After each member boots, verify:
SW1# show version | include Version
Cisco IOS XE Software, Version 17.12.01
Cisco IOS Software [Dublin], Catalyst L3 Switch Software (CAT9K_IOSXE)SW1# show install summary
[ Switch 1 ] Installed Package(s) Information:
State (St): I - Inactive, U - Activated & Uncommitted,
C - Activated & Committed, D - Deactivated & Uncommitted
Type St Filename/Version
---- -- --------
IMG C 17.12.01.0.6The C (Committed) state confirms the upgrade is complete and stable. U (Uncommitted) means you still need to run install commit.
Common Failure Scenarios and Fixes
Failure 1: Stack Member Stuck in “Provisioning” State
Symptom: After an upgrade, one or more stack members shows Provisioning in show switch and never comes online.
Root cause: Version mismatch between the active switch and a member. This commonly happens when a stack member doesn’t receive the new image correctly, or when a member was added to the stack after the upgrade without being updated.
Fix:
! Force auto-upgrade of the member
SW1# request platform software package install switch 2 file flash:cat9k_iosxe.17.12.01.SPA.bin
! Or enable auto-upgrade globally
SW1(config)# switch 2 provision C9300Cat9K stacks support automatic version synchronization — the active switch will push its image to any new member. Enable this on all production stacks:
SW1(config)# software auto-upgrade enableFailure 2: ROMMON Boot After Upgrade
Symptom: Switch boots into ROMMON prompt instead of IOS-XE after an upgrade attempt.
Root cause: Usually one of three causes — corrupted image, incorrect boot variable, or insufficient flash space during the install.
Recovery from ROMMON:
rommon 1 > dir flash:
rommon 2 > boot flash:cat9k_iosxe.17.09.04a.SPA.binIf the previous image is still on flash, boot it directly from ROMMON. Once back in IOS-XE, clean up flash, re-verify space, and retry the install. If the previous image was cleaned before the failed upgrade, you’ll need to TFTP boot:
rommon 1 > IP_ADDRESS=10.0.0.10
rommon 2 > IP_SUBNET_MASK=255.255.255.0
rommon 3 > DEFAULT_GATEWAY=10.0.0.1
rommon 4 > TFTP_SERVER=10.0.0.5
rommon 5 > TFTP_FILE=cat9k_iosxe.17.09.04a.SPA.bin
rommon 6 > tftpdnldThis is why you never delete the previous working image until the upgrade is committed and verified. Keep two versions on flash unless space is critically short.
Failure 3: ISSU Pre-Check Failures
Symptom: install activate issu fails with pre-check errors.
SW1# show issu state detail
--- Starting installation state synchronization ---
[1]: Finished installation state synchronization
--- Starting local lock acquisition on switch 1 ---
[1]: Finished local lock acquisition on switch 1
--- Starting installation changes ---
[1] ISSU Not Supported: Incompatible versionsCommon ISSU failure reasons:
- Cross-train upgrade (e.g., 16.12 → 17.09) — ISSU not supported across trains
- Feature set mismatch (DNA Advantage vs. DNA Essentials)
- Stack member version mismatch — all members must be on the same version before ISSU
- Active spanning tree topology changes in progress — wait for STP to converge before attempting ISSU
If ISSU isn’t available, fall back to a standard reload with a maintenance window. Plan your change window carefully, communicate to stakeholders, and have a rollback plan tested and documented before you start.
Failure 4: Corrupted packages.conf
Symptom: Switch boots but throws errors about packages.conf, or show install summary shows inconsistent state.
Fix:
! Rebuild packages.conf from installed packages
SW1# request platform software package install switch all file flash: clean
! Verify
SW1# show install summary
SW1# show bootIn persistent cases, manually set the boot variable:
SW1(config)# no boot system
SW1(config)# boot system flash:packages.conf
SW1# write memoryChoosing the Right IOS-XE Release Train
Picking the wrong IOS-XE release is one of the most avoidable causes of post-upgrade pain. Cisco’s IOS-XE release strategy for Cat9K follows a predictable pattern worth understanding before you commit to a version.
Standard Maintenance Releases (SMR) are the most common. They ship on a quarterly cadence and include bug fixes plus incremental feature additions. Most enterprise networks should target the latest SMR of an established train — for example, 17.09.x or 17.12.x are mature trains with long fix histories.
Long-Term Support (LTS) releases receive an extended 36-month maintenance window. If your organization prioritizes stability over new features, running an LTS release (look for the “LTS” designation in Cisco’s release notes) gives you a longer support runway without forcing frequent upgrades.
Before selecting a version, always cross-reference against Cisco’s Recommended Releases document for the Cat9300, Cat9400, or Cat9500 (platform-specific). This document flags releases with known high-severity bugs and highlights which SMRs are considered stable for production deployment.
! After upgrading, confirm the train designation
SW1# show version | include IOS XE
Cisco IOS XE Software, Version 17.12.01
Cisco IOS Software [Dublin], Catalyst L3 Switch Software (CAT9K_IOSXE), Version 17.12.1The release naming convention matters: 17.12.01 represents train 17.12, the first SMR in that train. A subsequent release of 17.12.02 would carry additional bug fixes. Always check the bug ID list for your current version against the target — sometimes a single fixed bug that directly affects your environment justifies an upgrade to the next maintenance release.
Post-Upgrade Verification
After a successful upgrade and commit, run through this verification sequence:
! Verify version on all stack members
SW1# show version | include Version
SW1# show switch detail
! Verify all interfaces are up
SW1# show interfaces status | exclude notconnect
! Verify routing table is intact
SW1# show ip route summary
! Verify spanning tree is converged
SW1# show spanning-tree summary
! Check for any error messages
SW1# show logging | include %SYS|%LINK|%OSPF|%BGPPay particular attention to the logging output. STP topology changes, BGP session resets, or OSPF adjacency flaps immediately after an upgrade are worth investigating. Sometimes a new IOS-XE version changes default timer values or behavior — always review the release notes for your target version before upgrading production switches.
Rollback: When Things Go Wrong
One of the best features of install mode is clean rollback capability. If you haven’t run install commit yet, a simple reload rolls back automatically. If you have committed but need to go back:
! Check available rollback points
SW1# show install rollback
! Roll back to a previous version
SW1# install rollback to committed
! Or roll back to a specific version
SW1# install rollback to version 17.09.04aThis is why keeping the previous image on flash is non-negotiable until you’ve fully validated the upgrade over at least one complete business cycle. A latent bug that only manifests under specific traffic conditions might not show up until the next day.
Automating Cat9K Upgrades at Scale
If you manage dozens of Cat9K switches, manual upgrades quickly become untenable. Python-based automation with Netmiko or NAPALM can parallelize the process significantly. The key is building in the same pre-check logic you’d run manually — flash space verification, stack health checks, and post-upgrade validation. Our guide on network automation with Python, Netmiko, and NAPALM covers the framework for building this kind of workflow.
Cisco’s Network Services Orchestrator (NSO) and DNA Center also have native upgrade orchestration for Cat9K — worth considering if your environment is already using those platforms. DNA Center’s image management can handle ISSU across large switch deployments with pre-checks, scheduling, and rollback built in.
Key Lessons Learned
- Always verify flash space before starting — a failed install mid-process can leave a switch in an inconsistent state.
- Run
install commitimmediately after verifying a successful upgrade — don’t leave it uncommitted overnight. - Keep the previous image on flash until you’re confident the new version is stable.
- Understand your stacking topology — a partially-upgraded stack with version mismatches will cause spanning tree and protocol issues that are hard to diagnose without knowing the upgrade state.
- ISSU is not always available — check compatibility before committing to a zero-downtime upgrade window.
- Read the release notes for known issues, resolved bugs, and behavior changes in your target version.
Cat9K upgrades are routine once you’ve been through them a few times, but the consequences of getting it wrong in a production environment are significant. Understanding the install mode workflow, knowing the common failure points, and having a tested rollback plan makes the difference between a smooth maintenance window and an all-hands-on-deck outage. For related topics on managing and troubleshooting your Cisco infrastructure, see our posts on OSPF troubleshooting on IOS-XE and Cisco EEM scripts for automation.
Leave a Reply