vCenter ESXi /bootbank Alarm After Disk Replacement: Field Troubleshooting Checklist
Direct answer: If vCenter starts sending repeated ESXi /bootbank alarms after a host disk replacement, do not assume the new disk alone fixed the incident. Treat the alert as a host boot-device or bootbank-access problem until the host has been placed in maintenance mode, restarted cleanly, and monitored long enough to prove that the recurring alarm has stopped.
This field note is based on a real operational pattern: vCenter first reported a hardware object alarm on one ESXi host, a failed physical disk was found and replaced, but repeated /bootbank warning emails continued afterward. The host was eventually placed into maintenance mode with the hardware maintenance vendor, rebooted, and only after some additional monitoring time did the alarm mail stop. The host then returned to service without further recurrence.
Observed symptom timeline
| Stage | What was seen | Operational meaning | Risk if ignored |
|---|---|---|---|
| Initial alarm | vCenter reported another hardware object alarm on a specific host. | The incident began as a hardware-health event, not as a guest VM symptom. | The real failed part may be missed if the team only clears vCenter alarms. |
| Disk replacement | One failed disk was identified and replaced. | The obvious hardware fault was addressed. | Storage or boot media state can still remain inconsistent after replacement. |
| Recurring warning | /bootbank error mails arrived every few minutes or roughly hourly. | The host continued to report a bootbank access or validation problem. | Alarm fatigue can hide a host that may fail during the next reboot or patch. |
| Maintenance action | The host entered maintenance mode and was rebooted with vendor involvement. | The safest remediation was performed after workload evacuation. | Rebooting without maintenance mode risks avoidable VM impact. |
| Validation | The alarm stopped after the reboot and monitoring period. | The host state normalized after a clean restart. | Declaring success too early can miss delayed recurrence. |
Why a disk replacement may not immediately stop a bootbank alarm
In ESXi, /bootbank is tied to the host boot environment, image state, and boot-device accessibility. A vCenter hardware alarm and a /bootbank alarm can appear close together, but they are not always the same root cause. Broadcom documents cases where vCenter reports that bootbank cannot be found, and those cases can involve boot device communication, boot media corruption, or other resource conditions rather than a simple “replace disk and clear alarm” workflow.
That distinction matters in a VMware or VDI platform. A host can look serviceable after the failed disk is replaced, while the management layer continues to report warning evidence. If that host supports Horizon, VDI, databases, or other dense workloads, the cost of guessing is high: the next maintenance reboot may expose a boot problem that was only being hinted at by recurring mail alerts.
Evidence to collect before taking the host down
| Evidence | Where to check | What it helps prove | Do not do yet |
|---|---|---|---|
| Exact vCenter alarm text and frequency | vCenter alarms and email history | Whether the issue is continuous, periodic, or tied to a scheduled host task. | Do not simply acknowledge and forget the alarm. |
| Hardware storage status | Server management controller, RAID tool, vendor console | Whether the replaced disk is healthy and the array or controller is stable. | Do not assume replacement equals host recovery. |
| ESXi bootbank log entries | vmkernel.log, vmkwarning.log, vobd.log | Whether ESXi is reporting path, corruption, or access errors around bootbank. | Do not reboot before preserving useful evidence if the host is unstable. |
| Workload evacuation readiness | DRS, vMotion, cluster capacity, maintenance window | Whether maintenance mode can be entered without service impact. | Do not reboot a production host with running VMs unless there is an emergency. |
| Recent changes | Disk replacement time, firmware, driver, patch, boot media history | Whether the alarm aligns with hardware maintenance or another platform change. | Do not apply multiple changes at once. |
Safe remediation workflow
- Confirm the hardware event. Record the original vCenter hardware alarm, the failed disk slot, replacement time, and vendor maintenance notes.
- Check whether the
/bootbankalarm continues after the disk replacement. In this case, warning emails continued as frequently as within five minutes or roughly once per hour. - Do not clear the symptom as “resolved” too early. A recurring bootbank alert after hardware replacement means the host still deserves controlled maintenance.
- Plan maintenance mode. Evacuate VMs through DRS or manual vMotion and make sure cluster capacity can absorb the host.
- Reboot the ESXi host only after evacuation. The reboot should be treated as a controlled state refresh and validation step, not a blind fix.
- Wait after the reboot. In this case, the alarm mail did not necessarily stop the instant the host returned; the important signal was that it stopped after a short observation period.
- Keep monitoring before returning to normal operations. Watch vCenter alarms, host health, storage health, and management logs before declaring the host stable.
What to rule out if the alarm comes back
- Boot media or boot device instability, especially if the host boots from USB, SD, SATADOM, or mirrored internal media.
- RAID/controller state that remained degraded after disk replacement.
- Locker partition or ESXi bootbank corruption indicators in
vmkwarning.logorvobd.log. - A scheduled ESXi task or backup-state operation repeatedly touching
/bootbank. - AIO handle exhaustion or resource-specific conditions, especially in hosts running many vGPU-enabled VMs, because Broadcom documents a separate bootbank-alert scenario for that pattern.
- Firmware, driver, or ESXi version issues that align with the hardware replacement window.
Validation and rollback criteria
The host should not be treated as stable just because it exits maintenance mode. A better validation standard is: no recurring /bootbank alarm emails, no new host hardware alarms, clean host reconnect behavior in vCenter, normal datastore and boot-device health, and no abnormal VM task failures after the host rejoins the cluster.
If the alarm returns, rollback does not mean undoing the disk replacement. It means removing the host from normal workload placement again, preserving logs, re-entering maintenance mode if needed, and escalating with the vendor using a clean timeline: original hardware alarm, disk replacement, first bootbank warning after replacement, reboot time, alarm-stop time, and recurrence time.
Operator takeaway
The useful lesson is not “reboot fixes bootbank.” The better lesson is that a hardware replacement can fix the visible failed component while ESXi still needs a controlled maintenance cycle to normalize bootbank access and alarm state. For production VMware and VDI infrastructure, the safest pattern is evidence first, maintenance mode second, reboot third, and monitoring last.
Comments
Post a Comment