Skip to content

fix: one unhealthy drive no longer fails its controller's inventory - #107

Open
g-carre wants to merge 4 commits into
test/ARTESCA-17960-disk-failure-scenario-gherkinfrom
fix/ARTESCA-17960-unhealthy-drive-inventory
Open

g-carre wants to merge 4 commits into
test/ARTESCA-17960-disk-failure-scenario-gherkinfrom
fix/ARTESCA-17960-unhealthy-drive-inventory

Conversation

@g-carre

@g-carre g-carre commented Oct 15, 2026 •

Copy link
Copy Markdown
Contributor

Commits, one per adapter fix: megaraid, ssacli, storcli2 drives, storcli2 volumes.

Fixes ARTESCA-17960 (RD-2154). Stacked on #105: it targets #105's branch, so the diff only shows the fix. Retarget it to main once #105 is merged. It replaces #103 (stacked on the closed YAML PR #102) and supersedes #91.

Problem

One unhealthy drive could fail the inventory of its whole controller, so every healthy drive on that controller lost its device paths. On the RD-2154 node this blanked all disks in the UI.

Fixes

  • megaraid (the RD-2154 case). When the drive of a single-drive RAID0 volume fails, the volume goes offline (OfLn, Exposed to OS: No) and is no longer a block device. getPaths still tried to resolve its paths, failed, and failed LogicalVolumes for the whole controller.
    • A volume reported Exposed to OS: No now gets empty paths without error. Only this explicit value (spaces trimmed) skips resolution, since older storcli versions may not report the field.
    • OfLn is now reported Failed instead of Unknown.
  • ssacli (from fix: a single failed drive no longer blanks a controller's device paths #91). A failed or pulled drive whose /dev node is gone made the lsblk lookup fail, which aborted the whole inventory. When the device node is gone, a lookup failure now marks the device's usage as unknown instead. Any other lookup failure, such as lsblk missing or failing on a device that exists, is still an error, so it cannot silently blank every drive. A free drive whose usage is unknown is reported Unknown, never UnassignedGood, so a drive that may carry data is never offered for a new volume.
  • storcli2 drives (from fix: a single failed drive no longer blanks a controller's device paths #91). A Used JBOD drive whose by-id link or device node is missing is kept with empty paths instead of failing the whole inventory. Any other resolution failure, such as a permission error, is still an error.
  • storcli2 volumes. The storcli2 scenario of test: describe hardware scenarios in Gherkin, play them on the adapters #105 found that the offline volume of a failed drive was still reported with a permanent path built from its NAA Id, although it has no block device. A volume reported Exposed to OS: No now has no paths: neither the permanent path built from its NAA Id, nor a device path from a possibly stale OS drive name.
  • Volume creation. It still waits for the new volume: a volume listed before it is exposed to the OS has no paths yet, so it is treated as not settled, and creation keeps rescanning and retrying.
    • megaraid: a volume without paths no longer stops the search, so the offline volume of a failed drive still listing a requested slot cannot hide the new, exposed volume.
    • storcli2: CreateLV used to return a new volume with an NAA-based path even before it was exposed. It now waits, like megaraid, until the volume is exposed or NewVolumeSettleTimeout (60 s, also when left at zero) elapses. As in megaraid, an offline volume with the same drives cannot hide the new one. A volume counts as not exposed only when it has no path at all, so an exposed volume without an NAA Id (device path only) is returned at once.
  • DESIGN.md. Documents the contract that getter read paths degrade gracefully, which failures still are errors, and that the megaraid and storcli2 create paths are deliberately stricter.

#91's change for multi-drive megaraid volumes is not carried over, since ARTESCA does not use them.

Tests

  • Scenarios. In megaraid-disk-failure.feature and storcli2-disk-failure.feature, the scenario "A failed drive does not take the paths of the others" now reports its expected table, so both known bug ARTESCA-17960 reports instead: steps are removed. Until they were removed, those steps failed with "the bug looks fixed". All 12 scenarios pass (4 per controller), for example on MegaRAID:
    --- PASS: TestScenarios/All_drives_are_healthy
    --- PASS: TestScenarios/A_failed_drive_does_not_take_the_paths_of_the_others
    --- PASS: TestScenarios/A_replacement_drive_is_reported_as_a_new,_available_drive
    --- PASS: TestScenarios/A_replacement_drive_gets_the_paths_of_its_new_volume
    
  • megaraid unit tests, table-driven:
    • getPaths with Exposed to OS set to No (empty paths, no error), Yes, or empty (paths still resolved);
    • the volume status mapping, including OfLn → Failed;
    • the new volume selection: an offline volume sharing a slot, listed before or after the new volume, does not hide it.
  • Volume creation: a new volume first listed as not exposed must be awaited, not returned without paths.
  • ssacli: a failing lsblk lookup does not abort the inventory when the device node is gone, and is still an error when it exists. A free drive whose device could not be inspected is reported Unknown.
  • storcli2, from fix: a single failed drive no longer blanks a controller's device paths #91: an unresolvable JBOD drive, and a degraded storcli2 RAID volume. New: an offline volume not exposed to the OS gets no paths, and permanentPath covers Exposed to OS set to No, Yes and missing. CreateLV waits for a volume exposed on the third listing, gives up on one never exposed, is not hidden by an offline volume with the same drives listed first, and returns at once a volume exposed without an NAA Id.

Delivery

disk-management-agent picks this up once raidmgmt is released and the agent bumps it.

// A volume not exposed to the OS yet is listed without paths: it is
// not settled, the caller rescans and retries.
if lv.PermanentPath == "" {
return nil, errors.Errorf("new logical volume %s has no device yet", lv.ID)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This returns on the first volume that shares a drive slot and has no paths, and skips the rest of the list. hasMatchingPDs matches on any one shared slot. So if the offline volume of a failed drive still lists that slot after the drive is replaced, it can show up before the new volume, and CreateLV keeps retrying until it times out even though the new volume is already exposed. Use continue so the loop still checks the other volumes. It still ends in an error, and a retry, when no matching volume has paths.

Suggested change
return nil, errors.Errorf("new logical volume %s has no device yet", lv.ID)
continue

@g-carre
g-carre force-pushed the fix/ARTESCA-17960-unhealthy-drive-inventory branch from 697914a to 4ee0286 Compare October 15, 2026 16:35
@g-carre
g-carre requested a review from a team as a code owner October 15, 2026 16:35
@g-carre
g-carre requested review from ezekiel-alexrod and riadelhajjaji-scality and removed request for a team October 15, 2026 16:35
@g-carre
g-carre force-pushed the test/ARTESCA-17960-disk-failure-scenario-gherkin branch from 74a8271 to eb4261b Compare October 15, 2026 16:35
DevicePath: strings.TrimSpace(vd.Properties.OSDriveName),
PermanentPath: permanentPath(vd.Properties.SCSINAAID),
DevicePath: devicePath(vd.Properties),
PermanentPath: permanentPath(vd.Properties),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This also changes storcli2 CreateLV. logicalvolumemanager.StorCLI2.findNewLogicalVolume returns the first volume with a matching drive set, without checking its paths. The PR says a new volume can be listed before it is exposed to the OS. In that case CreateLV now returns the volume with empty DevicePath/PermanentPath, where before it got the NAA-based path. Add the same wait as in megaraid: treat an empty PermanentPath as not settled and retry, or make the creator re-query until the volume is exposed.

blockDevice, err := s.getBlockDevice(value)
if err != nil {
return errors.Wrapf(err, "failed to get block device for %s", value)
facts.blockDeviceUnknown = true

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This drops every getBlockDevice error, not just a missing device node. If lsblk is missing, fails for another reason, or its JSON format changes, every drive silently loses its DevicePath and every free drive becomes Unknown, and nothing reports why. That is the "all disks blanked" symptom this PR fixes, now with no error at all. Only degrade when the node is actually gone (for example, check that value exists before calling lsblk), and keep returning the wrapped error otherwise.

@g-carre
g-carre force-pushed the fix/ARTESCA-17960-unhealthy-drive-inventory branch from 4ee0286 to 594d87d Compare October 15, 2026 16:44
}

if lv.PermanentPath == "" {
return nil, errors.Wrapf(errNewVolumeNotExposed, "volume %s", lv.ID)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This has the same early-return issue that was fixed in megaraid's selectNewLogicalVolume. If the offline RAID0 volume of a failed drive still lists the replaced slot, it matches the single-drive set exactly. When it is listed before the new volume, this returns errNewVolumeNotExposed without checking the rest of the list, so CreateLV waits 60 s and fails even though the new volume is already exposed. Remember the unexposed match and continue, then return errNewVolumeNotExposed after the loop only if no matching volume has paths.

@g-carre
g-carre force-pushed the fix/ARTESCA-17960-unhealthy-drive-inventory branch from 594d87d to a1f75a3 Compare October 15, 2026 16:47
continue
}

if lv.PermanentPath == "" {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An empty PermanentPath does not only mean "not exposed". The getter also returns it when SCSI NAA Id is empty (see the empty case in TestStorCLI2PermanentPath). In that case an exposed new volume that has a DevicePath is treated as unsettled, so CreateLV waits 60 s and fails after the volume was already created. An unexposed volume has both paths empty, so check both.

Suggested change
if lv.PermanentPath == "" {
if lv.PermanentPath == "" && lv.DevicePath == "" {

@g-carre
g-carre force-pushed the fix/ARTESCA-17960-unhealthy-drive-inventory branch 2 times, most recently from dc6a89b to da9ef41 Compare October 15, 2026 21:21
@g-carre
g-carre force-pushed the test/ARTESCA-17960-disk-failure-scenario-gherkin branch from eb4261b to ae2f0cd Compare October 15, 2026 21:21
When the drive of a single-drive RAID0 volume fails, the volume goes offline
and is no longer a block device of the host: it has no by-id link, and the
paths of its failed drive cannot be computed. getPaths returned an error,
which failed LogicalVolumes for the whole controller, so the healthy drives
of that controller lost their paths too (RD-2154).

- A volume whose "Exposed to OS" is "No" gets empty paths without error.
  Only this explicit value, spaces trimmed, skips the resolution, as older
  storcli versions may not report the field.
- An offline volume (OfLn) is reported Failed instead of Unknown.
- Volume creation still waits for the new volume: a volume listed before it
  is exposed to the OS has no paths yet, so it is not settled and creation
  keeps rescanning and retrying. Such a volume no longer stops the search:
  the offline volume of a failed drive still listing a requested slot must
  not hide the new, exposed volume.

DESIGN.md states that getter read paths degrade gracefully while the create
path stays strict. The failed drive scenario of megaraid-disk-failure.feature
now reports its expected table: its known bug step is removed.

Fixes: ARTESCA-17960
A failed or pulled drive whose device node has disappeared made the lsblk
lookup of its Disk Name fail, which aborted the physical drive inventory of
the whole controller. The lookup only refines the status, so when it fails
and the device node is gone, it no longer aborts the inventory. Instead:

- the usage of the device is unknown, and a free drive whose usage is
  unknown is reported Unknown, never UnassignedGood, so a drive that may
  carry data is never offered for a new volume;
- the device path is dropped, as the kernel can reuse a vanished /dev/sdX
  name for another disk.

Array members and failed drives keep the status ssacli reports. A lookup
failure on a device that exists, such as lsblk missing or its output
changing, is still an error: ignoring it would silently blank every drive.

Refs: ARTESCA-17960
… resolved

A JBOD drive whose by-id link cannot be resolved, for instance because udev
has not settled yet, failed the physical drive inventory of the whole
controller. Keep it with empty paths instead, when its by-id link or device
node is missing (ComputePaths now returns ErrNoPermanentPath for a missing
link). Any other failure, such as a permission error, is still an error:
ignoring it would silently blank the paths of every JBOD drive. Also cover
a degraded RAID volume, whose paths come from the controller output only.

Refs: ARTESCA-17960
The storcli2 volume getter read the device path of a volume from its OS
drive name and built its permanent path from its NAA Id. The offline RAID0
volume of a failed drive keeps its NAA Id, and may keep a stale OS drive
name, but it is not exposed to the OS: it has no block device and no by-id
link, so both paths named nothing, or another disk. A volume whose "Exposed
to OS" is "No" now has no paths; only this explicit value drops them, as
older storcli2 versions may not report the field.

Volume creation used to get the NAA-based path of a new volume listed
before it is exposed to the OS. It now gets no paths, so CreateLV waits,
like megaraid, until the new volume is exposed or NewVolumeSettleTimeout
elapses (60s when zero). DESIGN.md says which failures degrade and which stay errors.

The failed drive scenario of storcli2-disk-failure.feature now reports its
expected table: its known bug step is removed.

Fixes: ARTESCA-17960
@g-carre
g-carre force-pushed the test/ARTESCA-17960-disk-failure-scenario-gherkin branch from ae2f0cd to 3791dcd Compare October 15, 2026 21:37
@g-carre
g-carre force-pushed the fix/ARTESCA-17960-unhealthy-drive-inventory branch from da9ef41 to f943de8 Compare October 15, 2026 21:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant