Marvell Sata controller not recognized on restart

[gargamon]

I'm have an issue with a Marvell 88SE9123 PCIe SATA card.

First, my environment.

ASRock H470M-ITX/ac
Marvell 88SE9123 PCIe SATA
Debian Buster with all the firmware backports

The Marvell card is recognized, along with the 2 hard drives connected to it when the system first starts(power on). If later I do a restart, the card is no longer recognized, and of course neither are the connected hard drives. Dmesg shows nothing for the card on the restart but on the cold start shows all the expected info. This is consistently repeatable.

Any ideas about what the cause of this could be? Any info I could gather that would help?

Thanks.

[arochester]

10 years ago?

https://bbs.archlinux.org/viewtopic.php?id=179006

[gargamon]

10 years ago?

https://bbs.archlinux.org/viewtopic.php?id=179006

Thanks for your attention.

I turned off virtualization as suggested in the referenced article and nothing changed.

I did, however, notice some things during testing that may change the scenario. It looks like if the system is running and the PCIe sata controller is currently recognized, a restart will not detect the card again. If the PCIe card is not currently recognized, a restart will detect it and consequently see the 2 attached drives. A series of restarts will see the card detected every second restart.

Very strange.

[Head_on_a_Stick]

I turned off virtualization as suggested in the referenced article

How exactly did you do that? And how did you confirm that your attempt succeeded?

Perhaps also try restarting udev to see if that picks up the controller:

Code: Select all

# udevadm trigger

[gargamon]

Thanks for your input.

I turned off "Intel Virtualization Technology" in the bios. I hope that would be sufficient.

"udevadm trigger" does nothing. When the controller was not mounted, there is no device in /sys/class/block associated with the controller.

[Head_on_a_Stick]

I turned off "Intel Virtualization Technology" in the bios. I hope that would be sufficient.

Have you tried to apply the iommu=soft kernel command line parameter?

[gargamon]

Have you tried to apply the iommu=soft kernel command line parameter?

Thanks for the response, and sorry for the delay in responding. Real life sometimes takes precedence.

I tried this option and it made no difference.

[p.H]

if (...) the PCIe sata controller is currently recognized, a restart will not detect the card again. If the PCIe card is not currently recognized, a restart will detect it

What do you mean exactly by "(not) recognized" and "(not) detect" ?

[CwF]

There are cards that don't reset correctly. It comes up in vfio passed things often, one of the pass/fail categories.

so, gargamon, if you are a tad familiar with sysfs in /sys, find the card when up from a 'cold' boot and look at parameters. Pay attention to addresses and the like, and somewhere, when enumerated, you may find reset, scan, rescan, hard_reset, link_reset files which you can echo into to perform the action.

Look for D3 power settings or listed capabilities, cat the files for values. This is where the design flaw is. The spec is PCI 2.3, where nothing can require a power cycle to re-initialize. Still, some cards need the cycle and don't reset on a 'warm' boot.

There is an off chance you can rescan the bus if you find where it's supposed to be. And don't blow it up!

to add, the /sys/tree of the device will disappear on the warm boot...

[gargamon]

if (...) the PCIe sata controller is currently recognized, a restart will not detect the card again. If the PCIe card is not currently recognized, a restart will detect it

What do you mean exactly by "(not) recognized" and "(not) detect" ?

In my mind, the card is recognized (or detected) when the system starts and the card is visible and the attached hard drives are available to the system. Again, to clarify, every second restart the card is visible.

[p.H]

the card is visible

What do you mean EXACTLY ? Visible how ?

[gargamon]

There are cards that don't reset correctly. It comes up in vfio passed things often, one of the pass/fail categories.

so, gargamon, if you are a tad familiar with sysfs in /sys, find the card when up from a 'cold' boot and look at parameters. Pay attention to addresses and the like, and somewhere, when enumerated, you may find reset, scan, rescan, hard_reset, link_reset files which you can echo into to perform the action.

Look for D3 power settings or listed capabilities, cat the files for values. This is where the design flaw is. The spec is PCI 2.3, where nothing can require a power cycle to re-initialize. Still, some cards need the cycle and don't reset on a 'warm' boot.

There is an off chance you can rescan the bus if you find where it's supposed to be. And don't blow it up!

to add, the /sys/tree of the device will disappear on the warm boot...

Thanks for the pointers.

Let me clarify that since the first post, I have determined that the card is recognized every second restart, that is, if the card is visible, a restart will make it not visible, and vice versa.

I don't think I currently have the level of expertise necessary to poke around with the sysfs stuff you mention. Here's the dump from lspci -vv for the associated card when it is visible. Hopefully from this I can get some pointers as to what to try next.

Code: Select all

01:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9123 PCIe SATA 6.0 Gb/s controller (rev 11) (prog-if 01 [AHCI 1.0])
	Subsystem: Marvell Technology Group Ltd. 88SE9123 PCIe SATA 6.0 Gb/s controller
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 131
	Region 0: I/O ports at 4090 [size=8]
	Region 1: I/O ports at 4080 [size=4]
	Region 2: I/O ports at 4070 [size=8]
	Region 3: I/O ports at 4060 [size=4]
	Region 4: I/O ports at 4050 [size=16]
	Region 5: Memory at b1421000 (32-bit, non-prefetchable) [size=2K]
	Expansion ROM at b1410000 [disabled] [size=64K]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [50] MSI: Enable+ Count=1/1 Maskable- 64bit-
		Address: fee003d8  Data: 0000
	Capabilities: [70] Express (v2) Legacy Endpoint, MSI 00
		DevCap:	MaxPayload 512 bytes, PhantFunc 0, Latency L0s <1us, L1 <8us
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 256 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <512ns, L1 <64us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
	Kernel driver in use: ahci
	Kernel modules: ahci

01:00.1 IDE interface: Marvell Technology Group Ltd. 88SE912x IDE Controller (rev 11) (prog-if 8f [PCI native mode controller, supports both channels switched to ISA compatibility mode, supports bus mastering])
	Subsystem: Marvell Technology Group Ltd. 88SE912x IDE Controller
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin B routed to IRQ 17
	Region 0: I/O ports at 4040 [size=8]
	Region 1: I/O ports at 4030 [size=4]
	Region 2: I/O ports at 4020 [size=8]
	Region 3: I/O ports at 4010 [size=4]
	Region 4: I/O ports at 4000 [size=16]
	Region 5: Memory at b1420000 (32-bit, non-prefetchable) [size=16]
	Expansion ROM at b1400000 [disabled] [size=64K]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit-
		Address: 00000000  Data: 0000
	Capabilities: [70] Express (v2) Legacy Endpoint, MSI 00
		DevCap:	MaxPayload 512 bytes, PhantFunc 0, Latency L0s <1us, L1 <8us
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 256 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <512ns, L1 <64us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [100 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
	Kernel driver in use: pata_marvell
	Kernel modules: pata_marvell, ata_generic

[CwF]

Well, guess you just shutdown and power on rather than reboot!
It appears it is not a boot device, so no report prior to the OS...even with the bios report I have seen cards miss that step and stay missing, some come up in the OS, some temperamental cards on bare metal are perfectly reliable in a vm, some take down the host...
I'd consider it another piece of art...I have two 8 port pci-x marvells, they're pretty!

[gargamon]

Well, guess you just shutdown and power on rather than reboot!
It appears it is not a boot device, so no report prior to the OS...even with the bios report I have seen cards miss that step and stay missing, some come up in the OS, some temperamental cards on bare metal are perfectly reliable in a vm, some take down the host...
I'd consider it another piece of art...I have two 8 port pci-x marvells, they're pretty!

Yeah, it's very pretty...

The box in question is an encrypted NAS setup that requires manual intervention every reboot/restart anyway, so I can easily live with the irregularities. At least (other than the Marvell card), the components are very stable and it stays up for months at a time.

I will leave this open for a bit to see if any other ideas surface, then likely close it. Hopefully someone thinking of buying a Marvell card may see it and change their thinking...

[p.H]

Here's the dump from lspci -vv for the associated card when it is visible.

How is the dump when the card is not "visible" (whatever this means) ?

[CwF]

Here's the dump from lspci -vv for the associated card when it is visible.

How is the dump when the card is not "visible" (whatever this means) ?

if the card is asleep and not responding to a reset it will appear like a unknown device. It's presence will be known, it just hasn't said anything! That's what I was getting at...the enumerated sysfs entries will be gone but the pci slot info will be there, bare.

[gargamon]

Blame it on the ASRock motrherboard. Changing the bios setting for the PCIe slot from Auto to Gen1 and the card is now recognized after all boots/restarts.

While not an ideal solution as it limits the speed of the PCIe slot to 8GB/s, its OK for me as that is more than SATA III anyway. Also the only disks mounted on the PCIe sata card are parity disks which are rarely accessed and never accesed for real work.

How do I mark this issue as closed?

[CwF]

bios setting for the PCIe slot from Auto to Gen1

I should have thought of that! I'd come to the same solution passing early gpu's. It's the card really.

How do I mark this issue as closed?

just go to your first post and edit, prepend or append [solved] in the title.

[p.H]

it limits the speed of the PCIe slot to 8GB/s, its OK for me as that is more than SATA III

Do you mean 8 GByte/s or 8 Gbit/s ? 8 Gbit/s is more than one SATA III link (6 Gbit/s) but less than two.

disks mounted on the PCIe sata card are parity disks

What do you mean ? RAID 4 parity disks ?