boot hangs on infinite fsck upgrading to stretch

Help with issues regarding installation of Debian

boot hangs on infinite fsck upgrading to stretch

Postby papatangonyc » 2018-12-27 01:45

I upgraded to Stretch (stable) from oldstable and now boot hangs with messages about "a start job is running". Boot never reaches a login prompt or an emergency shell ... nothing. I have to reboot. I can only get into my system using recovery mode. At first I couldnt examine boot logs and I had to set up journalctl to actually keep record of prior boots. My log files show that my boot/efi /home and swap partitions fail to mount. Relevant portions below. It seems an fsck check is timing out. I changed two timeout settings during boot for systemd to infinity within /etc/systemd/systemd.conf and left it overnight; 13 hours later and no progress. The total drive size is 500GB, not full, and partitioned into root, boot, home and swap.

I also already ensured that the uuids in fstab match those of the devices. And I already checked that the kernel config includes CONFIG_FHANDLE=y. I then commented out the swap partition from fstab. I added nobootwait, option, and nofail to the fstab options for a remote nfs mount and swap (before I completely commented swap out and the nfs mount). Nothing helped. I did the same for some pre-defined usb stick mounts before commenting those out too. I still cannot boot.

Then I booted off a live CD and ran fsck and e2fsck upon all the partitions that I could (swap was the only one for which no fsck-like tool exists I think). The result was that the filesystems are ok. No errors found.

I'm getting desperate looking for a solution and some help. I don't want to have to blow away this install and re-install, back up and potentially restore all data. I should note that his upgrade to Stretch has been one of the worst upgrade experiences I have ever gone through. I wasted a lot of time figuring out that plymouth is a worthless good-for-nothing broken package -- at least for me -- and I had to purge it since it prevented a normal boot sequence to X after upgrading to stretch. I was going to investigate the cause of its failure, but for a graphical splash screen I said to myself, my time is not worth it. I don't need a graphical boot sequence, nor should something that is bells and whistles stand in the way of critical path components toward booting up--thus I purged it.

Anyway, enough of my frustration. Any help or tips would be welcome. Thank you.



P.S: Further below is the relevant portion of my log. Elsewhere I also have errors that may or may not be relevant--but these occur during recovery mode boot without further issue:
kernel: tpm tpm0: A TPM error (6) occurred attempting to read a pcr value
blkmapd[328]: open pipe file /run/rpc_pipefs/nfs/blocklayout failed: No such file or directory

PPS: I also removed some pre-defined NFS exports, which also didn't help. I'm starting to throw everything at this problem. If fsck is timing out but fsck from a live cd shows no errors...I'm at a dead end here.

Here is some of my log:
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
systemd[1]: Started Flush Journal to Persistent Storage.
-- Subject: Unit systemd-journal-flush.service has finished start-up
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit systemd-journal-flush.service has finished starting up.
--
-- The start-up result is done.
systemd[1]: dev-sda3.device: Job dev-sda3.device/start timed out.
systemd[1]: Timed out waiting for device dev-sda3.device.
-- Subject: Unit dev-sda3.device has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit dev-sda3.device has failed.
--
-- The result is timeout.
systemd[1]: Dependency failed for Swap Partition.
-- Subject: Unit dev-sda3.swap has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit dev-sda3.swap has failed.
--
-- The result is dependency.
systemd[1]: dev-sda3.swap: Job dev-sda3.swap/start failed with result 'dependency'.
systemd[1]: dev-sda3.device: Job dev-sda3.device/start failed with result 'timeout'.
systemd[1]: dev-disk-by\x2duuid-C206\x2dD9FD.device: Job dev-disk-by\x2duuid-C206\x2dD9FD.device/start timed out.
systemd[1]: Timed out waiting for device dev-disk-by\x2duuid-C206\x2dD9FD.device.
-- Subject: Unit dev-disk-by\x2duuid-C206\x2dD9FD.device has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit dev-disk-by\x2duuid-C206\x2dD9FD.device has failed.
--
-- The result is timeout.
systemd[1]: Dependency failed for File System Check on /dev/disk/by-uuid/C206-D9FD.
-- Subject: Unit systemd-fsck@dev-disk-by\x2duuid-C206\x2dD9FD.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit systemd-fsck@dev-disk-by\x2duuid-C206\x2dD9FD.service has failed.


-- The result is dependency.
systemd[1]: Dependency failed for /boot/efi.
-- Subject: Unit boot-efi.mount has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit boot-efi.mount has failed.
--
-- The result is dependency.
systemd[1]: Dependency failed for Local File Systems.
-- Subject: Unit local-fs.target has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit local-fs.target has failed.
--
-- The result is dependency.
systemd[1]: local-fs.target: Job local-fs.target/start failed with result 'dependency'.
systemd[1]: local-fs.target: Triggering OnFailure= dependencies.
systemd[1]: boot-efi.mount: Job boot-efi.mount/start failed with result 'dependency'.
systemd[1]: systemd-fsck@dev-disk-by\x2duuid-C206\x2dD9FD.service: Job systemd-fsck@dev-disk-by\x2duuid-C206\x2dD9FD.service/start failed with result
systemd[1]: dev-disk-by\x2duuid-C206\x2dD9FD.device: Job dev-disk-by\x2duuid-C206\x2dD9FD.device/start failed with result 'timeout'.
systemd[1]: dev-disk-by\x2duuid-25ed3900\x2d5a41\x2d4a69\x2dadfd\x2d2bddacc30dc5.device: Job dev-disk-by\x2duuid-25ed3900\x2d5a41\x2d4a69\x2dadfd\x2d2
systemd[1]: Timed out waiting for device dev-disk-by\x2duuid-25ed3900\x2d5a41\x2d4a69\x2dadfd\x2d2bddacc30dc5.device.
-- Subject: Unit dev-disk-by\x2duuid-25ed3900\x2d5a41\x2d4a69\x2dadfd\x2d2bddacc30dc5.device has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit dev-disk-by\x2duuid-25ed3900\x2d5a41\x2d4a69\x2dadfd\x2d2bddacc30dc5.device has failed.
--
-- The result is timeout.
systemd[1]: Dependency failed for /home.
-- Subject: Unit home.mount has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit home.mount has failed.
--
-- The result is dependency.
systemd[1]: home.mount: Job home.mount/start failed with result 'dependency'.
systemd[1]: Dependency failed for File System Check on /dev/disk/by-uuid/25ed3900-5a41-4a69-adfd-2bddacc30dc5.
-- Subject: Unit systemd-fsck@dev-disk-by\x2duuid-25ed3900\x2d5a41\x2d4a69\x2dadfd\x2d2bddacc30dc5.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit systemd-fsck@dev-disk-by\x2duuid-25ed3900\x2d5a41\x2d4a69\x2dadfd\x2d2bddacc30dc5.service has failed.
--
-- The result is dependency.
systemd[1]: systemd-fsck@dev-disk-by\x2duuid-25ed3900\x2d5a41\x2d4a69\x2dadfd\x2d2bddacc30dc5.service: Job systemd-fsck@dev-disk-by\x2duuid-25ed3900\x
systemd[1]: dev-disk-by\x2duuid-25ed3900\x2d5a41\x2d4a69\x2dadfd\x2d2bddacc30dc5.device: Job dev-disk-by\x2duuid-25ed3900\x2d5a41\x2d4a69\x2dadfd\x2d2
systemd[1]: Starting Preprocess NFS configuration...
-- Subject: Unit nfs-config.service has begun start-up
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit nfs-config.service has begun starting up.
systemd[1]: Starting Enable support for additional executable binary formats...
-- Subject: Unit binfmt-support.service has begun start-up
-- Defined-By: systemd
-- Support: https://www.debian.org/support
--
-- Unit binfmt-support.service has begun starting up.
systemd[1]: Starting Raise network interfaces...
-- Subject: Unit networking.service has begun start-up
-- Defined-By: systemd
-- Support: https://www.debian.org/support
papatangonyc
 
Posts: 6
Joined: 2018-12-27 01:34

Re: boot hangs on infinite fsck upgrading to stretch

Postby p.H » 2018-12-27 09:05

papatangonyc wrote: boot hangs with messages about "a start job is running".

What start job ? Please post the full lines.

papatangonyc wrote:I can only get into my system using recovery mode.

Do you mean the "recovery mode" available in GRUB's "Advanced" menu (which removes "quiet" and adds "single" in the kernel command line) ? If yes, please post the output of
Code: Select all
df
ls /dev/sd*
ls /dev/disk/by-uuid

AFAIK there is no difference in fsck'ing and mounting filesystems between normal and recovery mode. The difference happens later in the init process.
By default the initramfs performs fsck on the root filesystem before mounting it. Later, the init system performs fsck on other filesystems according to /etc/fstab.

papatangonyc wrote:It seems an fsck check is timing out

"It seems" ? Facts please. I do not see in your post any evidence that fsck is timing out. All I can see is that devices cannot be found, so fsck and mount cannot even happen.

papatangonyc wrote:I already checked that the kernel config includes CONFIG_FHANDLE=y

If you feel the need to mention it, I suspect that you are not using a Debian stock kernel. What kernel are you using ? Does it use an initrd/initramfs ?

papatangonyc wrote:systemd[1]: dev-sda3.device: Job dev-sda3.device/start timed out.

Looks like your fstab uses /dev/sd* device names. Such use is not reliable because these names are not persistent. Consider using UUID or LABEL instead.
p.H
 
Posts: 709
Joined: 2017-09-17 07:12

Re: boot hangs on infinite fsck upgrading to stretch

Postby Head_on_a_Stick » 2018-12-27 12:33

p.H wrote:
papatangonyc wrote:systemd[1]: dev-sda3.device: Job dev-sda3.device/start timed out.

Looks like your fstab uses /dev/sd* device names. Such use is not reliable because these names are not persistent. Consider using UUID or LABEL instead.

The OP said that they have checked the UUIDs in /etc/fstab, I think those units are auto-generated.[1]

Looks like the filesystem modules are missing, perhaps rebuild the initramfs? And check the bootloader configuration, ofc.
"French riots get results! U lot are instagram zzzombies" — graffiti over Euston underpass
User avatar
Head_on_a_Stick
 
Posts: 8590
Joined: 2014-06-01 17:46
Location: /dev/chair

Re: boot hangs on infinite fsck upgrading to stretch

Postby p.H » 2018-12-27 12:56

Head_on_a_Stick wrote:The OP said that they have checked the UUIDs in /etc/fstab, I think those units are auto-generated.

My point was not whether the UUIDs in /etc/fstab are correct or not (I have no reason not to trust the OP about this) but the presence of /dev/sd* in /etc/fstab. AFAIK, dev-sd* systemd units are auto-generated when /etc/fstab contains /dev/sd* device names.
This specific point in my post was not related to the issue described in the OP.

I doubt the issue is caused by missing modules in the initramfs because IIUC it happens after the initramfs has handed over to the init system.
p.H
 
Posts: 709
Joined: 2017-09-17 07:12

Re: boot hangs on infinite fsck upgrading to stretch

Postby papatangonyc » 2018-12-27 13:02

Thank you for the quick replies.

On screen during boot I see:
Code: Select all
tpm tpm(0): A TPM error (6) occurred attempting to read a pcr value
tpm tpm(0): A TPM error (6) occurred attempting to read a pcr value
/dev/sda2: clean ... files ... blocks
[ *** ] (1 of 3) A start job is running for ddacc30dc5.device (Xs / <no limit or timeout value>

Then that last line alternates with two more version of itself:
Code: Select all
[ *** ] (2 of 3) A start job is running for 06\x2dD9FD (Xs / <no limit or timeout value>
[ *** ] (3 of 3) A start job is running for dev-sda3.device (Xs / <no limit or timeout value>

Where X is the time elapsed and the timeout value used to be 1:30 but then I changed it to infinity.

Yes, I did mean "recovery mode" available in GRUB's "Advanced" menu that removes "quiet" and adds "single" in the kernel command line. As I had mentioned I also tried removing quiet from a non-recovery mode boot option in GRUB's "Advanced" menu to see if I could get better logging, but I don't think it made a difference.

Output of df:
Code: Select all
Filesystem     1K-blocks      Used Available Use% Mounted on
udev             1953696         0   1953696   0% /dev
tmpfs             392940      1764    391176   1% /run
/dev/sda2       67154552  36723624  26996588  58% /
tmpfs            1964684     44364   1920320   3% /dev/shm
tmpfs               5120         4      5116   1% /run/lock
tmpfs            1964684         0   1964684   0% /sys/fs/cgroup
/dev/sda1         523248       132    523116   1% /boot/efi
/dev/sda4      403587808 193191236 189872388  51% /home
tmpfs             392936        16    392920   1% /run/user/119
tmpfs             392936        28    392908   1% /run/user/1000


Output of ls /dev/sd* is
Code: Select all
/dev/sda  /dev/sda1  /dev/sda2  /dev/sda3  /dev/sda4

Output of /dev/disk/by-uuid
Code: Select all
25ed3900-5a41-4a69-adfd-2bddacc30dc5  c0403138-8cdb-4412-b9a6-de4375ab60ff  C206-D9FD  d56d5024-9570-4066-b706-ddcd676612cb

When I read the log and saw this:
Code: Select all
systemd-fsck@dev-disk-by\x2duuid-C206\x2dD9FD.service: Job systemd-fsck@dev-disk-by\x2duuid-C206\x2dD9FD.service/start failed with result
systemd[1]: dev-disk-by\x2duuid-C206\x2dD9FD.device: Job dev-disk-by\x2duuid-C206\x2dD9FD.device/start failed with result 'timeout'.


...I interpreted that as a problem with fsck. Perhaps I was wrong. If you say that devices are not being found, I guess that would also make sense. I don't know. All I can do is read the log but I'm not an expert at interpreting these logs. Never had a problem like this before.

I only mention CONFIG_FHANDLE=y because I was casting about for possible causes of my problem and I read some posts on user forums that mention this option as being required or else some persons in the past have experienced similar symptoms. Also I did not change that option. I found it set already. I have not customized my kernel at all. Way over my head. All I did was an upgrade from oldstable to stable. One such example post: https://stackoverflow.com/questions/233 ... during-sta

As for my /etc/fstab, I was already using UUIDs for a long time. The only non-uuid identified drive is the cdrom with /dev/sr0.
Here it is as it is today and still cannot boot:
Code: Select all
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
# / was on /dev/sda2 during installation
UUID=c0403138-8cdb-4412-b9a6-de4375ab60ff /               ext4    discard,noatime,nodiratime,errors=remount-ro 0       1
# /boot/efi was on /dev/sda1 during installation
UUID=C206-D9FD  /boot/efi       vfat    defaults        0       1
# /home was on /dev/sda4 during installation
UUID=25ed3900-5a41-4a69-adfd-2bddacc30dc5 /home           ext4    discard,noatime,nodiratime 0       2
# swap was on /dev/sda3 during installation
# UUID=d56d5024-9570-4066-b706-ddcd676612cb none            swap    sw              0       0
/dev/sr0        /media/cdrom0   udf,iso9660 user,noauto     0       0
# /dev/sdb1       /media/usb0     auto    rw,user,noauto  0       0
# /dev/sdb2       /media/usb1     auto    rw,user,noauto  0       0
#192.168.1.3   mnt      nfs4   rw,option,noauto,nofail,nobootwait      0   0
#UUID=F765-51E3 /media/usb1 vfat rw,option,noauto,user,nosuid,nodev,relatime,uid=1000,gid=1000,fmask=0022,dmask=0077,codepage=437,iocharset=utf8,shortname=mixed,showexec,utf8,flush,errors=remount-ro,uhelper=udisks2w,nofail,nobootwait
#UUID=B2FB-1E2C /media/usb0 vfat rw,option,noauto,user,nosuid,nodev,relatime,uid=1000,gid=1000,fmask=0022,dmask=0077,codepage=437,iocharset=utf8,shortname=mixed,showexec,utf8,flush,errors=remount-ro,uhelper=udisks2w,nofail,nobootwait


Thanks again.
papatangonyc
 
Posts: 6
Joined: 2018-12-27 01:34

Re: boot hangs on infinite fsck upgrading to stretch

Postby papatangonyc » 2018-12-27 13:13

I realize now that perhaps I did "customize" my kernel, but I'm not sure what that term means.

I should mention that well over a year ago I did add modules to the kernel under /etc/modules after installing a package that provided the modules. It was tp_smapi and coretemp, from the tp-smapi-dkms package. But I commented those out. The package is still installed but I should purge it. I have no idea if that might be relevant. I doubt it but I mention it just in case. Home page: http://www.thinkwiki.org/wiki/Tp_smapi

Thanks again.
papatangonyc
 
Posts: 6
Joined: 2018-12-27 01:34

Re: boot hangs on infinite fsck upgrading to stretch

Postby llivv » 2018-12-30 22:26

you have to fix fstab field 5 for filesystems that need to be checked. see: man fstab
Code: Select all
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
UUID=c0403138-8cdb-4412-b9a6-de4375ab60ff /               ext4    discard,noatime,nodiratime,errors=remount-ro 0       1
UUID=25ed3900-5a41-4a69-adfd-2bddacc30dc5 /home           ext4    discard,noatime,nodiratime 0       2

# <file system> <mount point> <type> <options> <dump> <pass>
UUID=c0403138-8cdb-4412-b9a6-de4375ab60ff / ext4 discard,noatime,nodiratime,errors=remount-ro 1 1
UUID=25ed3900-5a41-4a69-adfd-2bddacc30dc5 /home ext4 discard,noatime,nodiratime 1 2


Personally, your ext4 options make little sense to me. But if you have special reasons to use those options that is your choice.
If fsck still bails after changing the dump entries for both / and /home
I'd first change the dump entry for /boot/efi reboot checking for the fsck bail
If still getting the fsck errors I'd then start changing the mount options to defaults
for eash hard disk filesystem and disconnecting the usb devices
to see which fs(s) are at fault.
In memory of Ian Ashley Murdock (1973 - 2015) founder of the Debian project.
User avatar
llivv
 
Posts: 5851
Joined: 2007-02-14 18:10
Location: cold storage

Re: boot hangs on infinite fsck upgrading to stretch

Postby Head_on_a_Stick » 2018-12-30 22:49

p.H wrote:AFAIK, dev-sd* systemd units are auto-generated when /etc/fstab contains /dev/sd* device names.

Nope :mrgreen:

@OP: with which kernel are you booted?
Code: Select all
uname -a
"French riots get results! U lot are instagram zzzombies" — graffiti over Euston underpass
User avatar
Head_on_a_Stick
 
Posts: 8590
Joined: 2014-06-01 17:46
Location: /dev/chair

Re: boot hangs on infinite fsck upgrading to stretch

Postby papatangonyc » 2018-12-31 22:31

Output of uname -a is
Code: Select all
Linux <hostname> 4.9.0-8-amd64 #1 SMP Debian 4.9.130-2 (2018-10-27) x86_64 GNU/Linux


I changed the 5th fstab fields for / and /home, and the options to defaults, and that failed. Then I also changed the 5th field for /boot/efi and that didn't help either.
Code: Select all
UUID=c0403138-8cdb-4412-b9a6-de4375ab60ff /               ext4    defaults 1       1
# /boot/efi was on /dev/sda1 during installation
UUID=C206-D9FD  /boot/efi       vfat    defaults        1       1
# /home was on /dev/sda4 during installation
UUID=25ed3900-5a41-4a69-adfd-2bddacc30dc5 /home           ext4    defaults 1       2

I did not yet try setting the 5th field just for efi and not for / and /home. I could try that right after posting and if it solves anything, or gives different results in any way, I can post again.

The boot options were due to my purchase long ago of an ssd and read someplace that to optimize for ssd drives I could use certain ext4 options. Perhaps that was a mistake. I'm no expert. I reverted to defaults, like I said, without any effect. And as for that 5th field...I don't have dump installed, so could it really make a difference?

Here is the entire log (available for the next 14 days) of my most recent boot from journalctl wherein / and /home/ and /boot/efi have the 5th field changed to 1, and all options are set to defaults, just in case I am missing something relevant.
https://pastebin.com/2uNt8SKE

Disconnecting the usb devices? Just to be clear there are no usb drives. There are no usb devices at all of any kind at boot time. I sometimes plug them in well after boot, but there are no external usb devices bearing any kind of fs at boot time. And I commented out those lines in fstab anyway.



Everybody have a Happy New Year in the meantime and again, thanks for the help.
papatangonyc
 
Posts: 6
Joined: 2018-12-27 01:34

Re: boot hangs on infinite fsck upgrading to stretch

Postby llivv » 2019-01-01 22:00

A couple of last ditch efforts here keeping in mind that if you have made the changes posted here before you upgraded - I beleive your upgrade would have been a much different experience.
I say that becasue after reading the journal there are still entries from the old config before you made changes to it still trying to load and failing. Especially when systemd starts running the bootup.
It looks as if systemd is loading in parallel making it really difficult to follow entries in the journal.

from the journal:
Code: Select all
/boot/vmlinuz-4.9.0-8-amd64 root=UUID=c0403138-8cdb-4412-b9a6-de4375ab60ff ro i915.i915_enable_rc6=1 pcie_aspm=force acpi=noirq quiet splash

edit from the grub menu at boot using
Code: Select all
e
key to edit the boot parameters in the grub menu.
scroll down to the linux entry and remove
Code: Select all
ro i915.i915_enable_rc6=1 pcie_aspm=force acpi=noirq quiet splash

from that line.
than from the edited grub menu
Code: Select all
CTRL +X

to boot
if that helps at all please save another journal to disk

You could also try rebuilding the initrd.img
not sure it will help at all at this stage but also worth a try seeing how boot looks to me like it hasn't picked up most of the config changes you've made so far.
ie: systemd is still trying to load swap and failing - there are still entries for NFS - etc etc
Code: Select all
# update-initramfs -uk "$uname"

reboot
If? either if these adjustments make any noticeable improvement in booting please repost another journal.

If no noticeable improvement,
I'd suggest you restore from jessie backup
than make all the changes to jessie pre-upgrade to stretch

than post a journal from restored jessie with config changes you've made
before the upgrade.

Just my view from reading the journal.

Edit: if you still have jessie 3.16 kernel installed you could see if that still boots to a usable system.
edit2: for ssd the recommended option is
Code: Select all
UUID    /      ext4        defaults,discard      0      1
and run fstrim weekly
the 5th entry in fstab [dump] calls fsck at mounting the filesystem so for an ssd the 0 is correct (don't run fsck of the filesystem) ie: use periodic trim instead.
edit3: the journal also shows you have LVM which just adds another layer of management to deal with especially during upgrades. Do you remember how and when you set it up?
edit4: keep in mind that restoring from backup without cleaning the ssd my make an even bigger mess on disk,,, it's hard to tell what other issues my be lurking in the filesystem.
In memory of Ian Ashley Murdock (1973 - 2015) founder of the Debian project.
User avatar
llivv
 
Posts: 5851
Joined: 2007-02-14 18:10
Location: cold storage

Re: boot hangs on infinite fsck upgrading to stretch

Postby papatangonyc » 2019-01-02 17:29

Booted twice as per your suggestion without all the additional args via grub command editing, both old and new kernels. Specifically I removed
Code: Select all
ro i915.i915_enable_rc6=1 pcie_aspm=force acpi=noirq quiet splash

Logs are here for the next 2 weeks:
Code: Select all
https://pastebin.com/r1MZtLiH
https://pastebin.com/grPL0Zhm

But still timing out on boot up. Logs look pretty much the same.

I say that becasue after reading the journal there are still entries from the old config before you made changes to it still trying to load and failing. Especially when systemd starts running the bootup.

I was wondering about that. It seems that way to me too by reading the logs, but how can that be possible?! Since when are config files ignored during boot, such as fstab? What then is the point of fstab? This is news to me and somewhat troubling. I don't know much about the boot process but I thought fstab is supposed to be consulted during boot, granted it resides on a fs that needs to be mounted during boot, so I'm a bit unclear on when and how that was done during boot in the past, and how it is being done today...something I should read up on. But still, something is seriously wrong here if I cannot remove a bad fs from boot, regardless.

It looks as if systemd is loading in parallel making it really difficult to follow entries in the journal.

I thought systemd by design and default performs boot tasks in parallel; so that should not be a surprise. I was also wondering if there is any way to force a serialized boot sequence to make reading logs easier. But so far I have only found posts on other user forums that state it is impossible. If you know otherwise, please share.

If I included LVM during install that was a mistake, since this is a single ssd machine. There's little point to using LVM, or at least I never intended to use it anyway, and I have never taken advantage of it.

I will attempt
Code: Select all
update-initramfs -uk "$uname"
later today.

This seems to be heading for no solution.
if you have made the changes posted here before you upgraded - I beleive your upgrade would have been a much different experience.

I seriously doubt that. Or at least I never go the memo about a few fstab options being able to totally screw up an upgrade. That would seem a bit brittle.
I'd suggest you restore from jessie backup
than make all the changes to jessie pre-upgrade to stretch

than post a journal from restored jessie with config changes you've made
before the upgrade.

I don't have the time to go back to Jessie, etc., not to mention I never backup the OS and binaries, just my user data, ie, that which cannot be reinstalled. If update-initramfs fails I will have to back up user data again and remove this installation of Debian.
papatangonyc
 
Posts: 6
Joined: 2018-12-27 01:34

Re: boot hangs on infinite fsck upgrading to stretch

Postby Head_on_a_Stick » 2019-01-02 18:03

If the journal is recording then the initramfs stage is passed, as p.H noted earlier.

The root partition is mounted but /home, swap & the ESP fail.

@OP: try adding x-systemd.automount or nofail to the fstab lines for the /home, swap & EFI partitions.

https://bbs.archlinux.org/viewtopic.php?id=238554&p=3

Your posted journal seems to show that systemd is attempting to mount UUID 25ed3900-5a41-4a69-adfd-2bddacc30dc5 as both /home and swap.

Is this a GPT disk?

Can we please see the output of
Code: Select all
# gdisk -l /dev/sda

I think we need further details of your upgrade procedure from jessie, check the official guide and see if you missed anything:

https://www.debian.org/releases/stable/ ... ng.en.html
"French riots get results! U lot are instagram zzzombies" — graffiti over Euston underpass
User avatar
Head_on_a_Stick
 
Posts: 8590
Joined: 2014-06-01 17:46
Location: /dev/chair

Re: boot hangs on infinite fsck upgrading to stretch

Postby sunrat » 2019-01-02 21:50

llivv wrote:edit2: for ssd the recommended option is
Code: Select all
UUID    /      ext4        defaults,discard      0      1
and run fstrim weekly
the 5th entry in fstab [dump] calls fsck at mounting the filesystem so for an ssd the 0 is correct (don't run fsck of the filesystem) ie: use periodic trim instead.

Just a side note - discard and fstrim are not both needed. The current recommendations I have read are to use fstrim weekly (fstrim.service) with a systemd timer and not use discard at all.
I'm not in Stretch right now but fstrim.timer may be set up and just needs to be started. You can check with:
Code: Select all
systemctl status fstrim.timer
“ computer users can be divided into 2 categories:
Those who have lost data
...and those who have not lost data YET ”
Remember to BACKUP!
User avatar
sunrat
 
Posts: 2509
Joined: 2006-08-29 09:12
Location: Melbourne, Australia

Re: boot hangs on infinite fsck upgrading to stretch

Postby papatangonyc » 2019-01-03 02:14

I could have sworn I already tried nofail, perhaps in combination with some other options. I definitely already tried nobootwait.
In any event, with defaults,discard,nofail on /home, /boot/efi and swap, I could not believe my eyes...boot up actually worked.
Here's the log:
Code: Select all
https://pastebin.com/vnj6rqH0

So as per https://bbs.archlinux.org/viewtopic.php?id=238554&p=3 this is all caused by an LVM bug? And I have nothing more to worry about? I don't suppose I could remove LVM at this point? Too late?

I never installed fstrim. My next step.

It seems moot now but here is the output of gdisk:
Code: Select all
GPT fdisk (gdisk) version 1.0.1

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present

Found valid GPT with protective MBR; using GPT.
Disk /dev/sda: 1000215216 sectors, 476.9 GiB
Logical sector size: 512 bytes
Disk identifier (GUID): 94E91A1A-22F1-4971-B032-4F2E4B6F01EB
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 1000215182
Partitions will be aligned on 2048-sector boundaries
Total free space is 24558189 sectors (11.7 GiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048         1050623   512.0 MiB   8300 
   2         1050624       137768959   65.2 GiB    8300 
   3       137768960       155346943   8.4 GiB     8200 
   4       155346944       975659007   391.2 GiB   8300


Can we call this case closed? Thank you all.
papatangonyc
 
Posts: 6
Joined: 2018-12-27 01:34

Re: boot hangs on infinite fsck upgrading to stretch

Postby llivv » 2019-01-03 03:37

papatangonyc wrote:
Code: Select all
https://pastebin.com/vnj6rqH0

see lines #173 and #1106 in the log for cleaning up your custom boot parameters....
papatangonyc wrote:I never installed fstrim. My next step.

fstrim is part of the util-linux package
I'd encourage to read of both manpages for fstrim and fsfreeze
papatangonyc wrote:Can we call this case closed? Thank you all.

It boots - if you are happy so am I :wink:
In memory of Ian Ashley Murdock (1973 - 2015) founder of the Debian project.
User avatar
llivv
 
Posts: 5851
Joined: 2007-02-14 18:10
Location: cold storage

Next

Return to Installation

Who is online

Users browsing this forum: No registered users and 8 guests

fashionable