Debugging a strange disk layout on my Linode Arch VPS
Background #
I am running an Arch Linux VPS on Linode, and it has not been updated for years. In fact, I cannot even remember how did I installed it. The only thing I know is that it is installed in 2021 spring.
I ssh’ed onto it and typed pacman -Syu. According to recent Grub breaking changes, I need to run grub-install ... and grub-mkconfig again. That’s not a big problem, and I had been doing this for many other machines, so let’s go.
I double checked that the machine is using BIOS rather than UEFI boot.
… and let’s … go?
grub-install? #
% grub-install --target=i386-pc /dev/sda
Installing for i386-pc platform.
grub-install: warning: File system `ext2' doesn't support embedding.
grub-install: warning: Embedding is not possible. GRUB can only be installed in this setup by using blocklists. However, blocklists are UNRELIABLE and their use is discouraged..
grub-install: error: will not proceed with blocklists.
What? I had never seen this before. Had it just destroyed my boot disk? Was I installing it on the right disk?
I also confirmed that it is booting using Grub (I was thinking whether it is possible for Linode to boot the kernel directly): Yes, since Grub appends BOOT_IMAGE= cmdline option, according to MkfsSion@archlinuxcn_group.
I thought I needed to double check my disks.
Disks … partitions? Where are you? #
I was going to check if my partitions were still here:
% fdisk -l /dev/sda
Disk /dev/sda: 24.5 GiB, 26306674688 bytes, 51380224 sectors
Disk model: QEMU HARDDISK
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x00000000
What? Shouldn’t there be partitions?
I also noticed that there is a disk called sdb. Nothing on this disk is mounted, but again let’s check it.
% fdisk -l /dev/sdb
Disk /dev/sdb: 512 MiB, 536870912 bytes, 1048576 sectors
Disk model: QEMU HARDDISK
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Still nothing, not even Disklabel type: dos, and there aren’t other disks. How did it mount the root?
I was sure I am mounted. #
% mount | grep sda
/dev/sda on / type ext4 (rw,relatime,errors=remount-ro)
% cat /etc/fstab
/dev/sda / ext4 rw,relatime,errors=remount-ro,data=ordered 0 1
This did not seem right. Not a single person or manual told me that I should mount the whole hard disk as root (except when I have a separate boot device, which is what I am doing on my workstation), especially on a VPS where there is actually a single boot device.
However, the system is working. Although it has been running for 168 days, I am sure that I did not alter anything related to partitions or bootloaders since it booted.
Where did I get there? What the hell is going on?
Dumping the disk #
The safest option was to have a dump of the first several KiB of the disks locally for investigation.
% dd if=/dev/sda of=./sda count=1024
1024+0 records in
1024+0 records out
524288 bytes (524 kB, 512 KiB) copied, 0.00814989 s, 64.3 MB/s
% file ./sda
./sda: DOS/MBR boot sector
% dd if=/dev/sdb of=./sdb count=1024
1024+0 records in
1024+0 records out
524288 bytes (524 kB, 512 KiB) copied, 0.0153935 s, 34.1 MB/s
% file ./sdb
./sdb: Linux swap file, 4k page size, little endian, version 1, size 131071 pages, 0 bad pages, no label, UUID=c0899d8a-1100-40fe-abfb-87e6ee0535e1
Hmm, we could safely eliminate sdb since it is a swap partition (also confirmed from Linode console).
For sda, it was more weird: file(1) said that it IS a MBR boot sector, also did fdisk(1). However, fdisk(1) did not show any partition. Moreover, it is a mounted ext4 partition. What?
Investigating the dump #
To have more data, I dumped the first 8MiB of sda and pulled it to my local PC.
I firstly viewed the raw hex of this dump, and I found some strings related to Grub. (Apologizes for not remembering the MBR partition tabe layout or ext4 header structure).
I also checked using binwalk(1). My intent was to see if it actually has a valid ext4 filesystem on it. However, binwalk(1) did not show anything related to filesystems. Perhaps this tool does not support it?
% binwalk ./sda
DECIMAL HEXADECIMAL DESCRIPTION
--------------------------------------------------------------------------------
1511464 0x171028 Unix path: /lib/systemd/systemd
1726250 0x1A572A Unix path: /lib/python3.5/pdb.py
1838632 0x1C0E28 Unix path: /etc/python3.5/sitecustomize.py
1906216 0x1D1628 Unix path: /etc/ssl/openssl.cnf
1997354 0x1E7A2A Unix path: /lib/snapd/snap-confine
1997866 0x1E7C2A Unix path: /lib/snapd/snapctl
1998376 0x1E7E28 Unix path: /usr/lib/snapd/snap-device-helper
3281192 0x321128 Unix path: /etc/alternatives/view.it.1.gz
3294760 0x324628 Unix path: /etc/alternatives/rmt
3295528 0x324928 Unix path: /lib/x86_64-linux-gnu/libpng12.so.0
3299368 0x325828 Unix path: /etc/alternatives/w
3300392 0x325C28 Unix path: /etc/alternatives/rview
3301416 0x326028 Unix path: /etc/alternatives/pager
3309352 0x327F28 Unix path: /lib/x86_64-linux-gnu/ld-2.23.so
7077928 0x6C0028 Unix path: /var/lib/usbutils/usb.ids
7095592 0x6C4528 Unix path: /etc/alternatives/rsh
7328808 0x6FD428 Unix path: /lib/systemd/system-generators/systemd-dbus1-generator
7329320 0x6FD628 Unix path: /lib/systemd/system/timers.target
7329832 0x6FD828 Unix path: /lib/systemd/system/shutdown.target
7330344 0x6FDA28 Unix path: /lib/systemd/system/sound.target
7330856 0x6FDC28 Unix path: /lib/systemd/system/bluetooth.target
7331368 0x6FDE28 Unix path: /lib/systemd/system/printer.target
7331880 0x6FE028 Unix path: /lib/systemd/system/paths.target
7332392 0x6FE228 Unix path: /lib/systemd/system/smartcard.target
7332904 0x6FE428 Unix path: /lib/systemd/system/sockets.target
7333416 0x6FE628 Unix path: /lib/systemd/system/busnames.target
7509800 0x729728 Unix path: /etc/ssl/private
7510312 0x729928 Unix path: /etc/ssl/certs
8214568 0x7D5828 Unix path: /etc/alternatives/lzma
Yes, it has files, but what else?
lilydjwg@archlinuxcn_group suggested me to use file -sk:
% file -sk /dev/sda
/dev/sda: DOS/MBR boot sector\012- DOS/MBR boot sector\012- Linux rev 1.0 ext4 filesystem data, UUID=6cb1caab-630e-4194-8f99-3046b3283306 (needs journal recovery) (extents) (large files) (huge files)\012- data
She and @MkfsSion thought that it could be simple a raw ext4 partition without partition tables.
This is a valid guess, and it is aligned with the fact that sda is a mounted ext4 partition.
However, it still did not solve the mystery of being able to boot using Grub.
Here, MkfsSion@archlinuxcn_group pointed out that ext4 has 1024B unused at the beginning to allow installing MBR boot sectors.
Then, I made the hypothesis that the Grub boot sector was actually written to the first 512B of the ext4 partition, and the machine successfully boots without a partition table.
This is an educated guess, and it is the same with current discoveries.
There is one problem left: Grub does not allow writing to an ext4 without --force option, and this feature was added before the installation time of this machine. Although I cannot remember what exact options of grub-install(1) I used, I am sure that I did not do anything with force or blocklist. Hmm.
The discovery #
Upon this point, I was considering to replicate the installation process locally by choosing the same ArchISO I used, so I went to check the exact time I installed this VPS, and this shows up:
% head -n1 /var/log/pacman.log
[2021-02-08T00:25:39+0000] [PACMAN] Running 'pacman -r /mnt -Sy --cachedir=/mnt/var/cache/pacman/pkg --noconfirm base linux lvm2 openssh reflector grub'
Wait, lvm2? Every time I use vps2arch, it will install that package for me, and I am always going to uninstall it. Having lvm2 in the pacman log indicates that I used vps2arch.
Then, it is clear: vps2arch could mess up Grub and ext4. It is easy to verify:
grub-install --target=i386-pc --recheck --force "$root_dev"
(from vps2arch#186)
Yes, vps2arch actually used the --force switch when installing Grub, causing grub-install(1) to write to the ext4 unused space.
Also, I checked that vps2arch won’t manually format filesystems. It will just use the old one from the previous Linux installation. Then, the sda-as-ext4 configuration must come from the Linode image I used, which was Ubuntu 16.04 LTS.
Finally, I created a new Linode VPS with Ubuntu 16.04 LTS, and this shows up:
Welcome to Ubuntu 16.04.7 LTS (GNU/Linux 4.4.0-210-generic x86_64)
* Documentation: https://help.ubuntu.com
* Management: https://landscape.canonical.com
* Support: https://ubuntu.com/advantage
The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.
root@localhost:~# mount | grep sda
/dev/sda on / type ext4 (rw,relatime,errors=remount-ro,data=ordered)
root@localhost:~# file -sk /dev/sda
/dev/sda: Linux rev 1.0 ext4 filesystem data, UUID=8e18baaa-f023-e588-8129-6a5c5d3ecb88, volume name "linode-root" (needs journal recovery) (extents) (large files) (huge files)\012- data
root@localhost:~#
Mystery solved.
Conclusion #
This is what happened:
-
I used the Ubuntu 16.04 LTS image from Linode.
-
This image does not have a partition table on sda. Instead, it formatted the whole disk to ext4.
-
I ran vps2arch on this Ubuntu, and the script did not change the partition table or reformat the partition.
-
The new Arch root was installed on sda.
-
The vps2arch script passed
--forceswitch to grub-install(1), causing it to install the Grub boot sector to the first 1024B unused space of ext4. -
I only noticed this issue when re-installing Grub today because Grub never needed re-installation.
I am not sure if it is an issue of Ubuntu or Linode, but the best practice is to backup, create a valid partition table, then format a new partition.
During the debugging process, I made the hypothesis that Grub was actully installed to the first 1024B unused space of ext4. Validating this hypothesis is left as an exercise to the readers: sda.img.
Thanks for reading. Thanks again to those people helped me in this process.