From Grub to Ext4 to MBR to Ubuntu: What the heck, Linode Ubuntu?

Debugging a strange disk layout on my Linode Arch VPS


I am running an Arch Linux VPS on Linode, and it has not been updated for years. In fact, I cannot even remember how did I installed it. The only thing I know is that it is installed in 2021 spring.

I ssh’ed onto it and typed pacman -Syu. According to recent Grub breaking changes, I need to run grub-install ... and grub-mkconfig again. That’s not a big problem, and I had been doing this for many other machines, so let’s go.

I double checked that the machine is using BIOS rather than UEFI boot.

… and let’s … go?


% grub-install --target=i386-pc /dev/sda
Installing for i386-pc platform.
grub-install: warning: File system `ext2' doesn't support embedding.
grub-install: warning: Embedding is not possible.  GRUB can only be installed in this setup by using blocklists.  However, blocklists are UNRELIABLE and their use is discouraged..
grub-install: error: will not proceed with blocklists.

What? I had never seen this before. Had it just destroyed my boot disk? Was I installing it on the right disk?

I also confirmed that it is booting using Grub (I was thinking whether it is possible for Linode to boot the kernel directly): Yes, since Grub appends BOOT_IMAGE= cmdline option, according to MkfsSion@archlinuxcn_group.

I thought I needed to double check my disks.

Disks … partitions? Where are you?

I was going to check if my partitions were still here:

% fdisk -l /dev/sda
Disk /dev/sda: 24.5 GiB, 26306674688 bytes, 51380224 sectors
Disk model: QEMU HARDDISK   
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x00000000

What? Shouldn’t there be partitions?

I also noticed that there is a disk called sdb. Nothing on this disk is mounted, but again let’s check it.

% fdisk -l /dev/sdb
Disk /dev/sdb: 512 MiB, 536870912 bytes, 1048576 sectors
Disk model: QEMU HARDDISK   
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Still nothing, not even Disklabel type: dos, and there aren’t other disks. How did it mount the root?

I was sure I am mounted.

% mount | grep sda
/dev/sda on / type ext4 (rw,relatime,errors=remount-ro)

% cat /etc/fstab 
/dev/sda                /               ext4            rw,relatime,errors=remount-ro,data=ordered      0 1

This did not seem right. Not a single person or manual told me that I should mount the whole hard disk as root (except when I have a separate boot device, which is what I am doing on my workstation), especially on a VPS where there is actually a single boot device.

However, the system is working. Although it has been running for 168 days, I am sure that I did not alter anything related to partitions or bootloaders since it booted.

Where did I get there? What the hell is going on?

Dumping the disk

The safest option was to have a dump of the first several KiB of the disks locally for investigation.

% dd if=/dev/sda of=./sda count=1024
1024+0 records in
1024+0 records out
524288 bytes (524 kB, 512 KiB) copied, 0.00814989 s, 64.3 MB/s

% file ./sda
./sda: DOS/MBR boot sector

% dd if=/dev/sdb of=./sdb count=1024
1024+0 records in
1024+0 records out
524288 bytes (524 kB, 512 KiB) copied, 0.0153935 s, 34.1 MB/s

% file ./sdb
./sdb: Linux swap file, 4k page size, little endian, version 1, size 131071 pages, 0 bad pages, no label, UUID=c0899d8a-1100-40fe-abfb-87e6ee0535e1

Hmm, we could safely eliminate sdb since it is a swap partition (also confirmed from Linode console).

For sda, it was more weird: file(1) said that it IS a MBR boot sector, also did fdisk(1). However, fdisk(1) did not show any partition. Moreover, it is a mounted ext4 partition. What?

Investigating the dump

To have more data, I dumped the first 8MiB of sda and pulled it to my local PC.

I firstly viewed the raw hex of this dump, and I found some strings related to Grub. (Apologizes for not remembering the MBR partition tabe layout or ext4 header structure).

I also checked using binwalk(1). My intent was to see if it actually has a valid ext4 filesystem on it. However, binwalk(1) did not show anything related to filesystems. Perhaps this tool does not support it?

% binwalk ./sda

1511464       0x171028        Unix path: /lib/systemd/systemd
1726250       0x1A572A        Unix path: /lib/python3.5/
1838632       0x1C0E28        Unix path: /etc/python3.5/
1906216       0x1D1628        Unix path: /etc/ssl/openssl.cnf
1997354       0x1E7A2A        Unix path: /lib/snapd/snap-confine
1997866       0x1E7C2A        Unix path: /lib/snapd/snapctl
1998376       0x1E7E28        Unix path: /usr/lib/snapd/snap-device-helper
3281192       0x321128        Unix path: /etc/alternatives/
3294760       0x324628        Unix path: /etc/alternatives/rmt
3295528       0x324928        Unix path: /lib/x86_64-linux-gnu/
3299368       0x325828        Unix path: /etc/alternatives/w
3300392       0x325C28        Unix path: /etc/alternatives/rview
3301416       0x326028        Unix path: /etc/alternatives/pager
3309352       0x327F28        Unix path: /lib/x86_64-linux-gnu/
7077928       0x6C0028        Unix path: /var/lib/usbutils/usb.ids
7095592       0x6C4528        Unix path: /etc/alternatives/rsh
7328808       0x6FD428        Unix path: /lib/systemd/system-generators/systemd-dbus1-generator
7329320       0x6FD628        Unix path: /lib/systemd/system/
7329832       0x6FD828        Unix path: /lib/systemd/system/
7330344       0x6FDA28        Unix path: /lib/systemd/system/
7330856       0x6FDC28        Unix path: /lib/systemd/system/
7331368       0x6FDE28        Unix path: /lib/systemd/system/
7331880       0x6FE028        Unix path: /lib/systemd/system/
7332392       0x6FE228        Unix path: /lib/systemd/system/
7332904       0x6FE428        Unix path: /lib/systemd/system/
7333416       0x6FE628        Unix path: /lib/systemd/system/
7509800       0x729728        Unix path: /etc/ssl/private
7510312       0x729928        Unix path: /etc/ssl/certs
8214568       0x7D5828        Unix path: /etc/alternatives/lzma

Yes, it has files, but what else?

lilydjwg@archlinuxcn_group suggested me to use file -sk:

% file -sk /dev/sda
/dev/sda: DOS/MBR boot sector\012-  DOS/MBR boot sector\012- Linux rev 1.0 ext4 filesystem data, UUID=6cb1caab-630e-4194-8f99-3046b3283306 (needs journal recovery) (extents) (large files) (huge files)\012- data

She and @MkfsSion thought that it could be simple a raw ext4 partition without partition tables.

This is a valid guess, and it is aligned with the fact that sda is a mounted ext4 partition.

However, it still did not solve the mystery of being able to boot using Grub.

Here, MkfsSion@archlinuxcn_group pointed out that ext4 has 1024B unused at the beginning to allow installing MBR boot sectors.

Then, I made the hypothesis that the Grub boot sector was actually written to the first 512B of the ext4 partition, and the machine successfully boots without a partition table.

This is an educated guess, and it is the same with current discoveries.

There is one problem left: Grub does not allow writing to an ext4 without --force option, and this feature was added before the installation time of this machine. Although I cannot remember what exact options of grub-install(1) I used, I am sure that I did not do anything with force or blocklist. Hmm.

The discovery

Upon this point, I was considering to replicate the installation process locally by choosing the same ArchISO I used, so I went to check the exact time I installed this VPS, and this shows up:

% head -n1 /var/log/pacman.log 
[2021-02-08T00:25:39+0000] [PACMAN] Running 'pacman -r /mnt -Sy --cachedir=/mnt/var/cache/pacman/pkg --noconfirm base linux lvm2 openssh reflector grub'

Wait, lvm2? Every time I use vps2arch, it will install that package for me, and I am always going to uninstall it. Having lvm2 in the pacman log indicates that I used vps2arch.

Then, it is clear: vps2arch could mess up Grub and ext4. It is easy to verify:

grub-install --target=i386-pc --recheck --force "$root_dev"

(from vps2arch#186)

Yes, vps2arch actually used the --force switch when installing Grub, causing grub-install(1) to write to the ext4 unused space.

Also, I checked that vps2arch won’t manually format filesystems. It will just use the old one from the previous Linux installation. Then, the sda-as-ext4 configuration must come from the Linode image I used, which was Ubuntu 16.04 LTS.

Finally, I created a new Linode VPS with Ubuntu 16.04 LTS, and this shows up:

Welcome to Ubuntu 16.04.7 LTS (GNU/Linux 4.4.0-210-generic x86_64)

 * Documentation:
 * Management:
 * Support:

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

root@localhost:~# mount | grep sda
/dev/sda on / type ext4 (rw,relatime,errors=remount-ro,data=ordered)
root@localhost:~# file -sk /dev/sda
/dev/sda: Linux rev 1.0 ext4 filesystem data, UUID=8e18baaa-f023-e588-8129-6a5c5d3ecb88, volume name "linode-root" (needs journal recovery) (extents) (large files) (huge files)\012- data

Mystery solved.


This is what happened:

  1. I used the Ubuntu 16.04 LTS image from Linode.

  2. This image does not have a partition table on sda. Instead, it formatted the whole disk to ext4.

  3. I ran vps2arch on this Ubuntu, and the script did not change the partition table or reformat the partition.

  4. The new Arch root was installed on sda.

  5. The vps2arch script passed --force switch to grub-install(1), causing it to install the Grub boot sector to the first 1024B unused space of ext4.

  6. I only noticed this issue when re-installing Grub today because Grub never needed re-installation.

I am not sure if it is an issue of Ubuntu or Linode, but the best practice is to backup, create a valid partition table, then format a new partition.

During the debugging process, I made the hypothesis that Grub was actully installed to the first 1024B unused space of ext4. Validating this hypothesis is left as an exercise to the readers: sda.img.

Thanks for reading. Thanks again to those people helped me in this process.