Kernel Panic VFS Error and TPM Bypass (Part 2)

Solution 2: Fixing Incomplete Kernel or System Package Installation

1. Create an EBS Snapshot:

   • Follow instructions to Create Amazon EBS snapshots.

2. Open Amazon EC2 Console:

   • Ensure you're in the correct Region.

3. Stop Impaired Instance:

   • Navigate to Instances, select the impaired instance.
   • Choose Instance State, Stop instance, and confirm.

4. Detach Root Volume:

   • In the Storage tab, under Block devices, select the Volume ID for /dev/sda1 or /dev/xvda.
   • Choose Actions, Detach Volume, and confirm. Note the Availability Zone.

5. Launch Rescue Instance:

   • Launch a rescue EC2 instance in the same Availability Zone.

6. Attach Detached Volume:

   • Once the rescue instance launches, go to Volumes, select the detached root volume of the impaired instance.
   • Choose Actions, Attach Volume.
   • Choose the rescue instance ID and set an unused device (e.g., /dev/sdf).

7. Connect to Rescue Instance:

   • Use SSH to connect to the rescue instance.

8. Mount Root Partition:

   • Run lsblk to view available disk devices.
   • Mount the root partition of the mounted volume to /mnt: mount -o nouuid /dev/xvdf1 /mnt.

9. Set Up Chroot Environment:

   • Mount /dev, /run, /proc, and /sys of the rescue instance to the same paths as the mounted volume: for m in dev proc run sys; do mount -o bind {,/mnt}/$m; done.
   • Enter the mount directory:

     1   chroot /mnt
     

10. Update GRUB Bootloader:

• Follow the appropriate procedure based on your OS:

For GRUB1 (Legacy GRUB) - Red Hat 6 and Amazon Linux 1: Use sed command to replace the corrupt kernel with the stable kernel in "/boot/grub/grub.conf":

1  sed -i '/^default/ s/0/1/' /boot/grub/grub.conf

For GRUB2 - Ubuntu 14 LTS, 16.04, and 18.04:

• Replace corrupt GRUB_DEFAULT=0 default menu entry with stable GRUB_DEFAULT=saved value in /etc/default/grub.

1  sed -i 's/GRUB_DEFAULT=0/GRUB_DEFAULT=saved/g' /etc/default/grub

• Run "update-grub" command.

1 update-grub

• Run "grub-set-default" command, so that the stable kernel loads at the next restart :

1  grub-set-default 1     

For GRUB2 - RHEL 7, Amazon Linux 2:

• Replace corrupt GRUB_DEFAULT=0 default menu entry with stable GRUB_DEFAULT=saved value in /etc/default/grub.

  1  sed -i 's/GRUB_DEFAULT=0/GRUB_DEFAULT=saved/g' /etc/default/grub
  

• Update GRUB to regenerate /boot/grub2/grub.cfg.

1   grub2-mkconfig -o /boot/grub2/grub.cfg

• Run grub2-set-default command.

1  grub2-set-default 1

For GRUB2 - RHEL 8 and CentOS 8: GRUB2 in RHEL 8 and CentOS 8 utilizes blscfg files and entries in /boot/loader for boot configuration, departing from the previous grub.cfg format. It's advisable to employ the grubby tool for managing blscfg files and retrieving information from /boot/loader/entries/. If these files are missing or corrupted, grubby may not yield any results, necessitating their regeneration for functionality restoration. Kernel indexing depends on .conf files in /boot/loader/entries and kernel versions, prioritizing the latest kernel with the lowest index.

• View Current Default Kernel:

1grubby --default-kernel

• List all Available Kernels and Indexes:

1grubby --info=ALL

Example Output:

1  index=0
2  kernel="/boot/vmlinuz-4.18.0-305.el8.x86_64"
3  ...
4  index=1
5  kernel="/boot/vmlinuz-0-rescue-0c75beb2b6ca4d78b335e92f0002b619"
6  ...
7  index=2
8  kernel="/boot/vmlinuz-4.18.0-305.3.1.el8_4.x86_64"
9  ...

• Set Default Kernel:

  • Identify the desired kernel path (e.g., /boot/vmlinuz-4.18.0-305.3.1.el8_4.x86_64).
  • Execute below command:

  1 grubby --set-default=/boot/vmlinuz-4.18.0-305.3.1.el8_4.x86_64
  

  Note: Replace "4.18.0-305.3.1.el8_4.x86_64" with your kernel's version number.

• Verify Default Kernel:

  1grubby --default-kernel
  

11. Unmount Volumes and Detach Root Volume:
    • Exit chroot:

    1  exit
    

    • Unmount /dev, /run, /proc, and /sys:

    1umount /mnt/{dev,proc,run,sys}
    

    • Stop the rescue instance and detach the root volume from it.
    • Attach the root volume to the impaired instance as the root volume (/dev/sda1), then start the instance.

The stable kernel should now load, and your instance should start successfully.