Making a portable full installation of Ubuntu on a USB HDD

I regularly have the need to try things out on Linux. Sometimes a virtual machine won't cut it for me typically due to memory, disk and performance limitations. Moreover, a decent, up-to-date, bootable Linux environment is a great backup in case all of my other computers are broken, infected or stolen. That entails having the Linux installation on an external, USB-attached hard disk drive which can boot with relative ease on any UEFI-enabled PC (driver compatibility notwithstanding). Moreover, all the preparatory work has to be performed using a single-boot Windows computer without ending up having a dual boot system. It sounds tough. It is tough, but I'm writing this from my portable Ubuntu Linux installation running off a USB-attached SSD!

What you need

  • An empty external (USB) hard drive. I used a USB 3 drive enclosure with a cheap 256Gb SSD. For those of you worrying about performance, the USB 3.0 port is faster than the maximum transfer rate of any SSD I've seen to this date.
  • Ubuntu Linux bootable USB drive. Very easy to create using Rufus on Windows. I used a cheap, promotional flash drive. Reduce, reuse, recycle FTW.
  • Windows System Repair Disc (a bootable USB drive with Windows recovery tools which you can make yourself) or a Windows installation or rescue CD-ROM / DVD-ROM. I used another promotional flash drive.

Remember to take backups of all your critical files and folders before continuing. You are going to make changes to your computer. If it breaks don't cry, I warned you. Also test both your bootable media. Twice. You will need them both!

Caveats

Having a dual- or multi-boot system isn't as straightforward as it used to be a decade ago. Nowadays, in the interest of security, performance and backwards compatibility, there are many semi-hidden options and features which can get in your way. Given enough experience and patience you can work around them. Below is a selection of problems that got me moderately stumped along the way.

Secure boot caveat

I have only tested these instructions with Secure Boot turned off. Even though Ubuntu does support Secure Boot (it comes with signed bootloaders) I have no idea if my method uses the signed bootloaders or not. I suggest turning off Secure Boot if possible.

FastBoot caveats

Many boards come with some sort of "fast boot" or "boot optimization" options. For example, my Intel NUC has an option called Fast Boot which won't let me choose an alternative boot device at startup. It also has an option to support Intel Rapid Start Technology which does get in the way of booting to multiple OS. I had to disable both.

It's worth noting that Windows 8 and 10 have an Fast Startup or Fast Boot feature. This feature takes some shortcuts when it comes to booting and also makes the NTFS filesystem remain in a dirty state, making it unwriteable from Linux. It's best to understand what it does and disable it if you plan on writing to your Windows drive from Linux.

USB host controller caveat

Some firmwares will present the USB host controller as UHCI (USB 1.1) at boot time. When Linux probes for an xHCI (USB 3) host controller during the boot process they will respond positively. At this time Linux loads the xHCI driver and the USB host controller resets itself.

However, your root filesystem is inside a device attached to this USB controller. Therefore the controller resetting means that Linux can no longer communicate with the USB-attached hard drive. Therefore the Linux boot will hang forever without any further indication as to what went wrong.

Most affected boards (including my Intel NUC) have an option to enable the xHCI host controller interface by default. Enabling the xHCI option in the BIOS fixes the hanging boot issue. If you are only using modern operating systems with USB 3 support (anything newer than and including Windows 8.1 and Ubuntu Linux 15.04) you can safely enable that option.

Installing Linux

Boot your computer from the Ubuntu Linux bootable USB drive. Remember that you may have to enter your computer's boot manager to do that (on my Intel NUC I have to press F10; on most other BIOS I've seen it's F9; consult your BIOS documentation).

Install Ubuntu Linux regularly. When it prompts you about the disk layout choose Something Else and partition your external HDD the way you want. I chose to create a modestly sized root partition (about 40Gb), a swap partition that's as big as the biggest RAM configuration I am going to be using this installation with plus one Gb (my computers max out at 16Gb so I made a 17Gb swap partition) and the rest of the disk went to a massive /home partition.

Caveat: I chose to use btrfs which – as I learned along the way – makes things a bit more complicated down the line. For your sanity's sake I recommend using ext4. This guide assumes the use of btrfs and will point out the caveats with this approach.

Fix Windows 10 boot

Unfortunately the Ubuntu Installer assumes that you want a dual booting configuration alongside Windows. Therefore it adds itself (actually, the bootloader it uses, GRUB2) to your computer's UEFI configuration. This causes two problems. For starters, the external HDD is not portable as you cannot boot with it on another computer.

Secondly, if you remove this external HDD your Windows won't boot. Bummer. We have to fix that.

  • Shut down your computer.
  • Disconnect the Ubuntu HDD
  • Boot from the Windows system repair disc USB drive (or a Windows installation or rescue CD-ROM / DVD-ROM).
  • Select Repair your computer.
  • Select the operating system and click Next.
  • Choose Command Prompt.
  • In the command prompt run 
    diskpart
sel disk 0
list vol
  • Verify that the EFI partition is using the FAT32 file system. It will have a volume ID, let's say 99. Now we need to assign a drive letter to it. Back in the command prompt type: 
    sel vol 99
assign letter=z:
exit
  • Now we need to fix the boot record. Again in the command prompt type: 
    z:
cd EFI/Microsoft/Boot
bootrec /FixBoot
  • Finally, we need to re-create the BCD store which tells the Microsoft boot loader where to find Windows so it can boot it. From our trusted command prompt: 
    ren BCD BCD.old
bcdboot c:\Windows /l en-us /s z: All
  • If this didn't work try 
    ren BCD BCD.old
bootrec /RebuildBcd

At this point exit the command prompt and shut down your computer.

Create an ESP on the Ubuntu HDD

A hard drive is not bootable with UEFI unless it has an ESP (EFI System Partition). An ESP is simply a FAT32 partition with a special flag that tells the EFI BIOS to look inside it for boot information. We have to create one on your hard drive.

  • Plug in your external HDD and the Ubuntu Linux bootable USB stick.
  • Boot with the Ubuntu Linux bootable USB stick using the option to try Ubuntu before installing.
  • Open a Terminal (CTRL-ALT-T)
  • Run sudo fdisk -l to get a list of partitions.
  • Identify from them the drive that has the Linux partitions, in my case /dev/sdb. I'll call it /dev/sdX from now on.
  • Also identify the partition that contains the root filesystem. I will call it /dev/sdXY from now on.
  • Launch GParted from the Terminal: 
    sudo gparted /dev/sdX
    Why not just click on GParted on your desktop? Well, I kept receiving errors about the Ubuntu Linux bootable USB stick because it was already in use. Of course it is, I am using it to run the computer off it, duh!
  • Resize the first partition on disk to have another 200 Mb of free space after it.
  • Create a new partition on the free space, changing the file system to fat32.
  • Apply operations. You need to do that now for the next step to be possible.
  • Right click the new partition.
  • Click on Manage Flags.
  • Set the boot and esp flags. This is what makes the partition "special" to the EFI BIOS.
  • One more thing! Note down the the partition that contains the ESP filesystem. I will call it /dev/sdXZ from now on.

Make sure the Ubuntu installation on the external HDD can see the ESP

The new ESP on the external drive must be visible by the Ubuntu installation in the HDD. Otherwise GRUB2, the Linux bootloader, won't be able to update itself, making your system unbootable after the next kernel update at the latest.

  • Launch GParted from the Terminal, as we saw above: 
    sudo gparted /dev/sdX
  • Double click the partition with your Linux root (/) filesystem on the external HDD
  • Note down the UUID, e.g. 01234567-89ab-cdef-0123-4567890abcde
  • Double click the new FAT32 partition and note down the UUID, e.g. 0123-ABCD
  • Close GParted
  • Open a Terminal

The process is different depending on the format of your root partition on the external hard disk.

If you DID NOT use btrfs (e.g. you used ext4)

sudo umount /media/ubuntu/01234567-89ab-cdef-0123-4567890abcde 
sudo mount /dev/sdXY /mnt

If you DID use btrfs

If you DID use btrfs, you made your life complicated. We need to mount the btrfs subvolume containing the root partition instead of the entire partition. Otherwise you'll never be able to install GRUB and you'll probably lose an entire day, like me.

btrfs subvolume list /media/ubuntu/01234567-89ab-cdef-0123-4567890abcde

This will give you a line with a numeric ID. Let's say 123. Note it down.

umount /media/ubuntu/01234567-89ab-cdef-0123-4567890abcde
mount /dev/sdXY -o subvolid=123 /mnt

The rest of the instructions are common, no matter if used btrfs, ext4 or something else

  • sudo nano /mnt/etc/fstab
  • There is a line with /boot/efi already in this file. Comment it by placing a # in front of it.
  • Add the following line: 
    UUID=0123-ABCD /boot/efi vfat defaults 0 1

Install GRUB2 on the external drive's EFI System Partition

Right now our external drive has an empty ESP. We need to put a bootloader in it to make it actually, well, bootable.

First caveat: all the instructions you find on-line assume you are using a dual boot system with Windows or macOS. When you have an external drive it is critical that you use the --removable option in the last step. This installs the EFI bootloader under the special "fallback path" EFI\Boot\bootx64.efi in the ESP. Normally this not supposed to be used for permanently installed Operating Systems. It's the mechanism used by EFI BIOS to boot arbitrary external media. Technically, that's exactly what our external hard drive is: arbitrary external media!

Second caveat: installing the bootloader is only possible from inside the Linux installation we want to boot. However, we need the bootloader to boot that installation, leading to a Catch-22 issue. The solution is to run the bootloader installation through a chroot jail. The actual caveat that got me stumped for a day comes from the fact that I am using btrfs (because it's so much better for SSDs!). btrfs has subvolumes. If you mount the entire partition instead of a subvolume the grub-install script can't figure out the mapping between paths and devices, therefore failing to install itself on the ESP, returning the cryptic error

/usr/sbin/grub-probe: error: cannot find a device for / (is /dev mounted?).

The error is misleading! /dev is mounted if you follow my instructions below. The actual problem, as I understand it, is that there is a discrepancy between the mounted device and the path to the chroot root. That's why I had you mount only the subvolume containing the root filesystem in the steps above. If you were not paying attention, you are not following the instructions step-by-step, you rebooted before this step or just came here directly looking for a solution to your problem about GRUB not installing look above for instructions on mounting the correct btrfs subvolume.

  • We need to prepare the chroot environment. The ESP must be mounted in the correct place and we have to bind system mount point for some special trees (most notably /dev). Moreover, we will copy the resolv.conf file to let the chroot environment have network access should it need it. 
    mount /dev/sdXZ /mnt/boot/efi
for i in /dev /dev/pts /proc /sys; do sudo mount -B $i /mnt/$i; done
cp /etc/resolv.conf /mnt/etc/
modprobe efivars
  • Finally we enter the chroot environment and install Grub in a way suitable for a removable device (see the first caveat above). 
    sudo chroot /mnt
grub-install -d /usr/lib/grub/x86_64-efi --efi-directory=/boot/efi/ --removable /dev/sdX

Now your external HDD is bootable. Reboot your computer, select it from the boot media selection of your UEFI BIOS and you're done!

201 comments

    • Your filesystem may have a problem and gets mounted read only. Why don't you start in single user / repair mode and do a fsck to see where's the problem? This is standard Linux admin :)
      • Hi,

        It says that "Inodes that were part of a corrupted orphan list were found."

        After I fsck, the system is still read-only.
        • With the power vested to me by Google I present thee a thread relevant to your question https://forums.opensuse.org/showthread.php/467196-Error-corrupted-orphan-linked-list

          It only took me a few hours until I had the time to do that 2-minute search you could have done yourself. 

          Hint: if you see something you don’t understand on your screen search for it online. If you expect other people to do your homework for free you’re setting yourself up for some major disappointment in the long run. 
  • For the step Fix Windows Boot, I use a software Disk Genius in Windows, delete the Ubuntu folder in Windows ESP partition, it works.
  • Almost all the way through... But, when I try to mount sdxz as sudo mount /dev/sdXZ/ /mnt/boot/efi , an error pops up: mount: /mnt/boot/efi: mount point does not exist.

    I changed the difectory to mnt and saw it was empty. Any suggestions?
    • The instructions right above that bit tell you how to mount your external disk into /mnt. You either didn't do that or you got an error and you missed it.
  • Thanks everybody.  I had an old boot of debian I think (It's been a while since I loaded it), and it was just cloned to other drives using gparted, and later, a copy of partition guru, then power iso.  It was so versatile I had a few freelance techs using it to run cleanup\recovery opps whenever we needed to.  We loaded that, and a WinPE rescue disk onto several drives, allowing us to clean up most problems just before they hit the "sorry, it's dead" stage.  Back then, we had a single tower with a P4 and a few gigs of ram.  A 1gb swap area was enough to be very fast by comparison to the winpe.  We had upgraded that one tower for a main software setup several times, as it ran ubuntu, winpe, and a full version of windows server 2012, which we used for running everything.  6 weeks ago, I took some time off, to handle family issues and school.  One of the newbees tossed the machine on the garbage heap.  So, I've decided that we're all going to carry a drive with some system tools, and we'll carry one of the working laptops for onsite work, rather than attempting to hook everything to a windows server running virtualboxes with winpe.  We'll upgrade that machine again, but it will be for the most extreme cases.  Otherwise, working laptops with an i5 or better suffice.
    Your posts here were a great refresher, and an update in the use of EFI, thank you.  ALL of you.  And to add my two cents...
    SSD's and usb drives are not meant as permanent storage.  They are less permanent than magnetic data at the moment, but are rapidly closing the gap.  They are meant for storage with fewer changes, but new drivers and controllers are allowing them to spread the workload out, stretching their lifespan.  They don't succumb to as many bad blocks, but sectors will burn out, and you may have to repair an install once in a while.  For DATA, cloud and HDD RAID provide speed with longevity.  Cloud as a service lets somebody else worry about the maintenance, while you have constant access.
    In corporate, it's a balancing act.  At home, I use an HDD for my data, and an ssd for system.
    • SSDs are definitely meant for permanent storage the last 8 or so years. 15 years ago, no they were not. MLC flash storage suffered from electron bleed which could cause bit flips if the device was not powered on for over 1 to 3 years. Technology has progressed ever since and now it's just as reliable as spinning disks as evidenced by its use in the data center and every single portable device save for cheap, crappy laptops.

      Now, USB sticks are a different issue altogether. The flash storage used there is not even near the level of a modern SSD and yes, they are neither as fast nor as reliable. That's why I would NEVER use them for anything more serious than temporary storage or file transfer. But putting them in the same basket as SSDs is just plain wrong.
  • I have tried that, it didn't work. Even disabling the Secure Boot, selecting the external HD in the boot menu, the system hangs in a black screen.
    • If you saw the GRUB menu the problem is probably that your hardware requires some kernel parameter. Google search is your friend. I remember that I HAD to use a parameter for my Intel NUC to activate the onboard graphics. I can't check what it is now since that machine is currently at my vacation home.

      If you didn't see the GRUB menu at all there are three possibilities. One, the EFI partition is not formatted as FAT32, is not marked as an EFI partition or does not contain the correct EFI binary. Check the instructions above carefully. Two, GRUB is not installed on the correct partition or the partitions you tell it to look for the kernels are wrong (hint: check the UUIDs in /etc/fstab and compare them with those returned by blkid). Three, everything is set up properly, the drive boots on another machine but your current machine is finicky / has a broken UEFI. My Intel NUC was giving me absolute hell with that. I had to unplug everything, wait 30", hold down the power button and then plug in the power. Once in five tries it worked. The same drive was booting perfectly with my Slimbook Katana KDE and even with a SurfaceBook Pro. Go figure.

      Booting from USB is hard and mostly a crapshoot. The more you read about how it works the less plausible it sounds that it can ever work, at all...
  • When I mount my newly made fat32 partition, there's nothing in it. i set the flags and everything haha. am i missing a step?
    • If you just created a brand new partition and formatted it as FAT32 it stands to reason that it has nothing inside it. Unlike EXT3/EXT4 partitions, FAT32 partitions don't have a lost+found directory (in fact, that name would be illegal in a FAT32 partition) since they don't have a feature for recovering lost inodes.
  • Hi nick,
    great guide and amazing dedication to the comments section by the way.
    Im having trouble with the USB host controller.
    It seems your fix is missing from my HP envy's bios settings :/ is there any other way to enable the xHCI on boot setting?
    Cheers
    • It depends entirely on the firmware of your laptop. FWIW the NUC gave me grief half of the time I tried booting it up. Ironically, a Microsoft Surface Book and a Mac Mini I was also using this solution on didn’t bat an eyelid and worked the first time around. So I guess it’s just very hit and miss.
  • Would remove the internal hard drive during the install make it easier? You wouldn't have to fix Windows 10 partition that way.
    • It depends on whether it's possible to physically remove the drive (newer laptops have NVMe drives soldered to the motherboard) and how your UEFI firmware reacts to its main boot option being removed between cold boots. Please read the previous comments where I have discussed about it in length.
  • Hi, NIcholas! I tried your instructions step by step, but it didn't work for me. I was trying to install Ubuntu 18.04 on Seagate M3 1TB external USB HDD, and after a long fight, I found that the problem was actually in the size of the FAT32 partition that had to be created for the boot loader. It has to be LARGER THAN 512 MB, otherwise it won't work. Here is what I did:
    1. I connected both the USB HDD and the USB flash drive with the Ubuntu startup burned on it to the computer (Laptop ASUS X55C).
    2. Started the computer. I had to press the "Esc" button to see the boot menu; I chose "UEFI: Udisk 8GB", which is my Ubuntu 18.04 flash drive.
    3. I created new partition table (msdos) in the Seagate HDD using gparted, and since the Ubuntu installation partitioner kept going on with the annoying message that I hadn't used the correct alignment, I made all the partitions needed for the installation with gparted. First, I made a logical swap of 16GB at the end of the freed space. Second, I made a primary 650MB FAT32 with boot, ESP flags at the beginning of the free space, and at last - in the remaining middle - an ext4 partition.
    4. After these actions, it was time (finally) to start the installer. When I was asked where to install, I chose "something else". On the partition menu I found the Seagate HDD mounted as /dev/sdb. /dev/sdb2 was my FAT32 recognized as EFI partition, /dev/sdb3 was my ext4 partition, and /dev/sdb1 was the swap partition. I order to set the mount point of the ext4, I had to select it and click "change" on the menu. Done.
    5. The final step: on the bottom of the partition menu, there is an applet where you choose, where to install the boot loader - I chose /dev/sdb (the Seagate USB HDD). This is very important.
    6. Installation.
    After installation I restarted... ''Esc'' button, and... YESSSS - a miracle! Finally I saw the long expected "UEFI: Seagate M3"... and it successfully booted!
    Before this success I got only "Seagate M3" on the menu, without "UEFI", which wouldn't boot at all.
    I hope this little clarification about the FAT32 partition will help someone, well - it's working for me.
    Wish you all the best!
    • It sounds like you put the entire /boot in the FAT32 partition which is wrong (but explains why you needed that much space). The FAT32 partition must be mounted to /boot/efi. You can either have a separate ext4 partition for /boot or just leave /boot in your root partition (as long as you are not using an encrypted disk, that is). /boot needs to be at least 512MB, ideally 1GB, in size since it holds all the kernels, GRUB2, initial RAM disks and a few odds bits and pieces. /boot/efi (the EFI system partition) needs to be FAT32 and can be as small as 100MB since it only holds the EFI stubs which kickstart the boot loader.

Login

Tags

Joomla! how-to opinion developers PHP Linux security review macOS Windows Mandriva book