remote authentication for whole disk encryption

the (skippable) background

Update 31August2019: Ugh. It all gets rather more complicated than I like by the time you get WiFi and the Tor hidden service running. I was thinking of creating an SD card image, but then I’d have to maintain it for each new release of Raspbian. Josh suggested that, instead, I create a bash script to do the whole install to a vanilla Raspbian. If you’ve tried creating this and got lost, stay tuned and I’ll do the installer as time allows. Email me if you want to encourage me to do it sooner.

I am a fan of whole-disk encryption. It is just about the only way to insure that a lost, stolen, confiscated, or discarded machine doesn’t leak information.

Continue reading remote authentication for whole disk encryption

TidBITS

TidBITS logo

I just renewed my membership to TidBITS. If you’ve never heard of TidBITS, you should check them out. They have been publishing continuously online for 28 years, and their content is second to none. They also host an amazing online discussion list, TidBITS-Talk. The site’s content is focused on Apple news and technology, but the quality of the journalism is so high and the coverage is broad enough that there is likely something there for everyone.

Membership is completely optional, so check it out. It is not hyperbole to say that TidBITS has improved the quality of my life every week for the past quarter century.

–Ron

Making Linux Bootable Clones

A bootable clone is a disk that holds a complete copy of a running system that’s ready to boot. Bootable clones can be a critical part of your backup strategy.

Why Bootable Clones

If you already know why you want a bootable clone, just skip to the recipe.

If you’re truly hardcore, and just want a command reference so you don’t leave out any steps, skip to the expert mode recipe.

If you have mission critical computers in your life, I hope you are already aware of the importance of good backups. I’ve noticed, though, that a lot of people think that just having copies of all their files is enough. They’ll set up, say, an automated cloud backup — and figure they’re done.

They’re probably in for a nasty surprise if disaster hits.

A lot of the information in your computer isn’t in the files, but in various nebulous kinds of metadata: where the files are located, various configuration files, license keys, special directories, file links, permissions… Just restoring all your files isn’t enough if you want to be back online as fast as possible.

My primary laptop is a macOS machine. I’ve long employed SuperDuper and Carbon Copy Cloner to make bootable clones. If my laptop gets destroyed, lost, or stolen, all I have to do is plug a clone into any recent-vintage Macintosh, reboot, and I’m back in production. I don’t even have to wait for files to copy. By contrast, last year I tried a scratch restore from a Time Machine backup, and it took almost two days to get reasonable working system, and several weeks after that before most of the little glitches were smoothed out. In fairness, my setup is a lot more complicated than the average user, but I also get the sense that most users are largely unaware of how tied they are to complicated configurations that they’ve (consciously or otherwise) fine-tuned to their workflow over the course of years.

I also maintain a hundred or so Linux-based systems. I’ve searched from time to time, but surprisingly I’ve never found a SuperDuper-like utility for making bootable clones. It’s long been on my list of things to create, but a recent systemd-related disaster left me with a crashed mission-critical server for over five days. That, combined with a horrible bug in the duplicity backup system, resulted in more downtime than all of my systems combined over the previous ten years.

Current Options

  • Clonezilla isn’t a bad option. It will create a sector-by-sector clone onto a new hard drive. It is file-system agnostic so your copy is almost guaranteed to work as well as the original. Optionally, you can put your clones into image files. They won’t boot, but can be restored to then-bootable media. However:
    • Clonezilla requires that you boot from a live CD to make your backups. That means that your system is unusable during the hours that it can take to make the backup (not acceptable for mission-critical servers).
    • Clonezilla requires that the system have a working optical drive.
    • When using Clonezilla, you’re pretty much stuck with working from the console (or spending time configuring ssh). In my world, that means a certain amount of physical discomfort. I much prefer the comfort of working from my mother’s basement, chugging Mountain Dew while my Dorito-dust covered fingers clack away on my sticky gaming keyboard.
    • Clonezilla requires that the destination disk be as large or larger than the source. I once tried to make a clone of a 2 terabyte server disk to an “identical” 2 TB disk. For whatever reason (I didn’t investigate, possibly some defective sectors), the destination drive was a few blocks smaller than the source disk. After many hours of copying, the backup failed.
  • Filesystem copies using cp, tar, or rsync can make sure that you get all your files. The usual advice is then “install a fresh system, and copy the important files from your backup to the new bootable system.” Guess what: it won’t work. Yes, you can copy your /home directory, but that won’t restore any of your system configuration. Over time, you’ve probably installed multiple software packages with multiple configuration files, perl and python modules, all scattered through the /etc and /var directories and configuration files, thoroughly mixed in with install-specific files and data that you can’t just replace with information from the old system. You’ll lose log files. You’ll lose startup sequences. Unless you run a vanilla system (and I doubt any plain-vanilla users are here reading this) you’ll be in for hours and hours of reconfiguration and figuring out which “important” files should be copied, and which must not be, and which configuration files need to be carefully merged with those on the new system.
  • dd. I use and love dd, and it meets my criterion for simplicity, but it has some real limitations.
    • The destination drive has to be as large or larger than the source drive.
    • It isn’t smart about what it copies. I have a server with a 1.5TB drive, but it’s only using 86GB (only a fraction of which changes between backups). When using dd, I pretty much have to copy the entire 1.5TB which takes hours. Yes, I could re-partition the drive and only dd the partition but then the clone wouldn’t be bootable and I’d lose the simplicity.
    • It doesn’t do incremental backup.

Simplicity is Key

People get cagey about backups. They want to be selective (who needs to be able to access three years of temporary cache files?). They want to conserve backup space. That’s all well and good, but your bootable clone is not the place to do this. Just buy a backup drive that’s as large or larger than your primary and back up everything. You don’t want to be scrambling to get back online and discover that a rule you created to save a few MB of drive space inadvertently blocked the backup of a critical file.

Creating a Bootable Clone

This technique has been tested with Ubuntu 14.04, Ubuntu 18.04, Devuan ascii, as well as recent Arch releases. It should work with Debian-derived distros and other systems that use the GRUB2 bootloader.

Get Ready

  • plug in backup drive
  • open a bash shell; you need to be root so…

sudo -s

  • enter your passphrase
  • next, figure out which drive you’re working with

fdisk -l

  • and identify which drive is the backup drive; we’re going to assume its /dev/sdx. So remember: whenever you see /dev/sdx, be sure to replace the x with the appropriate device indicator.

First Time? Prepare the Bare Drive

WARNING! This will erase all data on whatever drive you specify. Specify the wrong drive, and you’ll erase your source instead of your destination. Be careful!

  • If you already have a bootable clone, skip the next three steps and you’ll just incrementally update your clone.
  • The next three steps initialize a bare drive for first-time cloning. Whatever device you specify *will be erased,* so make sure you’re erasing the disk you think you are. Remember to change /dev/sdx to your actual device designation. You might have to repeat the “d” command several times if there are multiple existing partitions. When in doubt, use defaults.

fdisk /dev/sdx

  • here’s what your fdisk session should look like — though you will have to display the list of partition types and use the version-specific code for the EFI and swap partitions:
Welcome to fdisk (util-linux 2.29.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Command (m for help): d
Partition number (1-3, default 3):

Partition 3 has been deleted.
[repeat until you get an error because all the partitions have been deleted]

Command (m for help): n
Partition type
p primary (0 primary, 0 extended, 4 free)
e extended (container for logical partitions)
Select (default p):

Using default response p.
Partition number (1-4, default 1):
First sector (2048-468862127, default 2048):
Last sector, +sectors or +size{K,M,G,T,P} (2048-468862127, default 468862127): +512M

Created a new partition 1 of type 'Linux' and of size 512 MiB.

Command (m for help): t
Selected partition 1
Partition type (type L to list all types): XX
Changed type of partition 'Linux' to 'EFI (FAT-12/16/32)'.

Command (m for help): n
Partition type
p primary (1 primary, 0 extended, 3 free)
e extended (container for logical partitions)
Select (default p):

Using default response p.
Partition number (2-4, default 2):
First sector (1050624-468862127, default 1050624):
Last sector, +sectors or +size{K,M,G,T,P} (1050624-468862127, default 468862127): +213G
[subtract the desired swap size from the total disk size; this was a 223GB disk as reported by fdisk, and I want about a 10GB swap]

Created a new partition 2 of type 'Linux' and of size 213 GiB.

Command (m for help): n
Partition type
p primary (2 primary, 0 extended, 2 free)
e extended (container for logical partitions)
Select (default p):

Using default response p.
Partition number (3,4, default 3):
First sector (447744000-468862127, default 447744000):
Last sector, +sectors or +size{K,M,G,T,P} (447744000-468862127, default 468862127):

Created a new partition 3 of type 'Linux' and of size 10.1 GiB.

Command (m for help): t
Partition number (1-3, default 3):
Partition type (type L to list all types): XX

Changed type of partition 'Linux' to 'Linux swap'.

Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.
  • Format the UEFI partition. You might need to install dosfstools (eg. apt install dosfstools) to create the FAT partition.
mkfs.fat -F32 /dev/sdx1
  • Create the file system. Remember to edit /dev/sdx2 appropriately.

mkfs.ext4 /dev/sdx2

  • Set up the swap partition.
mkswap /dev/sdx3
  • Mount the file system. Remember to edit /dev/sdx2 first.

mount -t ext4 /dev/sdx2 /mnt

  • Create some special directories.

mkdir /mnt/dev
mkdir /mnt/dev/pts
mkdir /mnt/sys
mkdir /mnt/proc

Mount and Copy

  • Mount the clone you’re updating. Skip this command if you’re starting from scratch, as you’ve already mounted the new drive.

mount -t ext4 /dev/sdx2 /mnt

  • Because we’re using rsync, if a clone already exists on this drive, it will incrementally update which should save some time.

rsync --archive --verbose --delete --exclude=/dev --exclude=/sys --exclude=/proc --exclude=/mnt -xx / /mnt

Install GRUB2

  • Now you have the files copied, but you have to install a bootloader to make the drive bootable. To do this, we’ll chroot to the clone disk after making some special system directories available there.

mount --bind /dev /mnt/dev
mount --bind /dev/pts /mnt/dev/pts
mount --bind /sys /mnt/sys
mount --bind /proc /mnt/proc
mkdir /mnt/boot/efi
mount /dev/sdx1 /mnt/boot/efi
chroot /mnt

  • Install the bootloader. Remember that you have to fix up /dev/sdx in two places before executing these commands.
grub-install /dev/sdx
grub-install --recheck /dev/sdx
update-grub

Clean up /etc/fstab

  • This is a trickier part of the process. Systems are likely to be configured to use other disk partitions for /boot or /home or for swap space. In order to keep things clean and simple, I’ve made the clone into a monolithic file system. Unfortunately, the cloned /etc/fstab might cause the system to fail to boot. These commands attempt to create an fstab that’s good enough to boot. (Swap might be important to you; see the note at the end of this article for information on re-enabling swap on a clone.)
  • First, back up the existing fstab.

mv /etc/fstab /etc/fstab.bak

  • In this command, look out for the buried /dev/sdx2 that needs to be edited!

echo -e "UUID=`lsblk -no UUID /dev/sdx1`\t/\text4\tdefaults,noatime\t0\t1" >/etc/fstab

Clean Up and Reboot

  • Your clone is complete, but lets clean up a bit.

exit
umount /mnt/boot/efi /mnt/dev/pts /mnt/dev /mnt/sys /mnt/proc /mnt

  • When you want to test your clone (and you do!), you’ll have to abandon the comfort, safety, and copy/paste convenience of your remote shell and work from the console, as you have to modify boot parameters
  • During the boot process, press del or esc or F11 or F12 or whatever key allows you to change the BIOS or EFI boot device, and change it to the newly created disk

Expert Mode

  • Here’s the whole cloning recipe without the commentary. Remember that it won’t work (and can be dangerous!) without editing, but hardcore folks can put this in a window next to their terminal for copypaste profit.
sudo -s
fdisk -l
fdisk /dev/sdx ### fixup
mkfs.fat -F32 /dev/sdx1 ### fixup
mkfs.ext4 /dev/sdx1 ### fixup
mkswap /dev/sdx3 ### fixup
mount -t ext4 /dev/sdx2 /mnt ### fixup
mkdir /mnt/dev 
mkdir /mnt/dev/pts 
mkdir /mnt/sys 
mkdir /mnt/proc
rsync --archive --verbose --delete --exclude=/dev --exclude=/sys --exclude=/proc --exclude=/mnt -xx / /mnt
mount --bind /dev /mnt/dev 
mount --bind /dev/pts /mnt/dev/pts 
mount --bind /sys /mnt/sys 
mount --bind /proc /mnt/proc 
chroot /mnt
mkdir /boot/efi
mount /dev/sdx1 /boot/efi/ ### fixup
grub-install /dev/sdx ### fixup
grub-install --recheck /dev/sdx ### fixup
update-grub 
mv /etc/fstab /etc/fstab.bak
echo -e "UUID=`lsblk -no UUID /dev/sdx1`\t/\text4\tdefaults,noatime\t0\t1" >/etc/fstab ### fixup required
exit 
umount /mnt/boot/efi /mnt/dev/pts /mnt/dev /mnt/sys /mnt/proc /mnt

After Words

Remember that the “disaster” in “disaster recovery” can mean many things, from acts of nature to government seizures to theft or sabotage or facility breakdown. I keep two bootable clones of each machine and, each week, I swap them between two geographically-separate sites. That way, I always have a bootable clone that’s less than two weeks old even if one location is completely lost. Of course, that’s in addition to much-more-frequent rsync backups between sites and archiving cloud backups. Back up in depth — it’s good for your health.

No, I don’t actually juggle 200+ clones. Many of my boxen are very small (think whiteboxed routers). I use a small stack of 2TB portable USB3 drives and create rsync images in separate directories under root. If I need a clone, I can either copy the subdirectory to root on a fresh drive, or just move the subdirectory up to root. I then install GRUB2 and go.

Unlike the Apple use case above, using clones to run from different hardware isn’t quite as straightforward. Since the hardware doesn’t come from a corporate monoculture, more differences exist than in the Macintosh ecosystem. Surprisingly, this hasn’t been as big an issue as I expected, in spite of the fact that my infrastructure is built from fire-sale, castoff, and various other forms of junk computers ranging from Mac minis to repurposed doorstops. It is sometimes necessary to tweak /etc/fstab or to change networking parameters to switch from eth0 to eth1 or the like.

In order to keep things as simple as possible, I have not enabled swap (though we did reserve a swap partition). Linux systems tend to degrade more gracefully when swap is enabled. The ArchWiki has a good discussion on how to enable swap.

Beyond the Storm

Rough waters, stormy skies, I saw her leave the gale,
Sails white against a copper sunset sky.
Alone in comical dress against the storm, I followed for a time;
She flagged, but did not come about,
Leaving just that glimpse, feeling oddly much, knowing nought.
So vast a sea, such encounters ought not repeat, yet I did not forget
For half a thousand days and oft, when storm clouds
Roiled in approaching skies, I would see her sails in the distance.
Following after, I might escape the storm,
But following from so far kept her always safely out of reach.
Then on a day becalmed, I drifted without care into a harbor
Her sails unexpected, bright, no wind in sight,
We sat at anchor drifting here and there, closer, farther
With the tide; I hailed her at last.
The breeze returned and, with unaccustomed finality, she sailed.
Then came the storm, so I understood why she had come
But why, then, had she left?

Ron Risley – 2014-03-24