The bring-up (of tol-eressea v2)

Russell Bateman
October 2018
last update:

We slowly and ever so carefully bring tol-eressea's big disks back up because the last thing I can possibly endure would be to lose media and have to retranscode thousands of programs on hundreds of disks.

This started by me taking the new server down, which has been up for a week now, but only with its system disk on line, then plugging them back in. Here's what they look like in the cabinet. You're looking, here, at a schematic of the side of the stack of drives just as if you had the side off the case and were looking at them. (There's already a picture of this in the notes on setting up host tol-eressëa

Cable    Connector                             +----------------+
color    marking                               | CD/DVD drive   |
-----    ---------                             +----------------+
                    +-------------------------------------------+
black               | 1Tb Western Digital —system drive         |  (replaces crashed disk)
                    +-------------------------------------------+
                    +-------------------------------------------+
black    T          | 2Tb Western Digital —movie drive          |
                    +-------------------------------------------+
                    +-------------------------------------------+
black    B          | 2Tb Western Digital —movie drive          | (ZFS mirror of preceeding)
                    +-------------------------------------------+
                    +-------------------------------------------+
red      A          | 4Tb Seagate         —television drive     |
                    +-------------------------------------------+
                    +-------------------------------------------+
red      B          | 4Tb Seagate         —television drive     | (ZFS mirror of preceeding)
                    +-------------------------------------------+

First, get all the block I/O devices.

[email protected]:~# lsblk -io KNAME,TYPE,SIZE,MODEL
KNAME TYPE   SIZE MODEL
loop0 loop  86.9M
loop1 loop  87.9M
loop2 loop  87.9M
sda   disk   1.8T WDC WD20EARX-00P
sda1  part   1.8T
sda9  part     8M
sdb   disk   1.8T WDC WD20EARX-00P
sdb1  part   1.8T
sdb9  part     8M
sdc   disk 931.5G WDC WD10EZEX-08W
sdc1  part     1M
sdc2  part 931.5G
sdd   disk   3.7T ST4000DM000-1F21
sdd1  part   3.7T
sdd9  part     8M
sde   disk   3.7T ST4000DM000-1F21
sde1  part   3.7T
sde9  part     8M
sr0   rom   1024M DRW-24B1ST   c

Looks like our disks are in there; let's isolate them:

[email protected]:~# lsblk -io KNAME,TYPE,SIZE,MODEL | grep disk
sda   disk   1.8T WDC WD20EARX-00P
sdb   disk   1.8T WDC WD20EARX-00P
sdc   disk 931.5G WDC WD10EZEX-08W
sdd   disk   3.7T ST4000DM000-1F21
sde   disk   3.7T ST4000DM000-1F21

Now let's make a table of them so we can match them up:

Device Size Make and model Function
sda 1.8T WDC WD20EARX-00P Movie disk
sdb 1.8T WDC WD20EARX-00P Movie disk
sdc 931.5G WDC WD10EZEX-08W System disk
sdd 3.7T ST4000DM000-1F21 Television disk
sde 3.7T ST4000DM000-1F21 Television disk

The next trick is to correlate them with ZFS. Somehow, we must tell ZFS that these disks are already mirrored. Does ZFS keep some metadata on the individual disks or did we just lose all of that went the system went down? Let's copy the table and correlate the old device names with the new ones. We'll lose some of the fields too. To do this, we're consulting the original layout of disks for tol-eressea.

New device Old device Size Make Function
sda sda 1.8T WD Movie disk
sdb sdb 1.8T WD Movie disk
sdd sdd 3.7T Seagate Television disk
sde sde 3.7T Seagate Television disk
sdc sdc 931.5G WD System disk

It looks like they're all the same still. This is because I was careful about maintaining the exact physical order in the case—down to (and most importantly) the exact SATA connector being used with the exact (same) disk.

Restoring ZFS structure...

In summary, we have...

  1. ...a new system disk with no knowledge of ZFS except that we've installed the Ubuntu ZFS packages;
  2. ...2 × two pairs of disks originally set up to mirror each other as shown above (and also here):
    • 2 × Western Digital 2Tb (nominal) drives for Plex Media movies
    • 2 × Seagate 4Tb (nominal) drives for Plex Media television series and other, miscellaneous video media

The challenge is to know how to reacquaint the old drives, which we hope are intact, with the new Ubuntu 18.04.1 server operating systems. We don't know if ZFS, newly installed as mere software utilities on the new system disk, has any knowledge or understanding of the old disks. We don't know what ZFS metadata might be held on those original disks. We're blind at this point because we have no prior experience remounting a ZFS system especially under these circumstances.

We google for "zfs new system disk how to remount pool drives." Following this answer to How to inspect drive for ZFS, we are led to try this:

[email protected]:~# zpool import
   pool: television
     id: 3759371443983060498
  state: ONLINE
 status: The pool was last accessed by another system.
 action: The pool can be imported using its name or numeric identifier and the '-f' flag.
    see: http://zfsonlinux.org/msg/ZFS-8000-EY
 config:

	television                           ONLINE
	  mirror-0                           ONLINE
	    ata-ST4000DM000-1F2168_S300MZ7G  ONLINE
	    ata-ST4000DM000-1F2168_S300ELBZ  ONLINE

   pool: movies
     id: 17302323605392653091
  state: ONLINE
 status: The pool was last accessed by another system.
 action: The pool can be imported using its name or numeric identifier and the '-f' flag.
    see: http://zfsonlinux.org/msg/ZFS-8000-EY
 config:

	movies                                        ONLINE
	  mirror-0                                    ONLINE
	    ata-WDC_WD20EARX-00PASB0_WD-WCAZAJ004702  ONLINE
	    ata-WDC_WD20EARX-00PASB0_WD-WCAZAJ069805  ONLINE

From this we learn a) that ZFS is able to reach out to other disks on our sysem to identify exiting pools and b) that both disks in both pools appear healthy enough to be listed in their respective pools (this is indeed good news; last c) that we can use zpool import -f pool-name to reimport each pool formally. The use of option -f is suggested by ZFS itself, however, the responder in answering the question suggested trying the import command without it. Seeing this as a harmless undertaking, we'll try that first:

[email protected]:~# zpool import movies
cannot import 'movies': pool was previously in use from another system.
Last accessed by tol-eressea (hostid=7f0101) at Sun Sep 23 05:22:49 2018
The pool can be imported, use 'zpool import -f' to import the pool.

Fair enough. We'll do what ZFS itself said to do. If that appears to work on the movies pool, we'll do it for the television series pool:

[email protected]:~# zpool import -f movies
[email protected]:~# zpool list
NAME     SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
movies  1.81T   930G   926G         -      -    50%  1.00x  ONLINE  -
[email protected]:~# zpool import -f television
[email protected]:~# zpool list
NAME         SIZE  ALLOC   FREE  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
movies      1.81T   930G   926G         -      -    50%  1.00x  ONLINE  -
television  3.62T  1.50T  2.12T         -      -    41%  1.00x  ONLINE  -

I'm looking at the procedure followed in my original layout of my Plex Media server disks. At this point, I'm guessing that we only need to reset the ZFS mount points except for two things:

  1. there is no longer any command, zfs set mountpoint=/plex-movies movies,
  2. I observe that these mount-points I had to establish on my old system appear already to be in place:
    [email protected]:/plex-television$ ll /plex-movies
    total 234
    drwxr-xr-x  13 russ russ   13 Nov 24  2017  .
    drwxr-xr-x  25 root root 4096 Oct  6 17:40  ..
    drwxrwxr-x  30 russ russ   30 Jan  9  2018  Andrea
    drwxrwxr-x  46 russ russ   54 Jan  5  2018  Christmas
    drwxrwxr-x  18 russ russ   24 Jul 23 23:39  Documentaries
    drwxrwxr-x   2 russ russ   24 Jul 26 16:09  Foreign
    drwxrwxr-x  35 russ russ   73 Sep 11 02:25 'Mormon Cinema' *
    drwxr-xr-x 441 russ russ  442 Sep 19 01:51  Movies
    drwxrwxr-x  27 russ russ   27 Aug 31 01:54 'Musical Performances'
    drwxrwxr-x   2 russ russ   28 Sep 15  2015  Pagnol
    drwxrwxr-x   3 russ russ    8 Jul 22 03:08  saved
    drwxrwxr-x   2 russ russ   58 Aug  3 23:12 'Science Fiction'
    drwxrwxr-x   2 russ russ   33 Jun 16 05:32 'War Movies'
    [email protected]:/plex-television$ ll /plex-television
    total 203
    drwxr-xr-x  14 russ russ   14 Sep  1 19:26  .
    drwxr-xr-x  25 root root 4096 Oct  6 17:40  ..
    drwxrwxr-x   2 russ russ  193 Jul  2  2014 'Good Eats'
    drwxrwxr-x   2 russ russ   12 Apr 11  2017  Greg
    drwxrwxr-x   4 russ russ    6 May 12 20:39  Guitar
    drwxrwxr-x  23 russ russ   25 May 25 15:17  Miscellaneous
    drwxrwxr-x  25 russ russ   25 Jul 28  2017 'Mormon TV'
    drwxrwxr-x 110 russ russ  110 Feb 28  2016  MP3-Music
    drwxrwxr-x   3 russ russ    3 Sep  1 19:27  Music
    drwxrwxr-x  10 russ russ   10 Sep  5 15:11 'Musical TV'
    drwxrwxr-x   2 russ russ    2 Sep  1 19:27  music.old
    drwxrwxr-x   2 russ russ    2 Jun  1  2016  saved
    drwxrwxr-x  45 russ russ   46 Sep  1 04:36  TV
    drwxrwxr-x   5 russ russ    5 Jul 21 17:52  Vintage
    

* With Ubuntu 18.04.1, or maybe in Ubuntu 17—I don't know because I've never used that version, directory listings of files and subdirectories with spaces in their names are now single quote-delimited (which I find to be annoying and useless, but oh well...).

I believe that, in terms of "bring-up," there's nothing left to do in ZFS. The whole question appears much easier than I ever imagined.

[email protected]:~$ diskspace.sh
Filesystem on disk: /dev (device: udev)
   Total disk size: 7.7Gb
        Used space: 0b (0%)
        Free space: 7.7Gb
Filesystem on disk: /run (device: tmpfs)
   Total disk size: 1.6Gb
        Used space: 1.3Mb (1%)
        Free space: 1.6Gb
Filesystem on disk: / (device: /dev/sdc2)
   Total disk size: 916Gb
        Used space: 12Gb (2%)
        Free space: 859Gb
Filesystem on disk: /run (device: tmpfs)
   Total disk size: 1.6Gb
        Used space: 1.3Mb (1%)
        Free space: 1.6Gb
Filesystem on disk: /run (device: tmpfs)
   Total disk size: 1.6Gb
        Used space: 1.3Mb (1%)
        Free space: 1.6Gb
Filesystem on disk: /run (device: tmpfs)
   Total disk size: 1.6Gb
        Used space: 1.3Mb (1%)
        Free space: 1.6Gb
Filesystem on disk: /snap/core/4917 (device: /dev/loop0)
   Total disk size: 87Mb
        Used space: 87Mb (100%)
        Free space: 0b
Filesystem on disk: /snap/core/5328 (device: /dev/loop1)
   Total disk size: 88Mb
        Used space: 88Mb (100%)
        Free space: 0b
Filesystem on disk: /snap/core/5548 (device: /dev/loop2)
   Total disk size: 88Mb
        Used space: 88Mb (100%)
        Free space: 0b
Filesystem on disk: /run (device: tmpfs)
   Total disk size: 1.6Gb
        Used space: 1.3Mb (1%)
        Free space: 1.6Gb
Filesystem on disk: /plex-movies (device: movies)
   Total disk size: 1.8Tb
        Used space: 931Gb (52%)
        Free space: 868Gb
Filesystem on disk: /plex-television (device: television)
   Total disk size: 3.6Tb
        Used space: 1.6Tb (43%)
        Free space: 2.1Tb

A full listing of diskspace shows me that things are back to how I remember them before my system crashed.

Bringing the disks' content back to Plex Media Server...

...