Russell Bateman
October 2018
last update:
We slowly and ever so carefully bring tol-eressea's big disks back up because the last thing I can possibly endure would be to lose media and have to retranscode thousands of programs on hundreds of disks.
This started by me taking the new server down, which has been up for a week now, but only with its system disk on line, then plugging them back in. Here's what they look like in the cabinet. You're looking, here, at a schematic of the side of the stack of drives just as if you had the side off the case and were looking at them. (There's already a picture of this in the notes on setting up host tol-eressëa
Cable Connector +----------------+ color marking | CD/DVD drive | ----- --------- +----------------+ +-------------------------------------------+ black | 1Tb Western Digital —system drive | (replaces crashed disk) +-------------------------------------------+ +-------------------------------------------+ black T | 2Tb Western Digital —movie drive | +-------------------------------------------+ +-------------------------------------------+ black B | 2Tb Western Digital —movie drive | (ZFS mirror of preceeding) +-------------------------------------------+ +-------------------------------------------+ red A | 4Tb Seagate —television drive | +-------------------------------------------+ +-------------------------------------------+ red B | 4Tb Seagate —television drive | (ZFS mirror of preceeding) +-------------------------------------------+
First, get all the block I/O devices.
root@tol-eressea:~# lsblk -io KNAME,TYPE,SIZE,MODEL
KNAME TYPE SIZE MODEL
loop0 loop 86.9M
loop1 loop 87.9M
loop2 loop 87.9M
sda disk 1.8T WDC WD20EARX-00P
sda1 part 1.8T
sda9 part 8M
sdb disk 1.8T WDC WD20EARX-00P
sdb1 part 1.8T
sdb9 part 8M
sdc disk 931.5G WDC WD10EZEX-08W
sdc1 part 1M
sdc2 part 931.5G
sdd disk 3.7T ST4000DM000-1F21
sdd1 part 3.7T
sdd9 part 8M
sde disk 3.7T ST4000DM000-1F21
sde1 part 3.7T
sde9 part 8M
sr0 rom 1024M DRW-24B1ST c
Looks like our disks are in there; let's isolate them:
root@tol-eressea:~# lsblk -io KNAME,TYPE,SIZE,MODEL | grep disk
sda disk 1.8T WDC WD20EARX-00P
sdb disk 1.8T WDC WD20EARX-00P
sdc disk 931.5G WDC WD10EZEX-08W
sdd disk 3.7T ST4000DM000-1F21
sde disk 3.7T ST4000DM000-1F21
Now let's make a table of them so we can match them up:
| Device | Size | Make and model | Function |
|---|---|---|---|
| sda | 1.8T | WDC WD20EARX-00P | Movie disk |
| sdb | 1.8T | WDC WD20EARX-00P | Movie disk |
| sdc | 931.5G | WDC WD10EZEX-08W | System disk |
| sdd | 3.7T | ST4000DM000-1F21 | Television disk |
| sde | 3.7T | ST4000DM000-1F21 | Television disk |
The next trick is to correlate them with ZFS. Somehow, we must tell ZFS that these disks are already mirrored. Does ZFS keep some metadata on the individual disks or did we just lose all of that went the system went down? Let's copy the table and correlate the old device names with the new ones. We'll lose some of the fields too. To do this, we're consulting the original layout of disks for tol-eressea.
| New device | Old device | Size | Make | Function |
|---|---|---|---|---|
| sda | sda | 1.8T | WD | Movie disk |
| sdb | sdb | 1.8T | WD | Movie disk |
| sdd | sdd | 3.7T | Seagate | Television disk |
| sde | sde | 3.7T | Seagate | Television disk |
| sdc | sdc | 931.5G | WD | System disk |
It looks like they're all the same still. This is because I was careful about maintaining the exact physical order in the case—down to (and most importantly) the exact SATA connector being used with the exact (same) disk.
In summary, we have...
The challenge is to know how to reacquaint the old drives, which we hope are intact, with the new Ubuntu 18.04.1 server operating systems. We don't know if ZFS, newly installed as mere software utilities on the new system disk, has any knowledge or understanding of the old disks. We don't know what ZFS metadata might be held on those original disks. We're blind at this point because we have no prior experience remounting a ZFS system especially under these circumstances.
We google for "zfs new system disk how to remount pool drives." Following this answer to How to inspect drive for ZFS, we are led to try this:
root@tol-eressea:~# zpool import pool: television id: 3759371443983060498 state: ONLINE status: The pool was last accessed by another system. action: The pool can be imported using its name or numeric identifier and the '-f' flag. see: http://zfsonlinux.org/msg/ZFS-8000-EY config: television ONLINE mirror-0 ONLINE ata-ST4000DM000-1F2168_S300MZ7G ONLINE ata-ST4000DM000-1F2168_S300ELBZ ONLINE pool: movies id: 17302323605392653091 state: ONLINE status: The pool was last accessed by another system. action: The pool can be imported using its name or numeric identifier and the '-f' flag. see: http://zfsonlinux.org/msg/ZFS-8000-EY config: movies ONLINE mirror-0 ONLINE ata-WDC_WD20EARX-00PASB0_WD-WCAZAJ004702 ONLINE ata-WDC_WD20EARX-00PASB0_WD-WCAZAJ069805 ONLINE
From this we learn a) that ZFS is able to reach out to other disks on our sysem to identify exiting pools and b) that both disks in both pools appear healthy enough to be listed in their respective pools (this is indeed good news; last c) that we can use zpool import -f pool-name to reimport each pool formally. The use of option -f is suggested by ZFS itself, however, the responder in answering the question suggested trying the import command without it. Seeing this as a harmless undertaking, we'll try that first:
root@tol-eressea:~# zpool import movies
cannot import 'movies': pool was previously in use from another system.
Last accessed by tol-eressea (hostid=7f0101) at Sun Sep 23 05:22:49 2018
The pool can be imported, use 'zpool import -f' to import the pool.
Fair enough. We'll do what ZFS itself said to do. If that appears to work on the movies pool, we'll do it for the television series pool:
root@tol-eressea:~# zpool import -f movies root@tol-eressea:~# zpool list NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT movies 1.81T 930G 926G - - 50% 1.00x ONLINE - root@tol-eressea:~# zpool import -f television root@tol-eressea:~# zpool list NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT movies 1.81T 930G 926G - - 50% 1.00x ONLINE - television 3.62T 1.50T 2.12T - - 41% 1.00x ONLINE -I'm looking at the procedure followed in my original layout of my Plex Media server disks. At this point, I'm guessing that we only need to reset the ZFS mount points except for two things:
russ@tol-eressea:/plex-television$ ll /plex-movies total 234 drwxr-xr-x 13 russ russ 13 Nov 24 2017 . drwxr-xr-x 25 root root 4096 Oct 6 17:40 .. drwxrwxr-x 30 russ russ 30 Jan 9 2018 Andrea drwxrwxr-x 46 russ russ 54 Jan 5 2018 Christmas drwxrwxr-x 18 russ russ 24 Jul 23 23:39 Documentaries drwxrwxr-x 2 russ russ 24 Jul 26 16:09 Foreign drwxrwxr-x 35 russ russ 73 Sep 11 02:25 'Mormon Cinema' * drwxr-xr-x 441 russ russ 442 Sep 19 01:51 Movies drwxrwxr-x 27 russ russ 27 Aug 31 01:54 'Musical Performances' drwxrwxr-x 2 russ russ 28 Sep 15 2015 Pagnol drwxrwxr-x 3 russ russ 8 Jul 22 03:08 saved drwxrwxr-x 2 russ russ 58 Aug 3 23:12 'Science Fiction' drwxrwxr-x 2 russ russ 33 Jun 16 05:32 'War Movies' russ@tol-eressea:/plex-television$ ll /plex-television total 203 drwxr-xr-x 14 russ russ 14 Sep 1 19:26 . drwxr-xr-x 25 root root 4096 Oct 6 17:40 .. drwxrwxr-x 2 russ russ 193 Jul 2 2014 'Good Eats' drwxrwxr-x 2 russ russ 12 Apr 11 2017 Greg drwxrwxr-x 4 russ russ 6 May 12 20:39 Guitar drwxrwxr-x 23 russ russ 25 May 25 15:17 Miscellaneous drwxrwxr-x 25 russ russ 25 Jul 28 2017 'Mormon TV' drwxrwxr-x 110 russ russ 110 Feb 28 2016 MP3-Music drwxrwxr-x 3 russ russ 3 Sep 1 19:27 Music drwxrwxr-x 10 russ russ 10 Sep 5 15:11 'Musical TV' drwxrwxr-x 2 russ russ 2 Sep 1 19:27 music.old drwxrwxr-x 2 russ russ 2 Jun 1 2016 saved drwxrwxr-x 45 russ russ 46 Sep 1 04:36 TV drwxrwxr-x 5 russ russ 5 Jul 21 17:52 Vintage
* With Ubuntu 18.04.1, or maybe in Ubuntu 17—I don't know because I've never used that version, directory listings of files and subdirectories with spaces in their names are now single quote-delimited (which I find to be annoying and useless, but oh well...).
I believe that, in terms of "bring-up," there's nothing left to do in ZFS. The whole question appears much easier than I ever imagined.
russ@tol-eressea:~$ diskspace.sh Filesystem on disk: /dev (device: udev) Total disk size: 7.7Gb Used space: 0b (0%) Free space: 7.7Gb Filesystem on disk: /run (device: tmpfs) Total disk size: 1.6Gb Used space: 1.3Mb (1%) Free space: 1.6Gb Filesystem on disk: / (device: /dev/sdc2) Total disk size: 916Gb Used space: 12Gb (2%) Free space: 859Gb Filesystem on disk: /run (device: tmpfs) Total disk size: 1.6Gb Used space: 1.3Mb (1%) Free space: 1.6Gb Filesystem on disk: /run (device: tmpfs) Total disk size: 1.6Gb Used space: 1.3Mb (1%) Free space: 1.6Gb Filesystem on disk: /run (device: tmpfs) Total disk size: 1.6Gb Used space: 1.3Mb (1%) Free space: 1.6Gb Filesystem on disk: /snap/core/4917 (device: /dev/loop0) Total disk size: 87Mb Used space: 87Mb (100%) Free space: 0b Filesystem on disk: /snap/core/5328 (device: /dev/loop1) Total disk size: 88Mb Used space: 88Mb (100%) Free space: 0b Filesystem on disk: /snap/core/5548 (device: /dev/loop2) Total disk size: 88Mb Used space: 88Mb (100%) Free space: 0b Filesystem on disk: /run (device: tmpfs) Total disk size: 1.6Gb Used space: 1.3Mb (1%) Free space: 1.6Gb Filesystem on disk: /plex-movies (device: movies) Total disk size: 1.8Tb Used space: 931Gb (52%) Free space: 868Gb Filesystem on disk: /plex-television (device: television) Total disk size: 3.6Tb Used space: 1.6Tb (43%) Free space: 2.1Tb
A full listing of diskspace shows me that things are back to how I remember them before my system crashed.
...