Rockstable Wiki:

Clone

A running system shall be cloned from on disk to a fresh and different filesystem (in my case xfs to btrfs). dump works only for ext-filesystems (and restore versions maybe incompatible across the systems). I chose rsync, because it supports:

Live CD

You'll probably need a live-cd like the ones from https://grml.org.

GRML Testing Latest

Simply diskdump it to an usb stick.

   1 ll /dev/disk/by-path/*usb*
   2 sudo dd if=grml64-full_testing_latest.iso of=/dev/sdc

Try it, before you need it!

Remote control

I recommend controlling the live environment over ssh and tmux. So you can use the big monitor. :-)

On the live system

   1 passwd grml
   2 sudo -s
   3 systemctl start ssh.service
   4 tmux

On the remote controller

   1 ssh grml@target
   2 sudo -s
   3 tmux attach

Partition

parted /dev/vdb

   1 GNU Parted 3.2                                                     
   2 Using /dev/vdb                                                     
   3 Welcome to GNU Parted! Type 'help' to view a list of commands.     
   4 (parted) unit MiB                                                  
   5 (parted) mktable gpt
   6 (parted) mkpart bios_grub 1 2
   7 (parted) mkpart EFI 2 514
   8 (parted) mkpart swap 514 2562
   9 (parted) mkpart boot 2562 3074
  10 (parted) mkpart boot 3074 -1
  11 (parted) set 1 bios_grub on 
  12 (parted) print free 
  13 Model: Virtio Block Device (virtblk)
  14 Disk /dev/vdb: 30720MiB
  15 Sector size (logical/physical): 512B/512B
  16 Partition Table: gpt
  17 Disk Flags:
  18 
  19 Number  Start     End       Size      File system  Name       Flags
  20         0,02MiB   1,00MiB   0,98MiB   Free Space
  21  1      1,00MiB   2,00MiB   1,00MiB                bios_grub  bios_grub
  22  2      2,00MiB   514MiB    512MiB                 EFI
  23  3      514MiB    2562MiB   2048MiB                swap1
  24  4      2562MiB   3074MiB   512MiB                 boot
  25  5      3074MiB   30719MiB  27645MiB               root
  26         30719MiB  30720MiB  0,98MiB   Free Space
  27 
  28 (parted)
  29 (parted) quit
  30 Information: You may need to update /etc/fstab.

Format

   1 root@template-stretch / # mkswap -L swap1 /dev/vdb3
   2 Setting up swapspace version 1, size = 2 GiB (2147479552 bytes)
   3 LABEL=swap1, UUID=1e82e970-7a9d-4b47-b959-b611834c3c37
   4 mkfs.btrfs -L boot /dev/vdb4
   5 mkfs.btrfs -L root /dev/vdb5

Mount

   1 mount /dev/vdb5 /mnt
   2 mkdir /mnt/boot
   3 mount /dev/vdb4 /mnt/boot

Copy

rsync

Used rsync options:

   1 rsync -aAxXhHPS / /mnt
   2 
   3 
   4 
   5 sent 3.91G bytes  received 1.30M bytes  10.31M bytes/sec
   6 total size is 4.01G  speedup is 1.02
   7 rsync -avAxXhH / /mnt  18,46s user 20,12s system 10% cpu 6:18,46 total

ssh

When your copy is limited by Gigabit line-speed use the parallel implementation of gzip pigz to compress the stream before it passes and saturates the network. It's terribly fast compared to its single threaded companions embedded in ssh or rsync. The more CPU-time you have the faster the compression of the stream. As long as you have enough CPU time, network may not be saturated to line speed and disk-IO will become the limiting factor.

Transfer whole disk to other host as a image via ssh to target host.

   1 DEVICE=/dev/sda
   2 dd if=$DEVICE \
   3         |pv -s "$(blockdev --getsize64 "$DEVICE")" \
   4         |pigz -c \
   5         |ssh user@target.host \
   6                 "cat - > \"rel/path/to/$HOSTNAME-$(basename \"$DEVICE\").img\""
   7 
   8 ### EXAMPLE OUTPUT WITH 8CORES (4COREs + SMT) -> LAPTOP-IMAGE
   9  476GiB 0:53:53 [ 151MiB/s] [==================================================================>] 100%            
  10 dd if="$DEVICE"  224.75s user 1358.92s system 48% cpu 53:53.38 total
  11 pv -s "$(blockdev --getsize64 "$DEVICE")"  43.63s user 1171.22s system 37% cpu 53:53.38 total
  12 pigz -c  11589.45s user 566.54s system 375% cpu 53:53.38 total
  13 ssh tobias@libertas "cat > \"$HOSTNAME-$(basename $DEVICE).img\""  304.02s user 281.79s system 18% cpu 53:53.43 total
  14 
  15 ### 512110190592/(53*60+53)/2**20 = 151.06 MiB/s
  16 ### IS FASTER THAN THE LINE-SPEED OF THE UNDERLYING 1GBASE-T
  17 ### AND ITS ALREADY COMPRESSED ON THE REMOTE SITE (56GiB) :-)
  18 
  19 ### WITH ELEVATED PRIVILEGES ON REMOTE SYSTEM
  20 DEVICE=/dev/sda
  21 dd if=$DEVICE \
  22         |pv -s "$(blockdev --getsize64 "$DEVICE")" \
  23         |pigz -c \
  24         |ssh -t user@target.host \
  25                 "sudo bash -c 'cat - > \"rel/path/to/$HOSTNAME-$(basename \"$DEVICE\").img\"'"

Tar a whole filesystem to a compressed archive on a remote host.

   1 MOUNT_POINT="/mnt"
   2 tar -cf - "$MOUNT_POINT" \
   3         |pv -s "$(du -bsx "$MOUNT_POINT"|cut -f1)" \
   4         |pigz -c \
   5         |ssh user@target.host \
   6                 "bash -c 'cat - > \"rel/path/to/$HOSTNAME-${MOUNT_POINT//\//_}.tar.gz\"'"

Bootloader

Chroot into new filesystem

   1 root@template-stretch /tmp # for DIR in dev dev/pts proc sys;do
   2         mount --bind "/$DIR" "/mnt/$DIR";
   3 done
   4 root@template-stretch /tmp # chroot /mnt 
   5 

Install bootloader (still no EFI)

   1 root@template-stretch / # grub-install /dev/vdb
   2 i386-pc wird für Ihre Plattform installiert.
   3 device node not found
   4 
   5 device node not found
   6 installation beendet. Keine Fehler aufgetreten.

Configure bootloader

   1 update-grub2

Append current information in a safe fashion to /etc/fstab

   1 blkid | sed -r 's/^/# /' >> /etc/fstab

Alter file system table /etc/fstab

   1 # /etc/fstab: static file system information.
   2 #
   3 # Use 'blkid' to print the universally unique identifier for a
   4 # device; this may be used with UUID= as a more robust way to name devices
   5 # that works even if disks are added and removed. See fstab(5).
   6 #
   7 #<file_system>                             <mount_point>  <type>  <options>                                <dump>  <pass>
   8 proc                                       /proc          proc    defaults                                 0       0
   9 UUID=d597c3bb-6c6c-4c4d-9b83-2442c3bf72fd  /boot          btrfs   defaults                                 0       2
  10 UUID=8d1f76ca-a06f-49de-8ac4-283ade0da7a5  /              btrfs   rw,noatime,compress=lzo,ssd,space_cache  0       1
  11 UUID=1e82e970-7a9d-4b47-b959-b611834c3c37  none           swap    sw                                       0       0
  12 
  13 # /dev/vda1: LABEL="bootfs" UUID="70d3b316-7f05-48d9-9498-8144b0cb4473" SEC_TYPE="ext2" TYPE="ext3"
  14 # /dev/vda2: LABEL="rootfs" UUID="5ef7d751-2911-4005-b97b-0ca9945ecfc4" TYPE="xfs"
  15 # /dev/vda5: UUID="7c19aebf-abcc-4c93-a7af-f5331aeb650c" TYPE="swap"
  16 # /dev/vdb4: LABEL="boot" UUID="d597c3bb-6c6c-4c4d-9b83-2442c3bf72fd" UUID_SUB="0413709e-d471-432c-8f94-64b99421d841" TYPE="btrfs"
  17 # /dev/vdb5: LABEL="root" UUID="8d1f76ca-a06f-49de-8ac4-283ade0da7a5" UUID_SUB="9f8a3778-b726-4d2c-beb1-84fd3267e53f" TYPE="btrfs"
  18 # /dev/vdb3: LABEL="swap1" UUID="1e82e970-7a9d-4b47-b959-b611834c3c37" TYPE="swap"
  19 

Confiure initrd

Fix initrd resume to swap /etc/initramfs-tools/conf.d/resume

   1 RESUME=UUID=1e82e970-7a9d-4b47-b959-b611834c3c37

Rebuild initrd

   1 root@template-stretch / # update-initramfs -k all -u
   2 update-initramfs: Generating /boot/initrd.img-4.8.0-1-amd64
   3 cp: der Aufruf von stat für '/etc/modprobe.d/*' ist nicht möglich: Datei oder Verzeichnis nicht gefunden
   4 /boot/initrd.img-4.7.0-1-amd64 does not exist. Cannot update.
   5 /boot/initrd.img-3.16.0-4-amd64 does not exist. Cannot update.
   6 /boot/initrd.img-3.14-2-amd64 does not exist. Cannot update.
   7 /boot/initrd.img-3.2.0-4-amd64 does not exist. Cannot update.
   8 /boot/initrd.img-3.2.0-1-amd64 does not exist. Cannot update.

Reboot

   1 shutdown -r now

Transform to EFI

EFI System partition (ESP)

Install FAT toolsuite

   1 aptitude install dosfstools

Format EFI system partition (ESP) with FAT12/16/32

   1 root@template-stretch /boot # mkfs.vfat -F32 -n EFI /dev/vda2
   2 mkfs.fat 4.0 (2016-05-06)

   1 root@template-stretch /boot # blkid | sed -r 's/^/# /' >> /etc/fstab
   2 root@template-stretch /boot # vi /etc/fstab
   3 

/etc/fstab

   1 # /etc/fstab: static file system information.
   2 #
   3 # Use 'blkid' to print the universally unique identifier for a
   4 # device; this may be used with UUID= as a more robust way to name devices
   5 # that works even if disks are added and removed. See fstab(5).
   6 #
   7 #<file_system>                             <mount_point>  <type>  <options>                                <dump>  <pass>
   8 proc                                       /proc          proc    defaults                                 0       0
   9 UUID=3452-16B9                             /boot/EFI      vfat    defaults                                 0       2
  10 UUID=d597c3bb-6c6c-4c4d-9b83-2442c3bf72fd  /boot          btrfs   defaults                                 0       2
  11 UUID=8d1f76ca-a06f-49de-8ac4-283ade0da7a5  /              btrfs   rw,noatime,compress=lzo,ssd,space_cache  0       1
  12 UUID=1e82e970-7a9d-4b47-b959-b611834c3c37  none           swap    sw                                       0       0
  13 
  14 # /dev/vda3: LABEL="swap1" UUID="1e82e970-7a9d-4b47-b959-b611834c3c37" TYPE="swap" PARTLABEL="swap1" PARTUUID="72314aa4-000a-4150-9996-313a9114d777"
  15 # /dev/vda4: LABEL="boot" UUID="d597c3bb-6c6c-4c4d-9b83-2442c3bf72fd" UUID_SUB="0413709e-d471-432c-8f94-64b99421d841" TYPE="btrfs" PARTLABEL="boot" PARTUUID="de599caa-78aa-4747-a5bc-91a4f71b333c"
  16 # /dev/vda5: LABEL="root" UUID="8d1f76ca-a06f-49de-8ac4-283ade0da7a5" UUID_SUB="9f8a3778-b726-4d2c-beb1-84fd3267e53f" TYPE="btrfs" PARTLABEL="root" PARTUUID="645044b0-0228-4c2b-92c4-555b2e72d59a"
  17 # /dev/vda1: PARTLABEL="bios_grub" PARTUUID="ec0c7283-0c5f-42ee-883f-29cc92cc1f81"
  18 # /dev/vda2: LABEL="EFI" UUID="3452-16B9" TYPE="vfat" PARTLABEL="EFI" PARTUUID="742b1a54-ff3a-49e1-8bfb-85f92da1778a"
  19 

Mount the new filesystems

   1 root@template-stretch /boot # mount -a 
   2 

GRUB-EFI

Install grub-efi and remove grub-pc

   1 aptitude install grub-efi grub-pc-

Install bootloader and create EFI-image

   1 root@template-stretch /boot/EFI # grub-install \
   2         --efi-directory=/boot/EFI --recheck \
   3         --directory=/usr/lib/grub/x86_64-efi /dev/vda
   4 x86_64-efi wird für Ihre Plattform installiert.
   5 EFI variables are not supported on this system.
   6 EFI variables are not supported on this system.
   7 installation beendet. Keine Fehler aufgetreten.

Update grub configuration

   1 root@template-stretch /boot # update-grub2 
   2 GRUB-Konfigurationsdatei wird erstellt …
   3 Linux-Abbild gefunden: /boot/vmlinuz-4.8.0-1-amd64
   4 initrd-Abbild gefunden: /boot/initrd.img-4.8.0-1-amd64
   5 erledigt

Libvirt with EFI

Please refer to Libvirt with EFI

Disk Dump Rescue (ddrescue)

About

Material

When to stop

I personally stop rescuing the disk when and non-trimmed sectors are read. On a example of a DVD -> this already took 4 days … ddrescue then tries to scrape the non-scraped blocks, which is another magnitude slower then the last stage. Time since last successfull read explodes. For me its time to stop and get the data.

   1      ipos:   4206 MiB, non-trimmed:  62002 KiB,  current rate:      66 B/s
   2      opos:   4206 MiB, non-scraped: 185878 KiB,  average rate:     635 B/s
   3 non-tried:        0 B,  bad-sector:  15500 KiB,    error rate:      66 B/s
   4   rescued:   4042 MiB,   bad areas:     7732,        run time:  4d 31m 16s
   5 pct rescued:   94.01%, read errors:     9881,  remaining time:  1d 10h  7m
   6                               time since last successful read:          0s
   7 Trimming failed blocks... (forwards)
   8 
   9      ipos:   3721 MiB, non-trimmed:        0 B,  current rate:       0 B/s
  10      opos:   3721 MiB, non-scraped: 223874 KiB,  average rate:     507 B/s
  11 non-tried:        0 B,  bad-sector:  23148 KiB,    error rate:      68 B/s
  12   rescued:   4058 MiB,   bad areas:     8858,        run time:  5d  9h 58m
  13 pct rescued:   94.39%, read errors:    13705,  remaining time: 28d 16h 51m
  14                               time since last successful read:      1m 31s
  15 Scraping failed blocks... (forwards)

But if the data is of extraordinary importance you should probably carry on. If you keep the media, the image and the map file you can continue the rescue later.

ddrescue

Install GNU ddrescue

   1 aptitude install gddrescue

   1 ### FIRST RUN FORWARD WITHOUT SCRAPING,
   2 ### FOR FAST RESULTS
   3 ddrescue \
   4         -vvv -B -n \
   5         --max-read-rate=$((48*2**20)) \
   6         /dev/sdc1 \
   7         /media/space/disk_images/old1/disk_old1_p1.raw \
   8         /media/space/disk_images/old1/disk_old1_p1.map
   9 
  10 ### SECOND RUN FROM REVERSE WITH 3 RETRIES
  11 ddrescue \
  12         -vvv -B -R -r 3 \
  13         --max-read-rate=$((48*2**20)) \
  14         /dev/sdc1 \
  15         /media/space/disk_images/old1/disk_old1_p1.raw \
  16         /media/space/disk_images/old1/disk_old1_p1.map
  17 
  18 ### OPTICAL MEDIUM WITH SCRAPING (3 RETRIES)
  19 ddrescue \
  20         -vvv -B -r 3 \
  21         -b 2048 \
  22         /dev/sr0 \
  23         /media/space/disk_images/sr0.img \
  24         /media/space/disk_images/sr0.map

Used options:

example output

   1 GNU ddrescue 1.23
   2 About to copy 194474 MiBytes from '/dev/sdc1' [UNKNOWN] (203921109504) to '/media/space/disk_images/old1/disk_old1_p1.raw' (203921109504)
   3     Starting positions: infile = 0 B,  outfile = 0 B
   4     Copy block size: 128 sectors       Initial skip size: 4096 sectors
   5 Sector size: 512 Bytes
   6 Max read rate:  49152 KiB/s
   7 Direct in: no     Direct out: no     Sparse: no     Truncate: no
   8 Trim: yes         Scrape: yes        Max retry passes: 3
   9 Reverse mode
  10 
  11 Press Ctrl-C to interrupt
  12 Initial status (read from mapfile)
  13 current position:   1880 MiB,     current sector: 3850349
  14  last block size: 192580 MiB
  15 rescued: 194473 MiB, tried: 802304 B, bad-sector: 157184 B, bad areas: 105
  16 
  17 Current status
  18      ipos:   1880 MiB, non-trimmed:        0 B,  current rate:       0 B/s
  19      opos:   1880 MiB, non-scraped:   645120 B,  average rate:       0 B/s
  20 non-tried:        0 B,  bad-sector:   157184 B,    error rate:       0 B/s
  21   rescued: 194473 MiB,   bad areas:      105,        run time:          0s
  22 pct rescued:   99.99%, read errors:        0,  remaining time:         n/a
  23                               time since last successful read:         n/a
  24 Scraping failed blocks... (backwards)

The program exits from time to time, when the device disappears and is found again. To encounter this, you may wrap the commands in a while-loop. (and for fun count its iterations). This loop will run until ddrescue exits successfully RC=0.

   1 RETURN=1         
   2 COUNT=0
   3 DIR_OUT="/media/space/disk_images/old1"
   4 [ -d "$DIR_OUT" ] || mkdir "$DIR_OUT"
   5 while [ "$RETURN" -eq 1 ]; do     
   6         echo '\n'"Number of iterations: "$((++COUNT)) 
   7         ddrescue \       
   8         -vvv -B -R -r 3 \
   9         /dev/sdc \
  10         "$DIR_OUT"/sdcard.img \
  11         "$DIR_OUT"/sdcard.map
  12         RETURN="$?"
  13 done

Get disk geometry from fdisk

   1 fdisk --units=cylinders -l /dev/sdc     
   2 Disk /dev/sdc: 189.9 GiB, 203928109056 bytes, 398297088 sectors
   3 Disk model: 2L000P
   4 Geometry: 255 heads, 63 sectors/track, 24792 cylinders
   5 Units: cylinders of 16065 * 512 = 8225280 bytes
   6 Sector size (logical/physical): 512 bytes / 512 bytes
   7 I/O size (minimum/optimal): 512 bytes / 512 bytes
   8 Disklabel type: dos
   9 Disk identifier: 0x33dbd580
  10 
  11 Device     Boot Start   End Cylinders   Size Id Type
  12 /dev/sdc1           1 24792     24792 189.9G  7 HPFS/NTFS/exFAT

ddrescueview

Take a look at the progress of the scape.

   1 # apt install gddrescueview
   2 

   1 # ddrescueview /media/space/disk_images/old1/disk_old1_p1.map
   2 

ddrescue-gui

   1 apt install libgtk-3-dev

Create a file requirements.txt

   1 getdevinfo
   2 wxPython

   1 git clone https://git.launchpad.net/ddrescue-gui
   2 cd ddrescue-gui
   3 virtualenv -p python3 venv
   4 source venv/bin/activate
   5 pip install -r requirements.txt
   6 python setup.py --verbose build
   7 python setup.py --verbose install

Either:

   1 ./DDRescue_GUI.py

Whole disk images

kpartx

   1 aptitude install kpartx

   1 # kpartx -av /home/tobias/sdcard.img
   2 add map loop2p1 (253:0): 0 250079232 linear 7:2 2048
   3 # lsblk -p /dev/loop2
   4 NAME                  MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
   5 /dev/loop2              7:2    0 119.3G  0 loop
   6 └─/dev/mapper/loop2p1 253:0    0 119.3G  0 part

Drive read-only

SD cards have a little physical switch on the side for write protection …

parted /dev/sdc

   1 parted /dev/sdc
   2 Warning: Unable to open /dev/sdc read-write (Das Dateisystem ist nur lesbar).  /dev/sdc has been opened read-only.
   3 GNU Parted 3.3                                                            
   4 Using /dev/sdc
   5 Welcome to GNU Parted! Type 'help' to view a list of commands.
   6 (parted) print free
   7 Model: Lexar microSD RDR (scsi)
   8 Disk /dev/sdc: 128GB
   9 Sector size (logical/physical): 512B/512B
  10 Partition Table: gpt
  11 Disk Flags: 
  12 
  13 Number  Start   End     Size    File system  Name         Flags
  14         17.4kB  1049kB  1031kB  Free Space
  15  1      1049kB  128GB   128GB   fat32        data_part  msftdata
  16         128GB   128GB   1032kB  Free Space
  17 
  18 (parted) quit

hdparm /dev/sdc

   1 /dev/sdc:
   2 SG_IO: bad/missing sense data, sb[]:  70 00 05 00 00 00 00 0a 00 00 00 00 20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
   3  multcount     =  0 (off)
   4  readonly      =  1 (on)
   5  readahead     = 256 (on)
   6  geometry      = 15566/255/63, sectors = 250083328, start = 0

hdparm -r0 -v /dev/sdc

   1 /dev/sdc:
   2  setting readonly to 0 (off)
   3 SG_IO: bad/missing sense data, sb[]:  70 00 05 00 00 00 00 0a 00 00 00 00 20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
   4  multcount     =  0 (off)
   5  readonly      =  0 (off)
   6  readahead     = 256 (on)
   7  geometry      = 15566/255/63, sectors = 250083328, start = 0

Yeah, did not work…/dev/sr0

Probably the flash device has detected some hardware errors and set itself read-only. Backup what's rescueable and go with another device.

Disable read-ahead

Just an idea: You may change the amount of sectors a drive reads ahead of the requested data (read-ahead) to limit the focus to the area being scraped.

   1 hdparm -a32 /dev/sr0

Set drive max. speed

   1 hdparm -E48 /dev/sr0

Rockstable Wiki: Clone (last edited 2020-05-15 12:45:09 by RockStable)