Clone
Contents
A running system shall be cloned from on disk to a fresh and different filesystem (in my case xfs to btrfs). dump works only for ext-filesystems (and restore versions maybe incompatible across the systems). I chose rsync, because it supports:
- resume from partial copies and
- may use the network over securely encrypted ssh
- …
Live CD
You'll probably need a live-cd like the ones from https://grml.org.
Simply diskdump it to an usb stick.
Try it, before you need it!
Remote control
I recommend controlling the live environment over ssh and tmux. So you can use the big monitor.
On the live system
On the remote controller
Partition
parted /dev/vdb
1 GNU Parted 3.2
2 Using /dev/vdb
3 Welcome to GNU Parted! Type 'help' to view a list of commands.
4 (parted) unit MiB
5 (parted) mktable gpt
6 (parted) mkpart bios_grub 1 2
7 (parted) mkpart EFI 2 514
8 (parted) mkpart swap 514 2562
9 (parted) mkpart boot 2562 3074
10 (parted) mkpart root 3074 -1
11 (parted) set 1 bios_grub on
12 (parted) print free
13 Model: Virtio Block Device (virtblk)
14 Disk /dev/vdb: 30720MiB
15 Sector size (logical/physical): 512B/512B
16 Partition Table: gpt
17 Disk Flags:
18
19 Number Start End Size File system Name Flags
20 0,02MiB 1,00MiB 0,98MiB Free Space
21 1 1,00MiB 2,00MiB 1,00MiB bios_grub bios_grub
22 2 2,00MiB 514MiB 512MiB EFI
23 3 514MiB 2562MiB 2048MiB swap1
24 4 2562MiB 3074MiB 512MiB boot
25 5 3074MiB 30719MiB 27645MiB root
26 30719MiB 30720MiB 0,98MiB Free Space
27
28 (parted)
29 (parted) quit
30 Information: You may need to update /etc/fstab.
Format
Mount
Copy
rsync
Used rsync options:
- mandatory:
-a, --archive archive mode; equals -rlptgoD (no -H,-A,-X)
-r, --recursive recurse into directories
-l, --links copy symlinks as symlinks
-p, --perms preserve permissions
-t, --times preserve modification times
-g, --group preserve group
-o, --owner preserve owner (super-user only)
-D same as --devices --specials
-A, --acls preserve ACLs (implies -p)
-x, --one-file-system don't cross filesystem boundaries
-X, --xattrs preserve extended attributes
-H, --hard-links preserve hard links
-S, --sparse turn sequences of nulls into sparse blocks
- optional:
--bwlimit=RATE maximum transfer rate for the data sent over the socket
-P same as --partial --progress
--partial keep partially transferred files
--progress show progress during transfer
-v, --verbose increase verbosity
-h, --human-readable output numbers in a human-readable format
-e, --rsh=COMMAND choose an alternative remote shell program with options for example:
-e 'ssh -p 2234'
-e 'ssh -o "ProxyCommand nohup ssh firewall nc -w1 %h %p"'
Transfer the root filesystem to a filesystem, that is mounted to /mnt
bit-wise locally with dd
Remember: Everything's a file.
From local file to local file.
bit-wise remote over ssh
When your copy is limited by Gigabit line-speed use the parallel implementation of gzip pigz to compress the stream before it passes and saturates the network. It's terribly fast compared to its single threaded companions embedded in ssh or rsync. The more CPU-time you have the faster the compression of the stream. As long as you have enough CPU time, network may not be saturated to line speed and disk-IO will become the limiting factor.
Transfer whole disk IF (input file) to target host via ssh and write it to an image file OF (output file).
1 IF="/dev/sda"
2 OF="$IF"
3 dd if="$IF" \
4 |pv -s "$(blockdev --getsize64 "$IF")" \
5 |pigz -c \
6 |ssh user@target.host \
7 "cat - > \"rel/path/to/$HOSTNAME-$(basename \"$OF\").img\""
8
9 ### EXAMPLE OUTPUT WITH 8CORES (4COREs + SMT) -> LAPTOP-IMAGE
10 476GiB 0:53:53 [ 151MiB/s] [==================================================================>] 100%
11 dd if="$DEVICE" 224.75s user 1358.92s system 48% cpu 53:53.38 total
12 pv -s "$(blockdev --getsize64 "$DEVICE")" 43.63s user 1171.22s system 37% cpu 53:53.38 total
13 pigz -c 11589.45s user 566.54s system 375% cpu 53:53.38 total
14 ssh tobias@libertas "cat > \"$HOSTNAME-$(basename $DEVICE).img\"" 304.02s user 281.79s system 18% cpu 53:53.43 total
15
16 ### 512110190592/(53*60+53)/2**20 = 151.06 MiB/s
17 ### IS FASTER THAN THE LINE-SPEED OF THE UNDERLYING 1GBASE-T
18 ### AND ITS ALREADY COMPRESSED ON THE REMOTE SITE (56GiB) :-)
19
20 ### WITH ELEVATED PRIVILEGES ON REMOTE SYSTEM
21 DEVICE=/dev/sda
22 dd if=$DEVICE \
23 |pv -s "$(blockdev --getsize64 "$DEVICE")" \
24 |pigz -c \
25 |ssh -t user@target.host \
26 "sudo bash -c 'cat - > \"rel/path/to/$HOSTNAME-$(basename \"$DEVICE\").img\"'"
Tar a whole filesystem to a compressed archive on a remote host.
Bootloader
Chroot into new filesystem
Install bootloader (still no EFI)
Configure bootloader
1 update-grub2
Append current information in a safe fashion to /etc/fstab
1 blkid | sed -r 's/^/# /' >> /etc/fstab
Alter file system table /etc/fstab
1 # /etc/fstab: static file system information.
2 #
3 # Use 'blkid' to print the universally unique identifier for a
4 # device; this may be used with UUID= as a more robust way to name devices
5 # that works even if disks are added and removed. See fstab(5).
6 #
7 #<file_system> <mount_point> <type> <options> <dump> <pass>
8 proc /proc proc defaults 0 0
9 UUID=d597c3bb-6c6c-4c4d-9b83-2442c3bf72fd /boot btrfs defaults 0 2
10 UUID=8d1f76ca-a06f-49de-8ac4-283ade0da7a5 / btrfs rw,noatime,compress=lzo,ssd,space_cache 0 1
11 UUID=1e82e970-7a9d-4b47-b959-b611834c3c37 none swap sw 0 0
12
13 # /dev/vda1: LABEL="bootfs" UUID="70d3b316-7f05-48d9-9498-8144b0cb4473" SEC_TYPE="ext2" TYPE="ext3"
14 # /dev/vda2: LABEL="rootfs" UUID="5ef7d751-2911-4005-b97b-0ca9945ecfc4" TYPE="xfs"
15 # /dev/vda5: UUID="7c19aebf-abcc-4c93-a7af-f5331aeb650c" TYPE="swap"
16 # /dev/vdb4: LABEL="boot" UUID="d597c3bb-6c6c-4c4d-9b83-2442c3bf72fd" UUID_SUB="0413709e-d471-432c-8f94-64b99421d841" TYPE="btrfs"
17 # /dev/vdb5: LABEL="root" UUID="8d1f76ca-a06f-49de-8ac4-283ade0da7a5" UUID_SUB="9f8a3778-b726-4d2c-beb1-84fd3267e53f" TYPE="btrfs"
18 # /dev/vdb3: LABEL="swap1" UUID="1e82e970-7a9d-4b47-b959-b611834c3c37" TYPE="swap"
19
Confiure initrd
Fix initrd resume to swap /etc/initramfs-tools/conf.d/resume
1 RESUME=UUID=1e82e970-7a9d-4b47-b959-b611834c3c37
Rebuild initrd
1 root@template-stretch / # update-initramfs -k all -u
2 update-initramfs: Generating /boot/initrd.img-4.8.0-1-amd64
3 cp: der Aufruf von stat für '/etc/modprobe.d/*' ist nicht möglich: Datei oder Verzeichnis nicht gefunden
4 /boot/initrd.img-4.7.0-1-amd64 does not exist. Cannot update.
5 /boot/initrd.img-3.16.0-4-amd64 does not exist. Cannot update.
6 /boot/initrd.img-3.14-2-amd64 does not exist. Cannot update.
7 /boot/initrd.img-3.2.0-4-amd64 does not exist. Cannot update.
8 /boot/initrd.img-3.2.0-1-amd64 does not exist. Cannot update.
Reboot
1 shutdown -r now
Transform to EFI
EFI System partition (ESP)
Install FAT toolsuite
1 aptitude install dosfstools
Format EFI system partition (ESP) with FAT12/16/32
/etc/fstab
1 # /etc/fstab: static file system information.
2 #
3 # Use 'blkid' to print the universally unique identifier for a
4 # device; this may be used with UUID= as a more robust way to name devices
5 # that works even if disks are added and removed. See fstab(5).
6 #
7 #<file_system> <mount_point> <type> <options> <dump> <pass>
8 proc /proc proc defaults 0 0
9 UUID=3452-16B9 /boot/EFI vfat defaults 0 2
10 UUID=d597c3bb-6c6c-4c4d-9b83-2442c3bf72fd /boot btrfs defaults 0 2
11 UUID=8d1f76ca-a06f-49de-8ac4-283ade0da7a5 / btrfs rw,noatime,compress=lzo,ssd,space_cache 0 1
12 UUID=1e82e970-7a9d-4b47-b959-b611834c3c37 none swap sw 0 0
13
14 # /dev/vda3: LABEL="swap1" UUID="1e82e970-7a9d-4b47-b959-b611834c3c37" TYPE="swap" PARTLABEL="swap1" PARTUUID="72314aa4-000a-4150-9996-313a9114d777"
15 # /dev/vda4: LABEL="boot" UUID="d597c3bb-6c6c-4c4d-9b83-2442c3bf72fd" UUID_SUB="0413709e-d471-432c-8f94-64b99421d841" TYPE="btrfs" PARTLABEL="boot" PARTUUID="de599caa-78aa-4747-a5bc-91a4f71b333c"
16 # /dev/vda5: LABEL="root" UUID="8d1f76ca-a06f-49de-8ac4-283ade0da7a5" UUID_SUB="9f8a3778-b726-4d2c-beb1-84fd3267e53f" TYPE="btrfs" PARTLABEL="root" PARTUUID="645044b0-0228-4c2b-92c4-555b2e72d59a"
17 # /dev/vda1: PARTLABEL="bios_grub" PARTUUID="ec0c7283-0c5f-42ee-883f-29cc92cc1f81"
18 # /dev/vda2: LABEL="EFI" UUID="3452-16B9" TYPE="vfat" PARTLABEL="EFI" PARTUUID="742b1a54-ff3a-49e1-8bfb-85f92da1778a"
19
Mount the new filesystems
GRUB-EFI
Install grub-efi and remove grub-pc
1 aptitude install grub-efi grub-pc-
Install bootloader and create EFI-image
1 root@template-stretch /boot/EFI # grub-install \
2 --efi-directory=/boot/EFI --recheck \
3 --directory=/usr/lib/grub/x86_64-efi /dev/vda
4 x86_64-efi wird für Ihre Plattform installiert.
5 EFI variables are not supported on this system.
6 EFI variables are not supported on this system.
7 installation beendet. Keine Fehler aufgetreten.
Update grub configuration
Libvirt with EFI
Please refer to Libvirt with EFI
Disk Dump Rescue (ddrescue)
About
The process is interuptable and resumable.
If the electronic is still alive, ddrescue can help.
- I love this little programm!!! Thanks for this little piece of hope.
Material
- You should get a external usb3 adapter to IDE 2.5, IDE 3.5 and SATA with a physical power-switch.
- I'm using a raspberry-pi to perform the hard work. Scraping takes an eternity.
- Low power consumption.
- Small, yet powerful.
When to stop
I personally stop rescuing the disk when and non-trimmed sectors are read. On a example of a DVD -> this already took 4 days … ddrescue then tries to scrape the non-scraped blocks, which is another magnitude slower then the last stage. Time since last successfull read explodes. For me its time to stop and get the data.
1 ipos: 4206 MiB, non-trimmed: 62002 KiB, current rate: 66 B/s
2 opos: 4206 MiB, non-scraped: 185878 KiB, average rate: 635 B/s
3 non-tried: 0 B, bad-sector: 15500 KiB, error rate: 66 B/s
4 rescued: 4042 MiB, bad areas: 7732, run time: 4d 31m 16s
5 pct rescued: 94.01%, read errors: 9881, remaining time: 1d 10h 7m
6 time since last successful read: 0s
7 Trimming failed blocks... (forwards)
8
9 ipos: 3721 MiB, non-trimmed: 0 B, current rate: 0 B/s
10 opos: 3721 MiB, non-scraped: 223874 KiB, average rate: 507 B/s
11 non-tried: 0 B, bad-sector: 23148 KiB, error rate: 68 B/s
12 rescued: 4058 MiB, bad areas: 8858, run time: 5d 9h 58m
13 pct rescued: 94.39%, read errors: 13705, remaining time: 28d 16h 51m
14 time since last successful read: 1m 31s
15 Scraping failed blocks... (forwards)
But if the data is of extraordinary importance you should probably carry on. If you keep the media, the image and the map file you can continue the rescue later.
ddrescue
Install GNU ddrescue
1 aptitude install gddrescue
1 ### FIRST RUN FORWARD WITHOUT SCRAPING,
2 ### FOR FAST RESULTS
3 ddrescue \
4 -vvv -B -n \
5 --max-read-rate=$((48*2**20)) \
6 /dev/sdc1 \
7 /media/space/disk_images/old1/disk_old1_p1.raw \
8 /media/space/disk_images/old1/disk_old1_p1.map
9
10 ### SECOND RUN FROM REVERSE WITH 3 RETRIES
11 ddrescue \
12 -vvv -B -R -r 3 \
13 --max-read-rate=$((48*2**20)) \
14 /dev/sdc1 \
15 /media/space/disk_images/old1/disk_old1_p1.raw \
16 /media/space/disk_images/old1/disk_old1_p1.map
17
18 ### OPTICAL MEDIUM WITH SCRAPING (3 RETRIES)
19 ddrescue \
20 -vvv -B -r 3 \
21 -b 2048 \
22 /dev/sr0 \
23 /media/space/disk_images/sr0.img \
24 /media/space/disk_images/sr0.map
25
26 ### WITH VARS
27 PATH_IMG="/home/tobias/workspace/disk_images"
28 DEV="sda1"
29 mkdir -p "$PATH_IMG"
30 ddrescue \
31 -vvv -B -n \
32 "/dev/$DEV" \
33 "$PATH_IMG/$DEV.raw" \
34 "$PATH_IMG/$DEV.map"
Used options:
-b bytes|--sector-size=bytes -> Sector (hardware block) size of input device in bytes (usually 512 for hard discs and 3.5" floppies, 1024 for 5.25" floppies, and 2048 for cdroms). Defaults to 512.
-B|--binary-prefixes -> Show units with binary prefixes (powers of 1024). SI prefixes (powers of 1000) are used by default. (See table below). 1MiB (SI IEC 60027-2 -> power of 2) instead of human readable (power of 10)
-c sectors|--cluster-size=sectors -> Number of sectors to copy at a time. Defaults to 64 KiB / sector_size. Try smaller values for slow drives. The number of sectors per track (18 or 9) is a good value for floppies.
-n|--no-scrape -> Skip the scraping phase. Avoids spending a lot of time trying to rescue the most difficult parts of the file.
-r n|--retry-passes=n -> Exit after the given number of retry passes. Defaults to 0. -1 means infinity. Every bad sector is tried only once in each pass. To retry bad sectors detected on a previous run, you must specify a non-zero number of retry passes.
-R|--reverse -> Reverse the direction of all passes (copying, trimming, scraping and retrying). Every pass that is normally run forwards will now be run backwards, and vice versa. '--reverse' does not modify the size of the blocks copied during each phase, just the order in which they are tried.
-v|--verbose -> Verbose mode. Further -v's (up to 4) increase the verbosity level.
example output
1 GNU ddrescue 1.23
2 About to copy 194474 MiBytes from '/dev/sdc1' [UNKNOWN] (203921109504) to '/media/space/disk_images/old1/disk_old1_p1.raw' (203921109504)
3 Starting positions: infile = 0 B, outfile = 0 B
4 Copy block size: 128 sectors Initial skip size: 4096 sectors
5 Sector size: 512 Bytes
6 Max read rate: 49152 KiB/s
7 Direct in: no Direct out: no Sparse: no Truncate: no
8 Trim: yes Scrape: yes Max retry passes: 3
9 Reverse mode
10
11 Press Ctrl-C to interrupt
12 Initial status (read from mapfile)
13 current position: 1880 MiB, current sector: 3850349
14 last block size: 192580 MiB
15 rescued: 194473 MiB, tried: 802304 B, bad-sector: 157184 B, bad areas: 105
16
17 Current status
18 ipos: 1880 MiB, non-trimmed: 0 B, current rate: 0 B/s
19 opos: 1880 MiB, non-scraped: 645120 B, average rate: 0 B/s
20 non-tried: 0 B, bad-sector: 157184 B, error rate: 0 B/s
21 rescued: 194473 MiB, bad areas: 105, run time: 0s
22 pct rescued: 99.99%, read errors: 0, remaining time: n/a
23 time since last successful read: n/a
24 Scraping failed blocks... (backwards)
The program exits from time to time, when the device disappears and is found again. To encounter this, you may wrap the commands in a while-loop. (and for fun count its iterations). This loop will run until ddrescue exits successfully RC=0.
1 RETURN=1
2 COUNT=0
3 DIR_OUT="/media/space/disk_images/old1"
4 [ -d "$DIR_OUT" ] || mkdir "$DIR_OUT"
5 while [ "$RETURN" -eq 1 ]; do
6 echo '\n'"Number of iterations: "$((++COUNT))
7 ddrescue \
8 -vvv -B -R -r 3 \
9 /dev/sdc \
10 "$DIR_OUT"/sdcard.img \
11 "$DIR_OUT"/sdcard.map
12 RETURN="$?"
13 done
Get disk geometry from fdisk
1 fdisk --units=cylinders -l /dev/sdc
2 Disk /dev/sdc: 189.9 GiB, 203928109056 bytes, 398297088 sectors
3 Disk model: 2L000P
4 Geometry: 255 heads, 63 sectors/track, 24792 cylinders
5 Units: cylinders of 16065 * 512 = 8225280 bytes
6 Sector size (logical/physical): 512 bytes / 512 bytes
7 I/O size (minimum/optimal): 512 bytes / 512 bytes
8 Disklabel type: dos
9 Disk identifier: 0x33dbd580
10
11 Device Boot Start End Cylinders Size Id Type
12 /dev/sdc1 1 24792 24792 189.9G 7 HPFS/NTFS/exFAT
Create a backup of the image
Create a backup (btrfs)
ddrescueview
Take a look at the progress of the scape.
ddrescue-gui
1 apt install libgtk-3-dev
Create a file requirements.txt
Either:
install directory images to RESOURCEPATH = '/usr/share/ddrescue-gui'
change RESOURCEPATH = '.'
1 ./DDRescue_GUI.py
Whole disk images
kpartx
1 aptitude install kpartx
Drive read-only
I received
dd: failed to open '/dev/sdc': Read-only file system
First, SD cards may have a little physical switch on the side for write protection.
- If the switch is towards the contacts, writing to the card is allowed.
- Write protection is detected by the device
that reads the card, not by the card itself.
parted /dev/sdc
1 parted /dev/sdc
2 Warning: Unable to open /dev/sdc read-write (Read-only file system). /dev/sdc has been opened read-only.
3 GNU Parted 3.3
4 Using /dev/sdc
5 Welcome to GNU Parted! Type 'help' to view a list of commands.
6 (parted) print free
7 Model: Lexar microSD RDR (scsi)
8 Disk /dev/sdc: 128GB
9 Sector size (logical/physical): 512B/512B
10 Partition Table: gpt
11 Disk Flags:
12
13 Number Start End Size File system Name Flags
14 17.4kB 1049kB 1031kB Free Space
15 1 1049kB 128GB 128GB fat32 data_part msftdata
16 128GB 128GB 1032kB Free Space
17
18 (parted) quit
hdparm /dev/sdc
hdparm -r0 -v /dev/sdc
1 /dev/sdc:
2 setting readonly to 0 (off)
3 SG_IO: bad/missing sense data, sb[]: 70 00 05 00 00 00 00 0a 00 00 00 00 20 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
4 multcount = 0 (off)
5 readonly = 0 (off)
6 readahead = 256 (on)
7 geometry = 15566/255/63, sectors = 250083328, start = 0
Yeah, did not work … /dev/sr0
Probably the flash device has detected some hardware errors and set itself read-only. Backup what's rescueable and go with another device.
Disable read-ahead
Just an idea: You may change the amount of sectors a drive reads ahead of the requested data (read-ahead) to limit the focus to the area being scraped.
1 hdparm -a32 /dev/sr0
Set drive max. speed
1 hdparm -E48 /dev/sr0
Data recovery
Recover files from FAT32
Determine file types
1 cd /home/tobias/workspace/disk_images
2 mkdir mnt
3 mount sda1.raw mnt
4 cd mnt
5
6 file --mime FOUND.*/*
7 FOUND.000/FILE0000.CHK: application/vnd.openxmlformats-officedocument.wordprocessingml.document; charset=binary
8 FOUND.000/FILE0001.CHK: application/octet-stream; charset=binary
9 FOUND.000/FILE0002.CHK: application/octet-stream; charset=binary
10 FOUND.000/FILE0003.CHK: application/octet-stream; charset=binary
11 FOUND.000/FILE0004.CHK: application/octet-stream; charset=binary
12 FOUND.000/FILE0005.CHK: application/vnd.openxmlformats-officedocument.wordprocessingml.document; charset=binary
13 FOUND.000/FILE0006.CHK: application/vnd.openxmlformats-officedocument.wordprocessingml.document; charset=binary
14 FOUND.000/FILE0007.CHK: application/octet-stream; charset=binary
15 FOUND.000/FILE0008.CHK: application/octet-stream; charset=binary
16 FOUND.000/FILE0009.CHK: application/zip; charset=binary
17
18 file FOUND.*/*
19 FOUND.000/FILE0000.CHK: Microsoft Word 2007+
20 FOUND.000/FILE0001.CHK: data
21 FOUND.000/FILE0002.CHK: data
22 FOUND.000/FILE0003.CHK: data
23 FOUND.000/FILE0004.CHK: data
24 FOUND.000/FILE0005.CHK: Microsoft Word 2007+
25 FOUND.000/FILE0006.CHK: Microsoft Word 2007+
26 FOUND.000/FILE0007.CHK: data
27 FOUND.000/FILE0008.CHK: data
28 FOUND.000/FILE0009.CHK: Zip archive data, at least v2.0 to extract, compression method=deflate
Testdisk
Install testdisk (and photorec)
Testdisk also operates well on disk images created by #ddrescue
Secure data removal
Takes an eternity.
DBAN
Darik's Boot and Nuke
Free Open-Source Data Wiping Software for Personal Use
Delete information stored on hard disk drives (HDDs) in PC laptops, desktops or servers. Plus, remove viruses/spyware from Microsoft Windows installations.
wipe
Wipe from the DBAN project
Overwrite target 35 times (incl. 8 random passes) which takes very long
Overwrite target in quick-mode 5 times (default: 4) with random patterns seeded by /dev/urandom, which takes a lot less time, but is less secure.
1 tmux
2 lsblk
3 ### CHECK YOUR DISK CAREFULLY!1!!
4 DISK=sda
5 [ -b "/dev/$DISK" ] && \
6 nice ionice -c3 \
7 wipe -i -qQ5 "/dev/$DISK" \
8 -R /dev/urandom \
9 |tee "$DISK_$(date +%F_%H%M).log"
10 Okay to WIPE 1 special file ? (Yes/No) yes
11 Wiping /dev/sda, pass 0 in quick mode [ 31032 / 12446784] ETA 11h08m
12 [0] 0:wipe*Z 1:man-
nwipe
About
- Fork of wipe with an ncurses gui
- The table of controls is in the footer
Start the gui
Shred
Shred the disk
Notes
IMHO: Shred seems to have some trouble with slightly affected disks. [#wipe] seems to be more robust and sophisticated.