oneliners

Contents

oneliners

Strip ANSI control codes

   1 sed -r "s/\x1B\[([0-9]{1,3}(;[0-9]{1,2};?)?)?[mGK]//g" < run.log > run_plain.log

Join line continuations

This may be useful for parsing /etc/aliases or /etc/postfix/main.cf.

In GNU-sed you can use Null-Byte as to separate lines and replace every newline, which is followed by at least 1 whitespace character with a single space.

   1 sed -zr 's/\n\s+/ /g'

With conventional sed use the following command.

   1 sed 'H;1h;$!d;g;s/\n\s\+/ /g'

grep multiline

This command is using experimental PCRE with --null-data to extract the SOA resource record from a BIND9 zone file.

   1 grep -H --color=always -Pzo \
   2         '(\$ORIGIN\s+.+\n)?(.*\n)+([[:print:]]+)\s+(IN\s+)?SOA\s+.+\((\n([    ]+[[:print:]]+)+)+\n' \
   3         /etc/bind/zones/*

pcregrep multiline

You may also install pcregrep and use conventional multiline syntax.

   1 apt install pcregrep
   2 
   3 pcregrep -H -M --color=always \
   4         '(^\$ORIGIN\s+.+\n$)?(.*\n)+(^[[:print:]]+)\s+(IN\s+)?SOA\s+.+\($(\n^\s+.*$)+' \
   5         /etc/bind/zones/*

Ask for password

I want want make sure that passwords are not recorded in shell-history, so i wrote a little alias that shortens the process of saving the password in a shell variable. This variable can then be specified in a shell command without "leaking" the password to the history.

Example

   1 mysql -u root@localhost -p"$PASSWORD" database_name

NB: This won't protect the password from being readable from the list of processes.

   1 $ yes echo $PASSWORD
   2 test
   3 test
   4 …

Meanwhile on another shell:

   1 $ ps faux | grep -v grep | grep -B1 "yes echo"
   2 tobias    3313  0.0  0.0  10556  6124 pts/9    Ss   19:32   0:00  |   \_ /usr/bin/zsh
   3 tobias    3526 38.2  0.0   5584   700 pts/9    S+   19:39   0:16  |       \_ yes echo test

pask bash

   1 alias pask='stty -echo; read -rp "Password: " PASSWORD; stty echo;'

More flexible implementation as shell function

   1 pask () {
   2         VARIABLE="PASSWORD"
   3         if [ -n "$1" ] &&
   4                 grep -qE '^[[:alnum:]]+' <<< "$1"; then
   5                 VARIABLE="$1"
   6         fi
   7 
   8         stty -echo;
   9         read -rp "Secret to store in '$VARIABLE': " $VARIABLE;
  10         echo; 
  11         stty echo;
  12 }

pask zsh

   1 alias pask='read -rs "PASSWORD?Password: "; echo'

More flexible implementation as shell function

   1 pask () {         
   2         VARIABLE="PASSWORD"
   3         if [ -n "$1" ] &&
   4                 grep -qE '^[[:alnum:]]+' <<< "$1"; then
   5                 VARIABLE="$1"
   6         fi
   7 
   8         read -rs "$VARIABLE?Secret to store in '$VARIABLE': ";
   9         echo;
  10 }

man - jump to regex in man page

serverfault.com: jump to man heading from cli

   1 man -P "less -p '^RegEx'" section_nr page

Add this little function to your .bashrc or .zshrc:

   1 mans () {
   2         local pages string
   3         if [[ -n "$2" ]]
   4         then
   5                 pages=(${@:2})
   6                 string="$1"
   7         else
   8                 pages="$1"
   9         fi
  10         # GNU man
  11         man ${2:+--pager="less -p '$string' -G"} ${pages[@]}
  12         # BSD man
  13         # man ${2:+-P "less -p '$string' -G"} ${pages[@]}
  14 }

This simplyfies the command to:

   1 mans '^RegEx' page

Transpose TABLE with multiline regex in python3

Table looks like this:

   1 VLAN:   1
   2 Type:   Static
   3 Name:   default
   4 Status: Active
   5 VLAN:   2
   6 Type:   Static
   7 Name:   foo
   8 Status: Active
   9 VLAN:   3
  10 Type:   Static
  11 Name:   bar
  12 Status: Active
  13 VLAN:   4
  14 Type:   Static
  15 Name:   baz
  16 Status: Active

Command to transpose

   1 % python3 -c "
   2 import sys,re,os;
   3 buffer=open('vlans_sw-int.txt').read();
   4 [sys.stdout.write(
   5          re.sub(  r'^(VLAN):\s+(.+)\n$^(Type):\s+(.*)\n$^(Name):\s+(.+)\n$^(Status):\s+(.+)',
   6                   r'\2\t\4\t\6\t\8',
   7                   buffer,
   8                   0,
   9                   re.MULTILINE)
  10          )
  11 ]"

Output looks like this:

   1 1       Static  default Active
   2 2       Static  foo     Active
   3 3       Static  bar     Active
   4 4       Static  baz     Active

Github

Get Latest Tag

   1 export USER="Icinga"
   2 export REPO="icingaweb2-module-reactbundle"
   3 curl "https://api.github.com/repos/$USER/$REPO/tags" 2>/dev/null \
   4         |jq -r 'sort_by(.name)|reverse|.[0].name'

Or as a batch-job with input file icingaweb2-modules.list.

   1 while read REPO; do
   2         USER="Icinga"
   3         echo -ne "$REPO\t"
   4         curl "https://api.github.com/repos/$USER/$REPO/tags" 2>/dev/null \
   5         |jq -r 'sort_by(.name)|reverse|.[0].name'
   6 done < "icingaweb2-modules.list"| column -t
   7 icingaweb2-module-audit          v1.0.1
   8 icingaweb2-module-aws            v1.0.0
   9 icingaweb2-module-cube           v1.1.0
  10 icingaweb2-module-doc            null
  11 icingaweb2-module-elasticsearch  v0.9.0
  12 icingaweb2-module-fileshipper    v1.1.0
  13 icingaweb2-module-generictts     v2.0.0
  14 icingaweb2-module-icingadb       null
  15 icingaweb2-module-idoreports     v0.9.1
  16 icingaweb2-module-jira           v1.0.1
  17 icingaweb2-module-pdfexport      v0.9.1
  18 icingaweb2-module-pnp            v1.1.0
  19 icingaweb2-module-pnp            v1.1.0
  20 icingaweb2-module-pnp4nagios     null
  21 icingaweb2-module-reporting      v0.9.1
  22 icingaweb2-module-toplevelview   v0.3.1
  23 icingaweb2-module-training       null
  24 icingaweb2-module-vsphere        v1.1.0
  25 icingaweb2-module-vspheredb      v1.1.0
  26 icingaweb2-module-x509           v1.0.0
  27 icingaweb2-theme-company         v1.0.0

Get Latest Release

Inspired by lukechilds/get_latest_release.sh

github_release.sh

   1 #!/bin/bash
   2 
   3 PROGRAM="${0#*/}"
   4 OPTIONS_WGET=("-q" "--backups" "0")
   5 GET_ASSETS=false
   6 GET_TARBALL=false
   7 GET_ZIPBALL=false
   8 GET_VERSION=false
   9 
  10 declare ASSETS
  11 
  12 usage () {
  13         cat <<-EOH
  14                 $PROGRAM [OPTIONS] -- PROJECT [PROJECT]
  15                 where:
  16                   PROJECT is something like "user/repo"
  17                   --            Breaks option parsing
  18                   []            optional parameters 
  19                   OPTIONS are:
  20                     -a|--assets Get assets
  21                     -h|--help   Print this page
  22                     -t|--tarball        Get tarball
  23                     -v|--verbose        More detailed output
  24                     -V|--version        Get version
  25                     -z|--zipball        Get zipball
  26         EOH
  27 }
  28 
  29 TEMP=$(getopt \
  30         -o 'ahtvVz' \
  31         --long 'assets,help,tarball,verbose,version,zip' \
  32         -n "$PROGRAM" \
  33         -- "$@")
  34 
  35 if [ $? -ne 0 ]; then
  36         echo 'Terminating...' >&2
  37         exit 1
  38 fi
  39 
  40 # Note the quotes around "$TEMP": they are essential!
  41 eval set -- "$TEMP"
  42 unset TEMP
  43 
  44 while true; do
  45         case "$1" in
  46                 '-a'|'--assets')
  47                         GET_ASSETS=true
  48                         shift
  49                         continue
  50                 ;;
  51                 '-h'|'--help')
  52                         usage
  53                         exit 0
  54                 ;;
  55                 '-t'|'--tarball')
  56                         GET_TARBALL=true
  57                         shift
  58                         continue
  59                 ;;
  60                 '-v'|'--verbose')
  61                         WGET_OPTIONS+=" -v"
  62                         shift
  63                         continue
  64                 ;;
  65                 '-V'|'--version')
  66                         GET_VERSION=true
  67                         shift
  68                         continue
  69                 ;;
  70                 '-z'|'--zipball')
  71                         GET_ZIPBALL=true
  72                         shift
  73                         continue
  74                 ;;
  75                 '--')
  76                         shift
  77                         break
  78                 ;;
  79                 *)
  80                         echo 'Internal error!' >&2
  81                         exit 1
  82                 ;;
  83         esac
  84 done
  85 
  86 release_info () {
  87         local PROJECT="$1"
  88 
  89         URL_API="https://api.github.com/repos/$PROJECT/releases/latest"
  90         RESPONSE="$(curl --silent "$URL_API")"
  91         VERSION="$(jq -r .tag_name <<< "$RESPONSE")"
  92         URL_TARBALL="$(jq -r .tarball_url <<< "$RESPONSE")"
  93         URL_ZIPBALL="$(jq -r .zipball_url <<< "$RESPONSE")"
  94         while read -r LINE; do
  95                 ASSETS+=( "$LINE" )
  96         done <<< "$(jq -r '.assets|.[]|.browser_download_url' <<< "$RESPONSE")"
  97 }
  98 
  99 release_download () {
 100         local PROJECT="$1"
 101         local VERSION="$2"
 102 
 103         echo "Downloading \"$PROJECT\" in \"$VERSION\""
 104         DIR_DL="release/$PROJECT/$VERSION"
 105         mkdir -p "$DIR_DL"
 106         pushd "$DIR_DL" >/dev/null || exit 2
 107         $GET_TARBALL && wget "${OPTIONS_WGET[@]}" "$URL_TARBALL" -O "$PROJECT_BASENAME-$VERSION.tar.gz"
 108         $GET_ZIPBALL && wget "${OPTIONS_WGET[@]}" "$URL_ZIPBALL" -O "$PROJECT_BASENAME-$VERSION.zip"
 109         if $GET_ASSETS; then
 110                 for URL in "${ASSETS[@]}"; do
 111                         wget "${OPTIONS_WGET[@]}" "$URL"
 112                 done
 113         fi
 114         popd >/dev/null || exit 2
 115 }
 116 
 117 ### MAIN
 118 for PROJECT in "${@}"; do
 119         if ! egrep '\w+/\w+' <<< "$PROJECT"; then
 120                 echo "Project \"$PROJECT\" does not seem to be a valid."
 121                 break
 122         fi
 123 
 124         PROJECT_BASENAME="$(basename "$PROJECT")"
 125 
 126         release_info "$PROJECT"
 127         $GET_VERSION && echo "$VERSION"
 128         if $GET_ASSETS || $GET_TARBALL || $GET_ZIPBALL; then
 129                 release_download "$PROJECT" "$VERSION"
 130         fi
 131 done

Compute hashes

Linux

Vaious hashes can be calculated.

mkpasswd -m help

   1 Verfügbare Methoden:
   2 yescrypt        Yescrypt
   3 gost-yescrypt   GOST Yescrypt
   4 scrypt          scrypt
   5 bcrypt          bcrypt
   6 bcrypt-a        bcrypt (obsolete $2a$ version)
   7 sha512crypt     SHA-512
   8 sha256crypt     SHA-256
   9 sunmd5          SunMD5
  10 md5crypt        MD5
  11 bsdicrypt       BSDI extended DES-based crypt(3)
  12 descrypt        standard 56 bit DES-based crypt(3)
  13 nt              NT-Hash

Debian standard is SHA512 and rounds 5000 in /etc/login.defs

   1 mkpasswd -m sha512crypt

Linux 6

/usr/local/bin/linux6.sh

   1 #!/bin/bash
   2 cat <<-EOF
   3 Generate a salted sha256 password algorithm for usage in "/etc/shadow".
   4 
   5 EOF
   6 
   7 read -sp"Enter password:  " PASSWORD
   8 echo
   9 read -sp"Repeat password: " CONFIRM
  10 echo
  11 
  12 if [ "$PASSWORD" = "$CONFIRM" ]; then
  13         RESULT="$(cracklib-check <<< "$PASSWORD" \
  14                 | cut -f2 -d\: \
  15                 | sed -r 's/^\s+//')"
  16 
  17         if [ "$RESULT" = "OK" ]; then
  18                 ### REDHAT CODE SNIPPET
  19                 python -c "import crypt; print crypt.crypt('$PASSWORD')"
  20         else
  21                 echo -e "$RESULT.\nExiting …"
  22         fi
  23 else
  24         echo -e "Passworts do not match.\nExiting …"
  25 fi

Cisco

Works great
github.com BrettVerney/ciscoPWDhasher

Cisco 5

/usr/local/bin/cisco5.sh

   1 #!/bin/bash
   2 cat <<-EOF
   3 Generate a salted MD5 based BSD password algorithm a.k.a. "cisco5".
   4 Please see "man openssl-password".
   5 
   6 EOF
   7 
   8 read -sp"Enter password:  " PASSWORD
   9 echo
  10 read -sp"Repeat password: " CONFIRM
  11 echo
  12 
  13 if [ "$PASSWORD" = "$CONFIRM" ]; then
  14         RESULT="$(cracklib-check <<< "$PASSWORD" \
  15                 | cut -f2 -d\: \
  16                 | sed -r 's/^\s+//')"
  17         
  18         if [ "$RESULT" = "OK" ]; then
  19                 SALT=$(pwgen 4)
  20 
  21                 ### SEE man openssl-passwd
  22                 openssl passwd -1 -salt "$SALT" "$PASSWORD"
  23         else
  24                 echo -e "$RESULT.\nExiting …"
  25         fi
  26 else
  27         echo -e "Passwords do not match.\nExiting …"
  28 fi

Cisco 8

Install dependencies

   1 apt install libcrypt-pbkdf2-perl

/usr/local/bin/cisco8.pl

   1 #!/usr/bin/env perl
   2 
   3 # Author: philsmd
   4 # released to the public domain (Nov 2014)
   5 # credits for the code, this public contribution and discovery
   6 # of the algorithm go to author and hashcat.net (@hashcat)
   7 
   8 # hashcat mode:
   9 # -m 9200 = Cisco $8$
  10 # PBKDF2-SHA256 based
  11 
  12 use strict;
  13 use warnings;
  14 
  15 use Crypt::PBKDF2;
  16 use MIME::Base64;
  17 
  18 # Example1
  19 
  20 my $password = "hashcat";
  21 my $salt     = "TnGX";
  22 
  23 # Example2 (args from command line)
  24 
  25 if (scalar (@ARGV) > 0)
  26 {
  27   $password = $ARGV[0];
  28 }
  29 
  30 if (scalar (@ARGV) > 1)
  31 {
  32   $salt = $ARGV[1];
  33 }
  34 
  35 # fixed PBKDF2 settings
  36 
  37 my $iterations = 20000;
  38 
  39 # base64 table: ./0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz
  40 
  41 my $BASE64_TRANS_CISCO = {'A', '.', 'B', '/', 'C', '0', 'D', '1', 'E', '2', 'F', '3', 'G', '4', 'H', '5', 'I', '6', 'J', '7', 'K', '8', 'L', '9', 'M', 'A', 'N', 'B', 'O', 'C', 'P', 'D', 'Q', 'E', 'R', 'F', 'S', 'G', 'T', 'H', 'U', 'I', 'V', 'J', 'W', 'K', 'X', 'L', 'Y', 'M', 'Z', 'N', 'a', 'O', 'b', 'P', 'c', 'Q', 'd', 'R', 'e', 'S', 'f', 'T', 'g', 'U', 'h', 'V', 'i', 'W', 'j', 'X', 'k', 'Y', 'l', 'Z', 'm', 'a', 'n', 'b', 'o', 'c', 'p', 'd', 'q', 'e', 'r', 'f', 's', 'g', 't', 'h', 'u', 'i', 'v', 'j', 'w', 'k', 'x', 'l', 'y', 'm', 'z', 'n', '0', 'o', '1', 'p', '2', 'q', '3', 'r', '4', 's', '5', 't', '6', 'u', '7', 'v', '8', 'w', '9', 'x', '+', 'y', '/', 'z'};
  42 
  43 
  44 # START of algo here:
  45 
  46 my $pbkdf2 = Crypt::PBKDF2->new
  47 (
  48   hasher     => Crypt::PBKDF2->hasher_from_algorithm ('HMACSHA2', 256),
  49   iterations => $iterations
  50 );
  51 
  52 my $hash_base64 = $pbkdf2->PBKDF2_base64 ($salt, $password);
  53 
  54 # crappy way to convert it to the cisco base64 table (but who cares?)
  55 
  56 my $hash = "";
  57 
  58 for (my $i = 0; $i < 43; $i++)
  59 {
  60   $hash .= $BASE64_TRANS_CISCO->{substr ($hash_base64, $i, 1)};
  61 }
  62 
  63 # just print it
  64 
  65 my $output = sprintf ("\$8\$%s\$%s", $salt, $hash);
  66 
  67 print $output . "\n";

Cisco 9

/usr/local/bin/cisco9.pl

   1 #!/usr/bin/env perl
   2 
   3 # Author: philsmd
   4 # released to the public domain (Nov 2014)
   5 # credits for the code, this public contribution and discovery
   6 # of the algorithm go to author and hashcat.net (@hashcat)
   7 
   8 # hashcat mode:
   9 # -m 9300 = Cisco $9$
  10 # scrypt based
  11 
  12 use strict;
  13 use warnings;
  14 use MIME::Base64;
  15 use Crypt::ScryptKDF qw (scrypt_b64);
  16 
  17 # Example1
  18 
  19 my $password = "hashcat";
  20 my $salt     = "2MJB";
  21 
  22 # Example2 (args from command line)
  23 
  24 if (scalar (@ARGV) > 0)
  25 {
  26   $password = $ARGV[0];
  27 }
  28 
  29 if (scalar (@ARGV) > 1)
  30 {
  31   $salt = $ARGV[1];
  32 }
  33 
  34 # default scrypt settings for -m 9300 = Cisco $9$
  35 
  36 my $N = 16384;
  37 my $r = 1;
  38 my $p = 1;
  39 
  40 # base64 table: ./0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz
  41 
  42 my $BASE64_TRANS_CISCO = {'A', '.', 'B', '/', 'C', '0', 'D', '1', 'E', '2', 'F', '3', 'G', '4', 'H', '5', 'I', '6', 'J', '7', 'K', '8', 'L', '9', 'M', 'A', 'N', 'B', 'O', 'C', 'P', 'D', 'Q', 'E', 'R', 'F', 'S', 'G', 'T', 'H', 'U', 'I', 'V', 'J', 'W', 'K', 'X', 'L', 'Y', 'M', 'Z', 'N', 'a', 'O', 'b', 'P', 'c', 'Q', 'd', 'R', 'e', 'S', 'f', 'T', 'g', 'U', 'h', 'V', 'i', 'W', 'j', 'X', 'k', 'Y', 'l', 'Z', 'm', 'a', 'n', 'b', 'o', 'c', 'p', 'd', 'q', 'e', 'r', 'f', 's', 'g', 't', 'h', 'u', 'i', 'v', 'j', 'w', 'k', 'x', 'l', 'y', 'm', 'z', 'n', '0', 'o', '1', 'p', '2', 'q', '3', 'r', '4', 's', '5', 't', '6', 'u', '7', 'v', '8', 'w', '9', 'x', '+', 'y', '/', 'z'};
  43 
  44 # START of algo here:
  45 
  46 my $hash_base64 = scrypt_b64 ($password, $salt, $N, $r, $p, 32);
  47 
  48 # crappy way to convert it to the cisco base64 table (but who cares?)
  49 
  50 my $hash = "";
  51 
  52 for (my $i = 0; $i < 43; $i++)
  53 {
  54   $hash .= $BASE64_TRANS_CISCO->{substr ($hash_base64, $i, 1)};
  55 }
  56 
  57 # just print it
  58 
  59 my $output = sprintf ("\$9\$%s\$%s", $salt, $hash);
  60 
  61 print $output . "\n";

Decrypt and encrypt RFC3394 - Watchguard

IETF RFC3394 Advanced Encryption Standard (AES) Key Wrap Algorithm
IETF RFC5649 Advanced Encryption Standard (AES) Key Wrap with Padding Algorithm
- Updates RFC3394
GitHub kurtbrose key_wrap.py
- Thanks for the library!
https://serverfault.com/questions/790339/merge-vpns-of-two-watchguard-firewalls-into-one-firewall/841942#841942
- Thanks for the Watchguard KeyEncryptionKey (KEK).
GitHub ikluft/AESKeyWrap

Watchguard uses this algorithm to obfuscate the passwords in the dumped configs. So if you want to recover your IPsec-S2S password …

Prepare

   1 apt install python-crypto
   2 mkdir key_wrap
   3 cd key_wrap
   4 wget 'https://gist.githubusercontent.com/kurtbrose/4243633/raw/526cd18f2c0f57a5c897364ddbff725b49c7b17b/key_wrap.py'

Decryption

   1 % python  
   2 Python 2.7.18 (default, Apr 20 2020, 20:30:41) 
   3 [GCC 9.3.0] on linux2
   4 Type "help", "copyright", "credits" or "license" for more information.
   5 >>> import key_wrap
   6 >>> key_wrap.test()
   7 >>> KEK = binascii.unhexlify("1d03f58287982bc701227394e498de23") ### WATCHGUARD KeyEncryptionKey (KEK)
   8 >>> KEK = binascii.unhexlify("000102030405060708090A0B0C0D0E0F") ### EXAMPLE KEK
   9 >>> CIPHER = binascii.unhexlify("1FA68B0A8112B447AEF34BD8FB5A7B829D3E862371D2CFE5")
  10 >>> key_wrap.aes_unwrap_key_and_iv(KEK, CIPHER)
  11 >>> ('\x00\x11"3DUfw\x88\x99\xaa\xbb\xcc\xdd\xee\xff', 12008468691120727718L)

This is the same as hex 00112233445566778899AABBCCDDEEFF.

You may notice paddings at the end like

\x02\x024 2 times 2 or
\x04\x04\x04\x04 4 times 4 or
\x08\x08\x08\x08\x08\x08\x08\x08 8 times 8.

There might be a alternative way using openssl. Schlüssel mit OpenSSL unter Linux verpacken

Java Console over ssh

/usr/local/bin/jc.sh

   1 #!/bin/bash
   2 
   3 function jc {
   4   # set this to the host you'll proxy through.
   5   host=$1
   6   jmxport=${2:-9000}
   7   proxy_port=${3:-8123}
   8  
   9   if [ "x$host" = "x" ]; then
  10   echo "Usage: jc [host] [jmx port]  [proxy port]"
  11   return 1
  12   fi
  13  
  14   # start up a background ssh tunnel on the desired port
  15   ssh -N -f -D$proxy_port $host
  16  
  17   # if the tunnel failed to come up, fail gracefully.
  18   if [ $? -ne 0 ]; then
  19   echo "Ssh tunnel failed"
  20   return 1
  21   fi
  22  
  23   ssh_pid=`ps awwwx | grep "[s]sh -N -f -D$proxy_port" | awk '{print $1}'`
  24   echo "ssh pid = $ssh_pid"
  25  
  26   # Fire up jconsole to your remote host
  27   jconsole -J-DsocksProxyHost=localhost -J-DsocksProxyPort=$proxy_port \
  28   service:jmx:rmi:///jndi/rmi://${host}:${jmxport}/jmxrmi
  29  
  30   # tear down the tunnel
  31   kill $ssh_pid
  32 }

video

Concat two files with ffmpeg

The parameter -i is fed by a process substitution to avoid creating a file that lists the inputs everytime. (That's why the paths must by absolute and -safe 0)

   1 ffmpeg -f concat \
   2         -safe 0 -i <(for FILE in *.avi; do echo "file '$PWD/$FILE'"; done) \
   3         -c copy \
   4         output_file.avi

Devide a file in parts of size

CD-ROM

max. extends on CD: 359849
physical block size on CD: 2048bit

Prelude> 359849 * 2048 / 2^20
702.830078125

Based on GPAC MP4Box

   1 apt install gpap
   2 ### TRANSCODE TO MP4
   3 MP4Box -a file.avi file.mp4
   4 ### SPLIT TO FIT CD (BUT LEAVE 1MiB MARGIN)
   5 MP4Box -split-size "$((359849 * 2 - 1024 ))" file.mp4

Based on ffmpeg

Based on a solution of https://stackoverflow.com/a/52158160/3018288|LukeLR on StackOverflow

split_video.sh

   1 #!/bin/bash
   2 # Short script to split videos by filesize using ffmpeg by LukeLR
   3 
   4 ### INITIALZE
   5 SELF="$(basename "$0")"
   6 DURATION_INPUT=0
   7 DURATION_CUR=0
   8 DURATION_NEW=0
   9 PART=1
  10 
  11 usage () {
  12         cat <<-EOL
  13         Illegal number of parameters. Needs 3 parameters:
  14         Usage:
  15           $SELF FILE_INPUT SIZELIMIT "FFMPEG_ARGS"
  16 
  17         Parameters:
  18             - FILE_INPUT:        Name of the video file to split
  19             - SIZELIMIT:   Maximum file size of each part (in bytes)
  20             - FFMPEG_ARGS: Additional arguments to pass to each ffmpeg-call
  21                            (video format and quality options etc.)
  22         EOL
  23 }
  24 
  25 if [ $# -ne 3 ]; then
  26         usage
  27         exit 1
  28 fi
  29 
  30 FILE_INPUT="$1"
  31 SIZELIMIT="$2"
  32 FFMPEG_ARGS="$3"
  33 
  34 # Filename of the source video (without extension)
  35 BASENAME="${FILE_INPUT%.*}"
  36 
  37 # Extension for the video parts
  38 EXTENSION="${FILE_INPUT##*.}"
  39 #EXTENSION="mp4"
  40 
  41 # Duration of the source video
  42 DURATION_INPUT="$(ffprobe \
  43         -i "$FILE_INPUT" -show_entries format=duration \
  44         -v quiet -of default=noprint_wrappers=1:nokey=1 \
  45         |cut -d. -f1)"
  46 
  47 echo "Duration of input: $DURATION_INPUT"
  48 
  49 # Until the duration of all partial videos has reached the duration of the source video
  50 while [ "$DURATION_CUR" -lt "$DURATION_INPUT" ]; do
  51         # Filename of the current video part
  52         FILE_CUR="$BASENAME-$PART.$EXTENSION"
  53 
  54         # Encode next part
  55         COMMAND="ffmpeg -ss '$DURATION_CUR' \
  56                         -i '$FILE_INPUT' \
  57                         -fs '$SIZELIMIT' $FFMPEG_ARGS \
  58                         '$FILE_CUR'"
  59         echo "$COMMAND"
  60         eval "$COMMAND"
  61 
  62         # Duration of the new part
  63         DURATION_NEW="$(ffprobe -i "$FILE_CUR" \
  64                 -show_entries format=duration -v quiet \
  65                 -of default=noprint_wrappers=1:nokey=1\
  66                 |cut -d. -f1)"
  67 
  68         # Total duration encoded so far
  69         DURATION_CUR="$((DURATION_CUR + DURATION_NEW))"
  70 
  71         PART="$((++PART))"
  72 
  73         echo "Duration of '$FILE_CUR': $DURATION_NEW"
  74         echo "Part No. $PART starts at: $DURATION_CUR"
  75 
  76 done

Use it like:

   1 ### TRY to COMPENSATE MUXING OVERHEAD (~0.9%)
   2 SIZE_MAX="$(printf %.0f $((0.991 * 359849 * 2048)))"
   3 ./split_video.sh \
   4         big_file.mp4 \
   5         "$SIZE_MAX" \
   6         "-c copy"
   7 ### OR
   8 ./split_video.sh \
   9         big_file.mp4 \
  10         "$SIZE_MAX" \
  11         "-c:v libx264 -crf 23 -c:a copy -vf scale=960:-1"

Transcode a video with mystiq

It's self explaining …

   1 apt install mystiq

Transcode a video with HandBrakeCLI

/usr/local/bin/transcode.sh

   1 #!/bin/bash
   2 
   3 FILE_INPUT="$1"
   4 FILE_OUTPUT="${FILE_INPUT%.avi}"
   5 
   6 HandBrakeCLI \
   7         --scan \
   8         -i "$FILE_INPUT"
   9 
  10 HandBrakeCLI \
  11         -i "$FILE_INPUT" \
  12         -o "$FILE_OUTPUT" \
  13         -f av_mp4 \
  14         --optimize \
  15         --vb 1300 \
  16         --two-pass \
  17         --all-audio \
  18         --aencoder mp3 \
  19         --comb-detect \
  20         --deinterlace \
  21         --decomb \
  22         --deblock=strong \
  23         --all-subtitles

avi to mp4

4 processes converting videos

   1 FILES="$(find . -type f -name '*.avi' \
   2         |sed -r 's/\.avi$//' \
   3         |sort)"
   4 <<< $FILES xargs -P4 -L1 -t -d'\n' -I{} -- \
   5         ffmpeg -i "{}.avi" -c:v copy -c:a copy -y "{}.mp4"

Copy mp4 files recursively to new directory

   1 find . -type f -name '*.mp4' -print0 \
   2         |sort -z \
   3         |cpio -pmd0 ../new_dir

Delete mp4 in source directory

   1 #find . -type f -name '*.mp4' -delete
   2

videos to gif

   1 ffmpeg -ss 30 -t 3 -i input.mp4 \
   2         -vf "fps=10,scale=320:-1:flags=lanczos,split[s0][s1];[s0]palettegen[p];[s1][p]paletteuse" \
   3         -loop 0 output.gif

   1 ffmpeg -i input.mp4 -vf "fps=10,scale=320:-1:flags=lanczos" \
   2         -c:v pam -f image2pipe - \
   3         |convert -delay 10 - -loop 0 -layers optimize output.gif

Reduce size of jpeg-images

   1 DIR="reduced"
   2 [ -d "$DIR" ] || mkdir "$DIR"
   3 find . -maxdepth 1 -regextype posix-extended \
   4         -iregex '.*\.jpg$' \
   5         |xargs -I{} -P $(nproc) convert -quality 25 {} "$DIR"/{}
   6 ls -lh "$DIR"

Or just use darktable.

Strip metadata from images

mogrify (ImageMagick) or jhead can do this

   1 ### SIMPLE
   2 mogrify -strip ./.*
   3 ### RECURSIVELY
   4 find . -maxdepth 1 -regextype posix-extended \
   5         -iregex '.*\.jpg$' \
   6         |xargs -I{} -P $(nproc) mogrify -strip {}

Images to gif

If you have very much images it may be necessary to adjust the image-magick memory-limit in
/etc/ImageMagick-6/policy.xml

   1     <!-- <policy domain="resource" name="memory" value="1024MiB"/> -->
   2     <policy domain="resource" name="memory" value="16384MiB"/>

Convert the image to a gif

   1 convert -delay 8 -loop 0 $(ls *.jpg) output.gif
   2 convert -delay 8 -loop 0 $(ls *.jpg) -coalesce -fuzz 8% +dither -remap "$(ls -1 *.jpg |head -1)" -layers Optimize output.gif

Images to webme

Make sure your pictures are in the right order (leading zero) and convert the series using ffmpeg.

   1 ffmpeg -f image2 -pattern_type glob -framerate 16 \
   2         -i 'IMG_20220819_201154_*CS.jpg' output.webm

Play DVD and BLURAY

Just for documentation purposes, no key material in this script. Yet, untested.

allow_bluray.sh

   1 #!/bin/bash
   2 
   3 # HOST_CERT CAN BE ACQUIRED FROM
   4 # https://forum.doom9.org/showpost.php?p=1883655&postcount=3
   5 # AND IS STORED IN HOST_CERT.cfg
   6 
   7 # KEYDB.cfg (MIND THE CASE) CAN BE ACQUIRED FROM
   8 # http://fvonline-db.bplaced.net/
   9 # http://fvonline-db.bplaced.net/fv_download.php?lang=eng
  10 # http://fvonline-db.bplaced.net/fv_download.php?lang=deu
  11 
  12 ### INITIALIZE
  13 DATE="$(date +%F)"
  14 TIME="$(date +%T)"
  15 LANG=deu
  16 LANG=eng
  17 URL_KEYDB_BASE='http://fvonline-db.bplaced.net/fv_download.php'
  18 URL_KEYDB="$URL_KEYDB_BASE?lang=$LANG"
  19 PATH_AACS="$HOME/.config/aacs"
  20 FILE_KEYDB="keydb_${DATE}_${LANG}.cfg"
  21 FILE_KEYDB_ZIP="$KEYDB.zip"
  22 KEYDB="KEYDB.cfg"
  23 HOST_CERT="HOST_CERT.cfg"
  24 VERBOSE="-v"
  25 
  26 cat <<-EOF
  27 ### SOME ADVISE TO READ ENCRYPTED MEDIA
  28 
  29 ### TO READ ENCRYPTED DVDS
  30 # PLEASE INSTALL libdvd-pkg TO DOWNLOAD AND BUILD libdvdcss
  31 
  32 apt install libdvd-pkg
  33 
  34 dpkg-reconfigure libdvd-pkg
  35 
  36 ### TO READ ENCRYPTED BLURAYS
  37 # PLEASE INSTALL
  38 libbdplus0      (https://www.videolan.org/developers/libbdplus.html)
  39 libbluray2      (https://www.videolan.org/developers/libbluray.html)
  40 libaacs         (https://www.videolan.org/developers/libaacs.html)
  41 
  42 apt install libbdplus0 libbluray2 libaacs
  43 
  44 ### LIBAACS STORES THE "$KEYDB" IN "$PATH_AACS"
  45 ### ON CASE SENSITIVE FILESYSTEMS THIS FILE IS NAMED "$KEYDB"
  46 
  47 EOF
  48 
  49 ### PREPARE
  50 echo "Configuring libaacs…"
  51 [ -d "$PATH_AACS" ] || mkdir $VERBOSE -p "$PATH_AACS"
  52 
  53 cd "$PATH_AACS" || exit 1
  54 
  55 ### MOVE OLD KEYDB TO THE SIDE
  56 if [ -L "$KEYDB" ]; then
  57         unlink "$KEYDB";
  58 else
  59         mv $VERBOSE "$KEYDB" "${KEYDB}_OLD_${DATE}_{$TIME}"
  60 fi
  61 
  62 ### DOWNLOAD AND EXTRACT NEW KEYDB.cfg
  63 wget -O "$PATH_AACS/$FILE_KEYDB_ZIP" "$URL_KEYDB"
  64 unzip "$FILE_KEYDB_ZIP"
  65 cat "$HOST_CERT" keydb.cfg > "$FILE_KEYDB"
  66 ln $VERBOSE -s "$FILE_KEYDB" "$KEYDB"
  67 
  68 ### CLEANUP
  69 rm $VERBOSE keydb.cfg

Create m3u playlist

   1 ls -1 > "$(basename "$PWD").m3u"

non-terminal chars in filenames

Print them

   1 printf %s\\0\\n * | sed -n l

Get and delete file by inode, but don't cross fs boundaries

   1 ls -i
   2 find . -xdev -inum $INODE -delete

Create index directory structure

Index by first letter in lower case

   1 DIR_SRC="/tmp/Projekt Gutenberg E15/Edition15"
   2 DIR_DST="/tmp/indexed"
   3 [ -d "$DIR_DST" ] || mkdir -v "$DIR_DST"
   4 for DIR in $DIR_SRC/*; do
   5         DIR_BASENAME="$(basename "$DIR"|tr A-Z a-z)"
   6         DIR_INDEX="${DIR_BASENAME:0:1}"
   7         [ -d "$DIR_DST/$DIR_INDEX" ] || mkdir -v "$DIR_DST/$DIR_INDEX"
   8         cp -rv "$DIR" "$DIR_DST/$DIR_INDEX/$DIR_BASENAME"
   9 done

Transform ASCII files to wiki markup

   1 for FILE in $(find package -type f); do
   2         if file "$FILE" |grep -qE 'ASCII text'; then
   3                 echo -e "{{{$FILE}}}"
   4                 echo '{{{#!highlight bash'
   5                 cat "$FILE"
   6                 echo '}}}'
   7         fi
   8 done

Delete duplicate files if their hashes match

   1 for FILE in *Kopie*; do
   2         BASE="$(sed -r 's/(.+)( - Kopie)\.(.+)/\1/'  <<< "$FILE")"
   3         EXT="$(sed -r 's/(.+)( - Kopie)\.(.+)/\3/'  <<< "$FILE")"
   4         FILE_BASE="$BASE.$EXT"
   5         SUM="$(sha1sum "$FILE" |cut -f1 -d\ )"
   6         SUM_BASE="$(sha1sum "$FILE_BASE" |cut -f1 -d\ )"
   7 
   8         echo
   9         echo "$FILE $FILE_BASE"
  10         echo "$SUM" "$SUM_BASE"
  11 
  12         if [ "$SUM" = "$SUM_BASE" ]; then
  13                 echo match
  14                 rm -v "$FILE"
  15         else
  16                 echo mismatch
  17         fi
  18 done

Random Sleep

Max sleep time is 3600s -> 1h

Concat the command with &&

   1 ### PERL
   2 perl -e 'sleep int(rand(3600))'
   3 ### PYTHON
   4 python -c 'import random; import time; time.sleep(random.random() * 3600)'
   5 ###SHELL
   6 sleep "$(((RANDOM+1.0)/32768*3600))"

Telnet Star Wars

   1 telnet towel.blinkenlights.nl

Connection Test

Upload, Download, Upload, … a nearly incompressible payload

   1 #!/bin/bash
   2 UNAME="remote_user"
   3 RHOST="remote-host"
   4 PAYLOAD="random_file.1GiB.dd"
   5 dd if=/dev/urandom of="random_file.1GiB.dd" bs=1m count=1024
   6 for ((i=0;i<10;i++)); do
   7         if [ $(( $i % 2 )) -eq 0 ]; then
   8                 scp -l $((6*8*2**10)) "$UNAME"@"$RHOST":$PAYLOAD .
   9         else
  10                 scp -l $((6*8*2**10)) "$PAYLOAD" "$UNAME"@"$RHOST":
  11         fi
  12         [ $? -eq 0 ] || break
  13 done

Diff two configs by key

Quite useful tools to compare two configs

diff_by_key.sh

   1 #!/bin/bash
   2 
   3 ### INIT
   4 SELF="$(basename $0)"
   5 
   6 ONE="$1"
   7 TWO="$2"
   8 
   9 ### SANITIZE
  10 
  11 if [ ! -f "$ONE" ]; then
  12         echo "First file '$ONE' is not a file. Exiting…"
  13         exit 1
  14 fi
  15 
  16 if [ ! -f "$TWO" ]; then
  17         echo "Second file '$TWO' is not a file. Exiting…"
  18         exit 1
  19 fi
  20 
  21 ### MAIN
  22 
  23 KEYS="$(grep -v '^\s*#' "$ONE" \
  24         |sed -r 's/^\s*//;s/\s*=.*$//' \
  25         |sort |uniq)"
  26 
  27 for KEY in $KEYS; do
  28         KEYRE='^\s*#*\s*'"$KEY"'(\s*=|$)'
  29         echo
  30         echo -n "1:$KEY:"
  31         grep -Eh "$KEYRE" "$ONE"
  32         echo -n "2:$KEY:"
  33         if ! grep -Eh "$KEYRE" "$TWO"; then
  34                 echo "KEY NOT FOUND."
  35         fi
  36 done

Extract debian packages

Extract configuration from package

   1 PACKAGE="jitsi-meet-turnserver"
   2 TMP_DIR="/tmp/$PACKAGE"
   3 install -m 777 -d "$TMP_DIR"
   4 cd "$TMP_DIR"
   5 apt download "$PACKAGE"
   6 ar x "$PACKAGE"*.deb
   7 tar -xf data.tar.xz
   8 ls -l -R usr

Remove all underscores from filenames

   1 for FILE in *; do
   2         if grep -q '_' <<< "$FILE";then
   3                 mv -v "$FILE" "${FILE//_/ }" 
   4         fi
   5 done

Remove regex from filenames

   1 for FILE in *; do
   2         unset DST
   3         if grep -Eq -- '-[0-9]+' <<< "$FILE";then
   4                 DST="$(sed -r 's/-[0-9]+\././' <<< "$FILE")"
   5                 mv -v "$FILE" "$DST"  
   6         fi
   7 done

Check directory permissions

Check permissions of all directories all the way up from the root to $PWD.

Bash

/usr/local/bin/check_pwd.bash

   1 #!/bin/bash
   2 echo "Checking '$PWD'"
   3 declare -a PATH_SEGS
   4 SPLIT="${PWD//\//$'/\n'}"
   5 mapfile -t PATH_SEGS <<< "$SPLIT"
   6 
   7 for ((i=0;i<${#PATH_SEGS[@]};i++));do
   8         PATH_SEG+=${PATH_SEGS[$i]}
   9         ls -ld "$PATH_SEG"
  10 done |column -t

Example output (with a problematic directory)

   1 tobias@libertas ~/sftp % check_pwd.bash
   2 Checking '/home/tobias/sftp'
   3 drwxr-xr-x  1  root    root    314   25.  May  14:21  /
   4 drwxr-xr-x  1  root    root    66    7.   Jun  16:56  /home/
   5 drwxr-xr-x  1  tobias  tobias  3814  7.   Jun  16:57  /home/tobias/
   6 drwxr-xr-x  1  root    root    6     7.   Jun  13:59  /home/tobias/sftp

Zsh

/usr/local/bin/check_pwd.zsh

   1 #!/bin/zsh
   2 echo "Checking '$PWD'"
   3 declare -a PATH_SEGS
   4 SPLIT=${PWD//\///\n}
   5 IFS="\n" read -r -A PATH_SEGS <<< "$SPLIT"
   6 
   7 for ((i=1;i<${#PATH_SEGS[@]}+1;i++));do
   8         PATH_SEG+=${PATH_SEGS[$i]}
   9         ls -ld "$PATH_SEG"
  10 done |column -t

Find all storage block sizes

   1 find /sys/class/block/*/queue \
   2         \( -name physical_block_size \
   3         -o -name logical_block_size \) \
   4         -exec grep -H "" {} \;

Convert Midi to MP3

Install timidity

   1 sudo apt-get install timidity

Convert to mp3 using lame

   1 FILE="my_file.mid"
   2 timidity "$FILE" -Ow -o - \
   3         |lame - -b 64 "${FILE%.mid}.mp3"

Convert to mp3 using ffmpeg

   1 FILE="my_file.mid"
   2 timidity "$FILE" -Ow -o - \
   3         |ffmpeg -i - -acodec libmp3lame -ab 64k \
   4                 "${FILE%.mid}.mp3"

Generate favicon from svg

evilmartians.com How to Favicon in 2021: Six files that fit most needs

favicon_gen.sh

   1 #!/bin/bash
   2 
   3 ### THIS SCRIPT IS INSPIRED BY
   4 ### https://evilmartians.com/chronicles/how-to-favicon-in-2021-six-files-that-fit-most-needs#the-ultimate-favicon-setup
   5 ### LICENSE: GPLv2+ Tobias Stein rockstable.it
   6 
   7 ### THIS SCRIPT COMES AS IS, NO FITNESS FOR A SPECIFIC PURPOSE, NO WARRANTY, NO MERCI, …
   8 
   9 ### INIT
  10 PROGRAM="${0##*/}"
  11 MANIFEST="manifest.webmanifest"
  12 TITLE="My Website"
  13 OPTIMIZE_SVG=false
  14 SIZES=(32 192 512)
  15 VERBOSITY=0
  16 VERBOSE=false
  17 QUITE=true
  18 
  19 usage () {
  20         cat <<-EOF
  21                 Usage:
  22                         $PROGRAM [OPTIONS] [--] file.svg [file1.svg […]]
  23 
  24                         Where options are: 
  25                                 [-a|--add-sizes]        Additional favicon sizes to create:
  26                                                         Example: 64,128 or "64 128"
  27                                 [-h|--help]             Print this page
  28                                 [-o|--optimize-svg]     Optimize input SVG with 'svgo'
  29                                                         to reduce the size
  30                                 [-t|--title]            Define website title (for the manifest)
  31                                 [-v|--verbose]          Print out more descriptive information
  32                                                         during the conversion
  33 
  34                         [] … optional
  35 
  36         EOF
  37 }
  38 
  39 # Note that we use "$@" to let each command-line parameter expand to a
  40 # separate word. The quotes around "$@" are essential!
  41 # We need TEMP as the 'eval set --' would nuke the return value of getopt.
  42 TEMP=$(getopt \
  43         -o 'a:hot:v' \
  44         --long 'add-sizes:,help,optimize-svg,title:,verbose' \
  45         -n "$PROGRAM" -- "$@")
  46 
  47 if [ $? -ne 0 ]; then
  48         echo 'Terminating...' >&2
  49         exit 1
  50 fi
  51 
  52 # Note the quotes around "$TEMP": they are essential!
  53 eval set -- "$TEMP"
  54 unset TEMP
  55 
  56 while true; do
  57         case "$1" in
  58                 '-a'|'--add-sizes')
  59                         SIZES+=( ${2//,/ } )
  60                         shift 2
  61                         continue
  62                 ;;
  63                 '-h'|'--help')
  64                         usage
  65                         exit 0
  66                 ;;
  67                 '-o'|'--optimize-svg')
  68                         OPTIMIZE_SVG=true
  69                         shift
  70                         continue
  71                 ;;
  72                 '-t'|'--title')
  73                         TITLE="$2"
  74                         shift 2
  75                         continue
  76                 ;;
  77                 '-v'|'--verbose')
  78                         VERBOSE=true
  79                         ((VERBOSITY++))
  80                         shift
  81                         continue
  82                 ;;
  83                 #'-c'|'--c-long')
  84                 #       # c has an optional argument. As we are in quoted mode,
  85                 #       # an empty parameter will be generated if its optional
  86                 #       # argument is not found.
  87                 #       case "$2" in
  88                 #               '')
  89                 #                       echo 'Option c, no argument'
  90                 #               ;;
  91                 #               *)
  92                 #                       echo "Option c, argument '$2'"
  93                 #               ;;
  94                 #       esac
  95                 #       shift 2
  96                 #       continue
  97                 #;;
  98                 '--')
  99                         shift
 100                         break
 101                 ;;
 102                 *)
 103                         echo 'Internal error!' >&2
 104                         exit 1
 105                 ;;
 106         esac
 107 done
 108 
 109 #echo 'Remaining arguments:'
 110 #for arg; do
 111 #       echo "--> '$arg'"
 112 #done
 113 
 114 if [ "$VERBOSITY" -ge 2 ]; then
 115         QUITE=false
 116 fi
 117 
 118 if [ "$1" ] && [ -e "$1" ]; then
 119         FAVICON="$1"
 120 else
 121         cat <<-EOF
 122                 No input file given as positional parameter.
 123                 Please use: $0 input_file.svg
 124                 Exiting…
 125         EOF
 126         exit 1
 127 fi
 128 
 129 if $OPTIMIZE_SVG; then
 130         FILE_OUT="$(basename "${FAVICON%.*}-optimized.svg")"
 131         $VERBOSE && echo -e '\n'"Optimizing input SVG"
 132         if ! npm ls svgo 1>/dev/null 2>&1; then 
 133                 npm install svgo
 134         fi
 135         SVGO_OPTION="-q"
 136         $QUITE || SVGO_OPTION=""
 137         npx svgo $SVGO_OPTION --multipass "$FAVICON" \
 138                 -o "$FILE_OUT"
 139 fi
 140 
 141 SIZES=( $(IFS=$'\n'; sort -n <<< ${SIZES[*]}; unset IFS;) )
 142 $VERBOSE && echo -e '\n'"Creating favicons of size: '${SIZES[*]}'"
 143 for SIZE in "${SIZES[@]}"; do
 144         FILE_OUT="$(basename "${FAVICON%.*}-$SIZE.png")"
 145         OUTPUT="$(inkscape "$FAVICON" \
 146                 --export-width="$SIZE" \
 147                 --export-filename="$FILE_OUT" 2>&1)"
 148         $QUITE || echo "$OUTPUT"
 149         FILES_OUT["$SIZE"]="$FILE_OUT"
 150 done
 151 
 152 for ICON in "$(basename "${FAVICON%.*}")"*.png; do
 153         FAVICON_SIZE="$(echo "${ICON##*/}" \
 154                 |sed -r 's/.+-([0-9]+)\.png/\1/')"
 155         if [ "$FAVICON_SIZE" -le 64 ]; then 
 156                 FAVICON_BUNDLE+=( "$ICON" )
 157         fi
 158 done
 159 
 160 $VERBOSE && cat <<-EOF
 161 
 162 Hint:
 163 You may put specially crafted files with
 164 name "${FAVICON%.*}-SIZE.png" into this directory and
 165 they will get included into the favicon.ico as a layer.
 166 EOF
 167 
 168 FAVICON_BUNDLE=( $(IFS=$'\n'; sort -n <<< ${FAVICON_BUNDLE[*]}; unset IFS;) )
 169 $VERBOSE && cat <<-EOF
 170 
 171         Bundling the following files to favicon.ico:
 172         $(fold -s <<< ${FAVICON_BUNDLE[*]})
 173 EOF
 174 
 175 convert "${FAVICON_BUNDLE[@]}" favicon.ico
 176 
 177 $VERBOSE && echo -e '\n'"Creating Apple touch icon"
 178 OUTPUT="$(\
 179         inkscape "$FAVICON" \
 180                 --export-width="140" \
 181                 --export-type=png \
 182                 --export-filename=tmp.png 2>&1; \
 183         convert tmp.png \
 184                 -colorspace sRGB \
 185                 -bordercolor white \
 186                 -border 20 \
 187                 "apple-touch-icon.png" \
 188                 2>&1; \
 189         rm tmp.png 2>&1;
 190         )"
 191 $QUITE || echo "$OUTPUT"
 192 
 193 $VERBOSE && echo -e '\n'"Generating webmanifest 'manifest.webmanifest'"
 194 cat > "$MANIFEST" <<EOF
 195 {
 196   "name": "$TITLE",
 197   "icons": [
 198 EOF
 199 for SIZE in "${SIZES[@]}"; do
 200         if [ "$SIZE" -ge "192" ]; then
 201         echo '    { "src": "'"${FILES_OUT["$SIZE"]}"'", "type": "image/png", "sizes": "'"$SIZE"x"$SIZE"'" },' \
 202                 >> "$MANIFEST"
 203         fi
 204 done
 205 cat >> "$MANIFEST" <<EOF
 206   ]
 207 }
 208 EOF
 209 
 210 $VERBOSE && echo -e '\n'"Include the html-snippet 'favicon.html' into your website."
 211 cat > favicon.html <<-EOF
 212         <title>$TITLE</title>
 213         <link rel="manifest" href="/$MANIFEST">
 214         <link rel="icon" href="/favicon.ico" sizes="any">
 215         <link rel="icon" href="/${FAVICON##*/}" type="image/svg+xml">
 216         <link rel="apple-touch-icon" href="/apple-touch-icon.png">
 217 EOF

Make it executeable and run it

   1 chmod +x ./favicon_gen.sh
   2 ./favicon_gen.sh rockstable-logo.svg -t "rockstable.it" -o -v

ls -l *.png *.svg *.html *.webmanifest

   1 -rw-r--r-- 1 tobias tobias 10137 16. Jan 20:29 apple-touch-icon.png
   2 -rw-r--r-- 1 tobias tobias   259 16. Jan 20:29 favicon.html
   3 -rw-r--r-- 1 tobias tobias   220 16. Jan 20:29 manifest.webmanifest
   4 -rw-r--r-- 1 tobias tobias  1444 16. Jan 20:28 rockstable-logo-01.svg
   5 -rw-r--r-- 1 tobias tobias  2150 16. Jan 20:29 rockstable-logo-01-32.png
   6 -rw-r--r-- 1 tobias tobias 13552 16. Jan 20:29 rockstable-logo-01-192.png
   7 -rw-r--r-- 1 tobias tobias 39038 16. Jan 20:29 rockstable-logo-01-512.png

Parallelize shell functions with xargs

This is a blue print to write shell functions with positional parameters that can be parallelized in sub-shells.

xargs_parallel_funtion.sh

   1 #!/bin/bash
   2 
   3 SEARCH_DIR="$1"
   4 
   5 THREADS_MAX="$(nproc)"
   6 THREAD_RATIO=0.8
   7 THREADS=$(bc <<< "scale=0;$THREADS_MAX * $THREAD_RATIO / 1" )
   8 export THREADS_MAX THREAD_RATIO THREADS
   9 #echo "Threads: $THREADS"
  10 
  11 func_name () {
  12         local FILE="$1"
  13         echo "FILE: '$FILE'"
  14 }
  15 
  16 export -f func_name
  17 
  18 find "$SEARCH_DIR" \
  19         -type f \
  20         -name "*" \
  21         -print0 \
  22         |xargs -0 -P "$THREADS" -I{} -- \
  23                 bash -c 'func_name "$@"' _ "{}"

Change files in place in parallel

This may be relevant for large sets of large files otherwise a simple serial for loop may be sufficient.

POC command

   1 seq 1 1 10 \
   2         |xargs -I{} -P5 -- \
   3                 bash -c 'FILENO={}; \
   4                        echo "F: $FILENO" \
   5                                |tee file${FILENO}.tmp; \               
   6                        mv -v file${FILENO}.tmp file${FILENO}${FILENO}'

With a intermediate output directory

   1 export ODIR=filtered
   2 [ -d "$ODIR" ] || mkdir "$ODIR"
   3 find . -maxdepth 1 -name '*.json' \
   4         |xargs -I{} -P5 -- \
   5                 bash -c 'FILE="{}"; \
   6                         jq -f ../../filter.jq "$FILE" > "$FILE.tmp" \
   7                         && mv "$FILE.tmp" "$ODIR/$FILE"'

Or as real inplace

   1 find . -maxdepth 1 -name '*.json' \
   2         |xargs -I{} -P5 -- \
   3                 bash -c 'FILE="{}"; \
   4                         jq -f ../../filter.jq "$FILE" > "$FILE.tmp" \
   5                         && mv "$FILE.tmp" "$FILE"'

Or with GNU moreutils sponge

   1 find . -maxdepth 1 -name '*.json' \
   2         |xargs -I{} -P5 -- \
   3                 bash -c 'FILE="{}"; \
   4                         jq -f ../../filter.jq "$FILE" \
   5                                 |sponge "$FILE.tmp"'

As an example shell script that filters multiple files simultaneously with jq by their attributes
filter_export.sh

   1 #!/bin/bash
   2 
   3 if [ -n "$1" ]; then
   4         export DIR_INPUT="$1"
   5 else
   6         cat <<-EOF
   7                 Please provide an input directory as first positional parameter.
   8                 Exiting …
   9         EOF
  10         exit 1
  11 fi
  12 
  13 if [ -n "$2" ]; then
  14         export DIR_OUTPUT="$2"
  15         [ -d "$DIR_OUTPUT" ] || mkdir "$DIR_OUTPUT"
  16 else
  17         export DIR_OUTPUT="$DIR_INPUT"
  18 fi
  19 
  20 # shellcheck disable=SC2016
  21 find "$DIR_INPUT" -maxdepth 1 -name '*.json' -type f -print0 \
  22         |xargs -0 -I{} -P$(nproc) -- \
  23                 bash -c 'FILE="{}"; \
  24                         FILE="${FILE##*/}"; \
  25                         jq -f filter.jq "$DIR_INPUT/$FILE" \
  26                                 > "$DIR_OUTPUT/$FILE.tmp" \
  27                         && mv "$DIR_OUTPUT/$FILE.tmp" "$DIR_OUTPUT/$FILE"'

When using xargs with sub-shells make sure to export the desired variables to the sub-shells or their values will be empty.

filter.jq

   1 [.[]|select(.metadata.readOnly.reason != "SYSTEM")]

Send email when process quits

notify-on-exit.sh

   1 #!/bin/bash
   2 # zsh: > /dev/null 2>&1 nohup ./notify-on-exit.sh "PROCESSNAME" &!
   3 
   4 PROGRAM="${0#*/}"
   5 PNAME="$1"
   6 INTERVAL="${2:-60}"
   7 RECIPIENTS=(
   8         "your-email-address1@example.com"
   9         "your-email-address2@example.com"
  10 )
  11 
  12 ### SANTITY CHECKS
  13 if [ -z "$PNAME" ]; then
  14         echo "No program name specified. Exiting…"
  15         exit 1
  16 fi
  17 
  18 if ! grep -q '^[0-9\.]\+$' <<< "$INTERVAL"; then
  19         echo "Interval is not a number. Exiting…"
  20         exit 1
  21 fi
  22 
  23 send_email () {
  24 mail -s "Host: '$HOSTNAME' - '$PNAME' not running" -- \
  25         "${RECIPIENTS[@]}" \
  26         <<EOH
  27         Date:           '$DATE'
  28         Hostname:       '$HOSTNAME'
  29         Process name:   '$PNAME'
  30         Status:         'not running'
  31 EOH
  32 }
  33 
  34 while sleep 1; do
  35         DATE="$(date)"
  36         # shellcheck disable=SC2009
  37         PROCESS_ID="$(ps a \
  38                 | grep "$PNAME" \
  39                 | grep -v -e grep -e "$PROGRAM" \
  40                 | cut -f2 -d\ )"
  41         if [ -z "$PROCESS_ID" ]; then
  42                 send_email
  43                 break
  44         fi
  45         [ "$INTERVAL" -gt 1 ] \
  46                 && sleep "$(( INTERVAL - 1 ))"
  47 done

Make executable:
chmod u+x notify-on-exit.sh

Create full system backup

Test size of resulting archive

   1 tar --one-file-system --acls --xattrs -cf - / /boot/efi \
   2         | pigz -c \
   3         | pv > /dev/null

Create, compress and instantly write archive to backup destination.

   1 tar --one-file-system --acls --xattrs -cf - / /boot/efi \
   2         | pigz -c \
   3         | pv > >(ssh user@host 'cat > /path/to/file.tar.gz')

Or using dd

   1 dd if=/dev/sda conv=sparse status=progress \
   2         | pigz -c \
   3         | pv > >(ssh user@host 'cat > /path/to/disk.raw.gz')