Migrate a DigitalOcean Droplet to exe.dev

Create the VM#

On your computer, create a VM with room for everything on the droplet: its disk use (df -h there, volumes included) and 10 GB more. Pick another name if my-server is taken:

ssh exe.dev new --name=my-server --disk=50GB
ssh my-server.exe.xyz

Everything below runs on the VM, as exedev.

Connect to the droplet#

Make the VM a key of its own, and an SSH alias, droplet, with the droplet's address and your user in place of 203.0.113.10 and root:

mkdir -p ~/droplet && chmod 700 ~/droplet
[ -f ~/.ssh/droplet ] || ssh-keygen -q -t ed25519 -N '' -C "droplet-move@$(hostname)" -f ~/.ssh/droplet
cat >> ~/.ssh/config <<EOF

Host droplet
  HostName 203.0.113.10
  User root
  IdentityFile $HOME/.ssh/droplet
  IdentitiesOnly yes
  StrictHostKeyChecking accept-new
  UserKnownHostsFile $HOME/.ssh/known_hosts
  ServerAliveInterval 30
EOF
chmod 600 ~/.ssh/config
cat ~/.ssh/droplet.pub

Now let the VM in from your computer, or paste the key into ~/.ssh/authorized_keys on the droplet from DigitalOcean's Droplet Console. Check that the VM gets root there:

ssh droplet sudo -n true && echo connected

Expect connected.

Survey the droplet#

This script writes down what a server runs, and changes nothing:

cat > ~/droplet/survey.sh <<'EOF'
#!/bin/bash
# survey.sh: write down what a server runs. It only reads. Run it as root:
#   ssh droplet sudo -n bash -s < ~/droplet/survey.sh > ~/droplet/survey.txt
# Each section starts with "## ". Secrets stay out: it names env files and
# users, but never prints their values.
export LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
cd /
section() { printf '\n## %s\n' "$1"; }
have() { command -v "$1" >/dev/null 2>&1; }
clean() { grep -Ev '^[[:space:]]*(#|$)' "$@" 2>/dev/null; }
homes() { ls -d /root /home/* 2>/dev/null; }
apps="/var/www /srv /opt /home /root /usr/local"

section system
. /etc/os-release
echo "os: $PRETTY_NAME"
echo "arch: $(uname -m)"
echo "hostname: $(hostname)"
echo "timezone: $(timedatectl show -p Timezone --value 2>/dev/null || cat /etc/timezone)"
echo "locale: $(clean /etc/default/locale | tr '\n' ' ')"
echo "locales: $(locale -a 2>/dev/null | tr '\n' ' ')"
echo "cpus: $(nproc)"
free -m | awk '/^Mem:/ {print "memory: " $2 " MB"} /^Swap:/ {print "swap: " $2 " MB"}'
df -hT -x tmpfs -x devtmpfs -x squashfs -x overlay -x efivarfs 2>/dev/null
echo "fstab:"
clean /etc/fstab | sed 's/^/  /'

section digitalocean
md=http://169.254.169.254/metadata/v1
for p in id hostname region interfaces/public/0/ipv4/address interfaces/public/0/ipv6/address \
  floating_ip/ipv4/ip_address reserved_ip/ipv4/ip_address tags/; do
  v=$(curl -fs -m 3 "$md/$p" 2>/dev/null | tr '\n' ' ')
  [ -n "$v" ] && echo "$p: $v"
done
echo "volumes: $(ls /dev/disk/by-id 2>/dev/null | grep -o 'DO_Volume_.*' | grep -v -- '-part' | tr '\n' ' ')"
dpkg-query -W -f='${Package} ${Version}\n' do-agent droplet-agent 2>/dev/null
echo "DigitalOcean services in app config:"
find $apps /etc -xdev -type f -size -1M -not -path '*/node_modules/*' -not -path '*/.git/*' 2>/dev/null |
  timeout 120 xargs -r grep -IlE 'ondigitalocean\.com|digitaloceanspaces\.com' 2>/dev/null | head -n 30 | sed 's/^/  /'

section disk
du -xsm $apps /mnt/* /var/lib/* /var/log /var/backups 2>/dev/null | sort -rn |
  awk '$1 >= 10 {print $1 " MB\t" $2}' | head -n 40

section users
awk -F: '$1 == "root" || ($3 >= 1000 && $3 < 65534) {print $1, "uid=" $3, "gid=" $4, "home=" $6, "shell=" $7}' /etc/passwd
echo "groups with members:"
awk -F: '$4 != "" {print "  " $1, "gid=" $3, $4}' /etc/group
for h in $(homes); do
  [ -s "$h/.ssh/authorized_keys" ] && echo "$h/.ssh/authorized_keys: $(clean "$h/.ssh/authorized_keys" | wc -l) keys"
done
echo "sudoers.d: $(ls /etc/sudoers.d 2>/dev/null | tr '\n' ' ')"
echo "lingering: $(ls /var/lib/systemd/linger 2>/dev/null | tr '\n' ' ')"

section packages
echo "apt sources added:"
for f in /etc/apt/sources.list.d/*; do
  [ -f "$f" ] && echo "  $f: $(clean "$f" | grep -Eo 'https?://[^ ]+' | sort -u | tr '\n' ' ')"
done
echo "installed by hand (apt history, oldest first):"
for f in $(ls -tr /var/log/apt/history.log* 2>/dev/null); do zcat -f "$f"; done |
  sed -n 's/^Commandline: //p' | grep -Ev 'unattended-upgr|apt(-get)? (-y )?(update|upgrade|dist-upgrade|full-upgrade)' |
  awk '!seen[$0]++' | tail -n 150 | sed 's/^/  /'
have snap && { echo "snaps:"; snap list 2>/dev/null | sed 's/^/  /'; }
echo "versions:"
for c in "nginx -v" "apache2 -v" "caddy version" "php -v" "node -v" "python3 --version" "ruby -v" \
  "go version" "java -version" "psql --version" "mysql --version" "redis-server --version" "mongod --version" \
  "docker --version" "certbot --version"; do
  have "${c%% *}" && echo "  $($c 2>&1 | head -n 1)"
done
echo "  php-fpm: $(ls /etc/php 2>/dev/null | tr '\n' ' ')"
for h in $(homes); do
  [ -d "$h/.nvm/versions/node" ] && echo "  nvm in $h: $(ls "$h/.nvm/versions/node" | tr '\n' ' ')"
done
echo "python virtualenvs:"
find $apps -xdev -maxdepth 6 -name pyvenv.cfg 2>/dev/null | head -n 30 |
  while read -r f; do echo "  ${f%/pyvenv.cfg}: $(grep -E '^(version|version_info) ' "$f" | head -n 1)"; done
echo "manually installed (apt-mark showmanual):"
apt-mark showmanual 2>/dev/null | tr '\n' ' ' | fold -s -w 100 | sed 's/^/  /'
echo
if ! have dpkg && have rpm; then
  echo "installed (rpm):"
  rpm -qa --qf '%{NAME}\n' | sort | tr '\n' ' ' | fold -s -w 100 | sed 's/^/  /'
  echo
fi

section services
echo "running: $(systemctl list-units --type=service --state=running --no-legend --plain 2>/dev/null | awk '{print $1}' | tr '\n' ' ')"
echo "enabled: $(systemctl list-unit-files --state=enabled --no-legend --plain 2>/dev/null | awk '{print $1}' | tr '\n' ' ')"
echo "failed: $(systemctl list-units --state=failed --no-legend --plain 2>/dev/null | awk '{print $1}' | tr '\n' ' ')"
echo "units added by hand:"
for f in /etc/systemd/system/*.service /etc/systemd/system/*.timer /etc/systemd/system/*.socket \
  /etc/systemd/system/*.path /etc/systemd/system/*.mount /etc/systemd/system/*.d; do
  [ -e "$f" ] && [ ! -L "$f" ] && ! dpkg -S "$f" >/dev/null 2>&1 && echo "  $f"
done
for h in $(homes); do
  [ -d "$h/.config/systemd/user" ] && echo "  user units in $h: $(ls "$h/.config/systemd/user" | tr '\n' ' ')"
done
echo "timers:"
systemctl list-timers --all --no-legend 2>/dev/null | grep -Eo '[^ ]+\.timer +[^ ]+\.service' | sed 's/^/  /'
echo "other process managers:"
for h in $(homes); do
  [ -f "$h/.pm2/dump.pm2" ] && echo "  pm2 in $h: $(python3 -c 'import json, sys
print(" ".join("%s=%s" % (a.get("name"), a.get("pm_cwd")) for a in json.load(open(sys.argv[1]))))' "$h/.pm2/dump.pm2" 2>/dev/null)"
done
[ -d /etc/supervisor/conf.d ] && echo "  supervisor: $(ls /etc/supervisor/conf.d | tr '\n' ' ')"
echo "  screen and tmux: $(ls /run/screen /tmp/tmux-* 2>/dev/null | tr '\n' ' ')"
echo "started by hand (not under a service):"
ps -eo user:12,unit:24,args --no-headers 2>/dev/null | awk '$2 ~ /^session-/' |
  grep -Ev ' (-?bash|-?sh|-?zsh|sudo|sshd:.*|ps|awk|grep|sort|head|tail|sed|tr|cat|xargs|timeout|systemd|\(sd-pam\)|tmux: client)( |$)' |
  head -n 30 | sed 's/^/  /'

section listening
ss -tulpnH 2>/dev/null | awk '{print "  " $1, $5, $7}' | sed 's/users:(("\([^"]*\)".*/\1/' | sort -u | head -n 60

section web
if have nginx; then
  echo "nginx sites: $(ls /etc/nginx/sites-enabled /etc/nginx/conf.d 2>/dev/null | tr '\n' ' ')"
  nginx -T 2>/dev/null | grep -E '^# configuration file|^[[:space:]]*(listen|server_name|root|proxy_pass|fastcgi_pass|return 30[1-8]|ssl_certificate) ' |
    grep -v '^# configuration file /etc/nginx/\(mime.types\|fastcgi\|snippets/\|modules-enabled\)' | head -n 200
fi
have apache2ctl && { echo "apache:"; apache2ctl -S 2>&1 | head -n 60; }
[ -f /etc/caddy/Caddyfile ] && { echo "Caddyfile:"; clean /etc/caddy/Caddyfile | head -n 80; }
echo "certificates:"
for c in /etc/letsencrypt/live/*/cert.pem; do
  [ -f "$c" ] && echo "  $(openssl x509 -in "$c" -noout -ext subjectAltName 2>/dev/null | grep -o 'DNS:[^,]*' | tr '\n' ' ')$(openssl x509 -in "$c" -noout -enddate)"
done

section docker
if have docker && docker info >/dev/null 2>&1; then
  docker info --format 'docker {{.ServerVersion}}, root {{.DockerRootDir}}, storage {{.Driver}}'
  docker compose version 2>/dev/null || docker-compose version 2>/dev/null | head -n 1
  echo "compose projects:"
  docker compose ls -a 2>/dev/null | sed 's/^/  /'
  echo "containers:"
  docker ps -aq | xargs -r docker inspect --format '  {{.Name}} image={{.Config.Image}} state={{.State.Status}} restart={{.HostConfig.RestartPolicy.Name}} project={{index .Config.Labels "com.docker.compose.project"}} service={{index .Config.Labels "com.docker.compose.service"}} dir={{index .Config.Labels "com.docker.compose.project.working_dir"}} ports={{range $p, $b := .HostConfig.PortBindings}}{{range $b}}{{.HostIp}}:{{.HostPort}}->{{$p}} {{end}}{{end}}mounts={{range .Mounts}}{{if eq .Type "volume"}}{{.Name}}{{else}}{{.Source}}{{end}}:{{.Destination}} {{end}}'
  echo "images:"
  docker images --format '{{.Repository}}:{{.Tag}} {{.ID}} {{.Size}}' | sed 's/^/  /'
  echo "volumes:"
  docker system df -v 2>/dev/null | awk '/^VOLUME NAME/ {v=1; next} /^$/ {v=0} v {print "  " $1, $NF}'
else
  echo "none"
fi

section databases
if have psql; then
  pg_lsclusters 2>/dev/null
  runuser -u postgres -- psql -XAtF ' ' -c "select datname, pg_size_pretty(pg_database_size(datname)), pg_encoding_to_char(encoding), datcollate from pg_database where not datistemplate order by 1" 2>&1 | sed 's/^/  postgres: /'
  echo "  postgres roles: $(runuser -u postgres -- psql -XAtc "select rolname from pg_roles where rolname !~ '^pg_'" 2>&1 | tr '\n' ' ')"
fi
if have mysql; then
  mysql -e 'select 1' >/dev/null 2>&1 || export MYSQL_PWD="$(sed -n 's/^root_mysql_pass="\(.*\)"$/\1/p' /root/.digitalocean_password 2>/dev/null)"
  mysql -NBe "select table_schema, concat(round(sum(data_length + index_length) / 1048576), ' MB') from information_schema.tables group by 1" 2>&1 | sed 's/^/  mysql: /'
  echo "  mysql users: $(mysql -NBe "select concat(user, '@', host, ':', plugin) from mysql.user" 2>&1 | tr '\n' ' ')"
fi
have redis-cli && echo "  redis: $(redis-cli INFO keyspace 2>&1 | grep -v '^#' | tr -d '\r' | tr '\n' ' ') dir=$(redis-cli CONFIG GET dir 2>&1 | tail -n 1)"
have mongod && echo "  mongodb: $(mongod --version | head -n 1)"
echo "sqlite files:"
find $apps /var/lib -xdev -type f \( -name '*.sqlite' -o -name '*.sqlite3' -o -name '*.db' \) -size +0 \
  -not -path '*/node_modules/*' -not -path '*/pkg/mod/*' -not -path '*/.cache/*' -not -path '*/.local/share/*' -not -path '/var/lib/docker/*' -not -path '/var/lib/dpkg/*' -not -path '/var/lib/apt/*' \
  -not -path '/var/lib/systemd/*' -not -path '/var/lib/command-not-found/*' -not -path '/var/lib/fwupd/*' -not -path '/var/lib/PackageKit/*' \
  2>/dev/null | head -n 30 | sed 's/^/  /'

section schedules
for f in /var/spool/cron/crontabs/*; do
  [ -f "$f" ] && { echo "crontab of ${f##*/}:"; clean "$f" | sed 's/^/  /'; }
done
for f in /etc/cron.d/* /etc/cron.hourly/* /etc/cron.daily/* /etc/cron.weekly/* /etc/cron.monthly/*; do
  [ -f "$f" ] && ! dpkg -S "$f" >/dev/null 2>&1 && case $f in */.placeholder) ;; *) echo "$f:"; clean "$f" | head -n 20 | sed 's/^/  /' ;; esac
done
echo "/etc/crontab: $(clean /etc/crontab | grep -vc 'run-parts') lines besides run-parts"

section other
have ufw && { echo "ufw:"; ufw status 2>/dev/null | sed 's/^/  /'; }
have postconf && echo "postfix: $(postconf -h relayhost myhostname inet_interfaces 2>/dev/null | tr '\n' ' ')"
echo "/etc/hosts additions:"
clean /etc/hosts | grep -Ev 'localhost|ip6-|^(127\.0\.1\.1|::1) ' | sed 's/^/  /'
echo "/etc/environment: $(sed -n 's/^\([A-Za-z_][A-Za-z0-9_]*\)=.*/\1/p' /etc/environment | tr '\n' ' ')"
echo "sysctl: $(ls /etc/sysctl.d 2>/dev/null | tr '\n' ' ')$(clean /etc/sysctl.conf | wc -l) lines in sysctl.conf"
echo "git checkouts:"
find $apps -xdev -maxdepth 5 -name .git -not -path '*/node_modules/*' 2>/dev/null | head -n 30 |
  while read -r g; do
    echo "  ${g%/.git} $(git -c safe.directory='*' -C "${g%/.git}" remote get-url origin 2>/dev/null | sed 's#//[^/@]*@#//#')"
  done
echo "env files:"
find $apps /etc -xdev -maxdepth 6 -type f \( -name '.env' -o -name '.env.*' -o -name '*.env' \) \
  -not -path '*/node_modules/*' 2>/dev/null | head -n 30 | sed 's/^/  /'
EOF
ssh droplet sudo -n bash -s < ~/droplet/survey.sh > ~/droplet/survey.txt
grep '^## ' ~/droplet/survey.txt

Expect a dozen sections, from ## system to ## other. Read all of it: the rest of the rehearsal works from it. Keep a copy of the droplet's /etc too, to read and compare; nothing copies it onto this VM's system:

sudo rsync -a --rsync-path='sudo -n rsync' -e "ssh -F $HOME/.ssh/config" \
  --exclude='/shadow*' --exclude='/gshadow*' --exclude='/ssh/ssh_host_*' droplet:/etc/ ~/droplet/etc/

Make room#

Compare the droplet's disk use (the system and disk sections) with the space free here (df -h /), keeping 10 GB to spare, and its memory with free -m. If the VM is short, ask the owner to run, on their computer:

ssh exe.dev resize my-server --disk=100GB --memory=8GB

If df -h / doesn't show the new size, restart the VM with sudo reboot.

Match the system#

Set the droplet's timezone, and generate the locales it uses, especially those of its databases (datcollate under databases):

sudo timedatectl set-timezone Etc/UTC
sudo apt-get install -y locales && sudo locale-gen en_US.UTF-8

Create the droplet's users and groups (under users) with the same names, homes, and shells, and the same IDs where they're free. They often aren't: exedev is 1000 here, and that's also the first user on most droplets. That's fine, because copy matches owners by name:

sudo groupadd -g 1001 deploy
sudo useradd -m -u 1001 -g deploy -G sudo,www-data -s /bin/bash deploy

Install what the droplet has and the VM lacks. These are its packages that aren't here:

sed -n '/^manually installed/,/^$/p' ~/droplet/survey.txt | tail -n +2 | tr ' ' '\n' | grep . | sort -u > ~/droplet/packages
comm -23 ~/droplet/packages <(dpkg-query -W -f='${Package}\n' | sort -u)

Add the third-party repositories it uses first (under apt sources added), and install the packages its software needs: web servers, language runtimes and their extensions, databases, and tools that apt history shows were installed by hand. Skip the kernel and bootloader, cloud-init, DigitalOcean's do-agent and droplet-agent, ufw and fail2ban (exe.dev's proxy is the only way in), certbot, and libraries that only belong to the droplet's Ubuntu release.

Match major versions. Where Ubuntu 24.04's differ from the droplet's, use the software's own repository:

# PHP, any version:
sudo add-apt-repository -y ppa:ondrej/php
# PostgreSQL, any version:
sudo apt-get install -y postgresql-common && sudo /usr/share/postgresql-common/pgdg/apt.postgresql.org.sh -y
# Python 3.7 to 3.11, where the virtualenvs under "python virtualenvs" look for it:
sudo add-apt-repository -y ppa:deadsnakes/ppa

Then, for example, sudo apt-get install -y php8.1-fpm php8.1-mysql postgresql-14 python3.10-venv. Node.js installed with nvm comes along with the user's home; from NodeSource, add its repository for the same major version. Install MySQL or MariaDB, whichever the droplet runs. If the droplet runs containers, turn Docker on for good, and load the images its containers use, bit for bit, private ones included:

sudo systemctl enable --now docker.service
images=$(ssh droplet "sudo -n docker ps -a --format '{{.Image}}'" | sort -u | tr '\n' ' ')
ssh droplet "sudo -n docker save $images" | sudo docker load

Expect a Loaded image line for each.

Write the plan#

~/droplet/plan lists what moves; copy, check, and move read it. Write yours from the survey. This one is for a WordPress droplet from DigitalOcean's 1-Click Marketplace app:

cat > ~/droplet/plan <<'EOF'
# What moves from the droplet. ~/droplet/copy, check, and move read this.
#
#   app UNIT  |  app DIR [SERVICE...]
#       Something that writes data: a systemd unit, or a Docker Compose
#       project by its directory (and some of its services). move stops it
#       on the droplet, and copy stops it here while it copies.
#   worker UNIT  |  worker DIR SERVICE...
#       Something that acts on its own: a queue worker, a timer, a bot.
#       Stopped like an app, and kept off here until the move.
#   files PATH
#       A directory or file, copied to the same place, owners matched by
#       name. Not / or /etc or /home or /root whole: list what's inside.
#   docker-files PATH
#       The same, for files that containers use: owners kept as numbers.
#   volume NAME
#       A Docker volume, copied file by file.
#   postgres  |  mysql
#       Every PostgreSQL, or MySQL or MariaDB, database, with its users,
#       dumped on the droplet and restored here.
#   keep PATH
#       Changed here for exe.dev: copied once, then left as it is here.
#   site HOST [PATH...]
#       A site the droplet serves, for check to compare (default path: /).

app php8.1-fpm.service
files /etc/nginx
files /etc/php/8.1
files /var/www/html
files /root/.digitalocean_password
files /var/spool/cron/crontabs/root
mysql
keep /etc/nginx/sites-available/wordpress
keep /etc/nginx/conf.d/exe-dev.conf
site example.com / /wp-login.php
site www.example.com
EOF

What to list:

• files: everything that isn't the operating system. App code and data (/var/www, /srv, /opt/<app>, /home/<user>, what's in /root), the configuration of what you installed (/etc/nginx, /etc/php/<v>, /etc/redis, /etc/supervisor), the units under units added by hand (/etc/systemd/system/<app>.service), crontabs (/var/spool/cron/crontabs/<user>) and files in /etc/cron.d, tools in /usr/local/bin, and volumes under /mnt. Not /etc/letsencrypt: exe.dev does TLS now. Not what packages own (/usr/bin, /usr/lib): install those here instead, with apt, npm install -g, or pip.
• docker-files for each Compose project's directory (dir= under docker) and other bind mounts, and volume for every volume a container mounts.
• postgres and mysql if the droplet runs them. For other databases, such as Redis or MongoDB, list the unit as an app and the data directory as files (/var/lib/redis), so that the move copies them stopped.
• app for each service and Compose project that writes: PHP-FPM, your app servers, Redis. Not the PostgreSQL or MySQL server, which the dumps need running.
• worker for whatever acts by itself: queue workers, timers under units added by hand, bots, and an app that runs its own schedule. The droplet's cron jobs are held back for you.
• site for every server_name (or Apache ServerName and ServerAlias), with a few paths worth comparing.

Copy#

This script copies the plan to the same places here. It only reads from the droplet, gently, and each run copies only what changed:

cat > ~/droplet/copy <<'EOF'
#!/bin/bash
# copy: bring what ~/droplet/plan lists from the droplet to the same places
# on this VM. Run it as often as you like: each run copies only what changed,
# and it only reads from the droplet.
#   ~/droplet/copy             copy everything in the plan
#   ~/droplet/copy --dry-run   check the plan and say what a copy would do
# While it copies, the plan's apps and workers here are stopped. The apps
# start again after; the workers wait for the move (~/droplet/move).
set -euo pipefail
here=$(cd "$(dirname "$0")" && pwd)
cd /
mode=${1:-copy}
case $mode in copy | --dry-run | --final) ;; *) echo "usage: $0 [--dry-run]" >&2; exit 2 ;; esac
ssh=(ssh -F "$HOME/.ssh/config" -o BatchMode=yes)
# Gentle on the droplet: rsync and the dumps run at low priority there.
low="nice -n 10 ionice -c 3"

die() { echo "copy: $*" >&2; exit 1; }
say() { printf '\n== %s\n' "$*"; }
# Run a command on the droplet as root.
on_droplet() { "${ssh[@]}" droplet "cd / && sudo -n $(printf '%q ' "$@")"; }

# This VM's own system, and exe.dev's plumbing: never copied into or over.
never=(/proc /sys /dev /run /boot /tmp /lost+found /etc/ssh /etc/passwd /etc/shadow /etc/group
  /etc/gshadow /etc/sudoers /etc/sudoers.d /etc/hostname /etc/hosts /etc/resolv.conf /etc/fstab
  /etc/netplan /etc/cloud /etc/apt /etc/docker /var/lib/dpkg /var/lib/apt /var/lib/docker
  /var/lib/containerd /var/lib/postgresql /var/lib/mysql /home/exedev /usr/bin /usr/sbin /usr/lib)
# These hold this VM's own files too: list what's inside them instead.
not_whole=(/ /bin /sbin /lib /lib64 /usr /usr/local /usr/local/bin /usr/local/lib /etc /etc/systemd
  /etc/systemd/system /etc/cron.d /var /var/lib /var/spool /var/spool/cron /var/spool/cron/crontabs
  /home /opt /srv /mnt /root)

check_path() {
  local p=$1 d
  for d in "${not_whole[@]}"; do
    [ "$p" = "$d" ] && die "plan: $p holds this VM's own files too; list what's inside it instead"
  done
  [[ $p == /* && $p != */ && $p != *//* && $p/ != */../* && $p/ != */./* ]] ||
    die "plan: $p: write an absolute path, without . or .. or a trailing /"
  case $p in *.wants | *.wants/* | *.requires | *.requires/*)
    die "plan: $p: enable units with systemctl instead of copying their links" ;;
  esac
  for d in "${never[@]}"; do
    case $p/ in "$d"/*) die "plan: $p is this VM's own; leave it out" ;; esac
    case $d/ in "$p"/*) die "plan: $p contains $d, which is this VM's own; list what's inside it instead" ;; esac
  done
}

# Read the plan.
[ -f "$here/plan" ] || die "no plan: write $here/plan first"
apps=() workers=() files=() numeric=() volumes=() keeps=() dbs=()
while read -r kind arg rest; do
  case $kind in
    '' | '#'*) continue ;;
    app) apps+=("$arg${rest:+ $rest}") ;;
    worker) workers+=("$arg${rest:+ $rest}") ;;
    files) check_path "$arg"; files+=("$arg") ;;
    docker-files) check_path "$arg"; numeric+=("$arg") ;;
    volume) volumes+=("$arg") ;;
    keep) [[ $arg == /* ]] || die "plan: keep $arg: write an absolute path"; keeps+=("$arg") ;;
    postgres | mysql) dbs+=("$kind") ;;
    site) ;;
    *) die "plan: unknown line: $kind $arg" ;;
  esac
  if [ -n "$rest" ]; then
    case $kind in files | docker-files | volume | keep) die "plan: one path per line: $kind $arg $rest" ;; esac
  fi
done < "$here/plan"

# An app or worker is a unit (nginx.service, app.timer) or a Compose project:
# its directory, then optionally some of its services.
is_unit() { [[ $1 != /* ]]; }
units() { local a; for a in "$@"; do if is_unit "$a"; then echo "$a"; fi; done; }
compose() { # compose ENTRY COMMAND...: run a Compose command on ENTRY's services
  local dir=${1%% *} svcs=()
  [[ $1 == *' '* ]] && read -ra svcs <<< "${1#* }"
  (cd "$dir" 2>/dev/null && sudo docker compose "${@:2}" "${svcs[@]}") 2>/dev/null
}

stop_here() {
  local a u
  for u in $(units "${workers[@]}" "${apps[@]}"); do sudo systemctl stop "$u" 2>/dev/null || true; done
  for a in "${workers[@]}" "${apps[@]}"; do
    is_unit "$a" || compose "$a" stop >/dev/null || true
  done
}

# Start the apps that are set up here. Workers stay stopped until the move.
start_here() {
  local a u
  for u in $(units "${apps[@]}"); do
    if systemctl is-enabled -q "$u" 2>/dev/null; then sudo systemctl start "$u"; fi
  done
  for a in "${apps[@]}"; do
    if ! is_unit "$a" && [ -n "$(compose "${a%% *}" ps -aq)" ]; then compose "$a" start >/dev/null; fi
  done
  if [ "$mode" = --final ]; then
    for u in $(units "${workers[@]}"); do sudo systemctl enable --now "$u"; done
    for a in "${workers[@]}"; do is_unit "$a" || compose "$a" up -d >/dev/null; done
  else
    for a in "${workers[@]}"; do is_unit "$a" || compose "$a" stop >/dev/null || true; done
  fi
}

# rsync from the droplet, at low priority there. Exit 24 is files that
# vanished while it ran, which a running server has.
pull() {
  local out rc=0
  out=$(sudo rsync -aHAX --delete --info=stats2 --rsync-path="sudo -n $low rsync" -e "${ssh[*]}" "$@") || rc=$?
  awk -F': ' '/^Number of regular files transferred/ {n = $2} /^Number of deleted files/ {d = $2}
    /^Total transferred file size/ {b = $2} END {print "  " n " files copied, " d " deleted (" b ")"}' <<< "$out"
  [ "$rc" = 0 ] || [ "$rc" = 24 ] || die "rsync $*: exit $rc"
}

copy_files() { # copy_files [--numeric-ids] PATH: the droplet's PATH to the same place here
  local opts=(--relative --no-implied-dirs) k
  [ "$1" = --numeric-ids ] && { opts+=(--numeric-ids); shift; }
  # What's kept is copied once, then left as it is here.
  for k in "${keeps[@]}"; do
    case $k/ in "$1"/*) if [ -e "$k" ] || sudo -n test -e "$k" 2>/dev/null; then opts+=(--exclude="$k"); fi ;; esac
  done
  if [ "$mode" = --dry-run ]; then echo "rsync ${opts[*]} droplet:$1 /"; else pull "${opts[@]}" "droplet:$1" /; fi
}

copy_volume() {
  local v=$1 from labels
  from=$(on_droplet docker volume inspect -f '{{.Mountpoint}}' "$v") || die "no Docker volume $v on the droplet"
  if ! sudo docker volume inspect "$v" >/dev/null 2>&1; then
    # The same labels, so Docker Compose takes it for its own.
    labels=$(on_droplet docker volume inspect -f '{{range $k, $v := .Labels}}--label={{$k}}={{$v}} {{end}}' "$v")
    sudo docker volume create $labels "$v" >/dev/null
  fi
  pull --numeric-ids "droplet:$from/" "$(sudo docker volume inspect -f '{{.Mountpoint}}' "$v")/"
}

copy_postgres() {
  local dump=$here/dumps/postgres.sql.gz log=$here/dumps/postgres.log
  command -v psql >/dev/null || die "install the droplet's PostgreSQL here first"
  on_droplet sh -c "runuser -u postgres -- $low pg_dumpall --clean --if-exists | gzip -1" > "$dump.part"
  mv "$dump.part" "$dump"
  # Nothing may hold a database open while it is replaced.
  sudo -u postgres psql -XAtq -d postgres -c \
    "select pg_terminate_backend(pid) from pg_stat_activity where pid <> pg_backend_pid() and datname <> 'postgres'" >/dev/null
  gunzip -c "$dump" | sudo -u postgres psql -Xq -d postgres > /dev/null 2> "$log" || true
  # pg_dumpall --clean also drops and creates the role it runs as, which can't be.
  if grep -v -e 'current user cannot be dropped' -e 'role "postgres" already exists' "$log" | grep -m 20 ERROR; then
    die "PostgreSQL: the restore had the errors above; they are all in $log"
  fi
  echo "PostgreSQL: $(sudo -u postgres psql -XAtc 'select count(*) from pg_database where not datistemplate') databases"
}

copy_mysql() {
  local dump=$here/dumps/mysql.sql.gz
  command -v mysql >/dev/null || die "install the droplet's MySQL or MariaDB here first"
  # The droplet's MySQL root password, if root needs one and it isn't in
  # DigitalOcean's /root/.digitalocean_password.
  local password=
  [ -f "$here/mysql-password" ] && password="export MYSQL_PWD=$(printf %q "$(cat "$here/mysql-password")")"
  # Users and grants first, then every database but the server's own. root
  # stays as it is on each side.
  "${ssh[@]}" droplet "cd / && sudo -n bash -s" > "$dump.part" <<EOS
set -euo pipefail
$password
mysql -e 'select 1' >/dev/null 2>&1 ||
  export MYSQL_PWD="\$(sed -n 's/^root_mysql_pass="\(.*\)"\$/\1/p' /root/.digitalocean_password 2>/dev/null)"
{
  mysql -NBre "select concat(quote(user), '@', quote(host)) from mysql.user where user not in
    ('', 'root', 'debian-sys-maint', 'mysql.sys', 'mysql.session', 'mysql.infoschema', 'mariadb.sys')" |
  while read -r u; do
    echo "DROP USER IF EXISTS \$u;"
    { mysql -NBre "SET SESSION print_identified_with_as_hex = ON; SHOW CREATE USER \$u" 2>/dev/null ||
      mysql -NBre "SHOW CREATE USER \$u"; } | sed 's/\$/;/'
    mysql -NBre "SHOW GRANTS FOR \$u" | sed 's/\$/;/'
  done
  dbs=\$(mysql -NBe 'show databases' | grep -Evx 'mysql|information_schema|performance_schema|sys' || true)
  gtid=\$(mysqldump --help | grep -q set-gtid-purged && echo --set-gtid-purged=OFF || true)
  [ -z "\$dbs" ] || $low mysqldump \$gtid --databases \$dbs --add-drop-database --single-transaction \\
    --routines --events --triggers --hex-blob --default-character-set=utf8mb4
  echo 'FLUSH PRIVILEGES;'
} | gzip -1
EOS
  mv "$dump.part" "$dump"
  gunzip -c "$dump" | sudo -H mysql || die "MySQL: the restore stopped at the error above; fix it and run copy again"
  echo "MySQL: $(sudo -H mysql -NBe 'show databases' | grep -Evxc 'mysql|information_schema|performance_schema|sys') databases"
}

if [ "$mode" = --dry-run ]; then
  echo "stop here: ${workers[*]} ${apps[*]}"
  for p in "${files[@]}"; do copy_files "$p"; done
  for p in "${numeric[@]}"; do copy_files --numeric-ids "$p"; done
  for v in "${volumes[@]}"; do echo "volume $v"; done
  for d in "${dbs[@]}"; do echo "dump and restore $d"; done
  echo "start here: ${apps[*]}"
  exit 0
fi

"${ssh[@]}" droplet sudo -n true || die "can't reach the droplet as root; try: ssh droplet sudo -n true"
mkdir -p "$here/dumps" && chmod 700 "$here/dumps"
start=$SECONDS
say "stopping the apps and workers here"
stop_here
trap '[ $? = 0 ] || echo "copy: stopped early, with the apps here stopped. Fix the problem and run copy again." >&2' EXIT
for p in "${files[@]}"; do say "files $p"; copy_files "$p"; done
for p in "${numeric[@]}"; do say "docker-files $p"; copy_files --numeric-ids "$p"; done
for v in "${volumes[@]}"; do say "volume $v"; copy_volume "$v"; done
for d in "${dbs[@]}"; do say "$d"; "copy_$d"; done
say "starting the apps here"
start_here
say "copied in $((SECONDS - start)) seconds"
EOF
chmod +x ~/droplet/copy
~/droplet/copy --dry-run

The dry run refuses paths that would overwrite this VM's own system, such as /etc or /home whole, and shows the rest. Then copy:

~/droplet/copy

Expect a line for each part and copied in N seconds at the end; move takes about as long as a second run does. If the droplet's MySQL root needs a password that isn't in /root/.digitalocean_password, put it in ~/droplet/mysql-password (mode 600).

Open the front door#

exe.dev's proxy handles HTTPS and forwards to port 8000 over plain HTTP, with X-Forwarded-Proto and X-Forwarded-For. So each site the droplet serves on 443 moves to 8000, without its certificate, and the redirects from HTTP to HTTPS go: here they would loop. For nginx, a site set up by certbot changes like this:

server {
    listen 8000 default_server;     # was: listen 443 ssl; (and [::]:443)
    listen [::]:8000 default_server;
    server_name example.com www.example.com;
    # ...as before, minus ssl_certificate, ssl_certificate_key, and
    # include /etc/letsencrypt/options-ssl-nginx.conf...
    location ~ \.php$ {
        include snippets/fastcgi-php.conf;
        fastcgi_param HTTPS $exe_https;
        fastcgi_pass unix:/run/php/php8.1-fpm.sock;
    }
}
# The certbot block for port 80, which only redirects to HTTPS: gone.

The main site gets default_server: it's what https://my-server.exe.xyz/ shows. Then tell the apps what the proxy saw:

sudo tee /etc/nginx/conf.d/exe-dev.conf > /dev/null <<'EOF'
# exe.dev's proxy is the only way in. It speaks HTTPS to visitors, and puts
# the visitor's address last in X-Forwarded-For.
set_real_ip_from 0.0.0.0/0;
set_real_ip_from ::/0;
real_ip_header X-Forwarded-For;
map $http_x_forwarded_proto $exe_https { https on; default ""; }
EOF
sudo nginx -t && sudo systemctl reload nginx

Where a site passes requests on with proxy_set_header X-Forwarded-Proto $scheme;, use $http_x_forwarded_proto instead. The same idea elsewhere:

• Apache: Listen 8000 and <VirtualHost *:8000> for the *:443 hosts, without SSLEngine and the certificate lines; drop the redirect hosts; add SetEnvIf X-Forwarded-Proto https HTTPS=on and RemoteIPHeader X-Forwarded-For (a2enmod remoteip).
• Caddy: http://example.com:8000, http://www.example.com:8000 as the site address, so Caddy stops getting certificates.
• A container that took 80 and 443 (Traefik, Caddy, nginx-proxy): publish its HTTP port as 8000, and turn off its HTTPS redirect and certificate resolver.
• An app the droplet served on a port of its own, say 3000: route it through port 8000 by host name. Ports 3000-9999 are reachable at https://my-server.exe.xyz:3000/, but only by people you share the VM with.

Add a keep line to the plan for every file you change, so that copy leaves it alone, and turn off certbot's renewals if it came along: sudo systemctl disable --now certbot.timer. On the VM, curl -sI -H 'Host: example.com' -H 'X-Forwarded-Proto: https' http://localhost:8000/ should now answer like the droplet does.

Start the services#

Enable what the droplet ran, the same way it ran it, apart from the workers:

• Units: sudo systemctl daemon-reload, then sudo systemctl enable --now each app and each service the droplet had enabled and running (enabled and running under services). Leave the plan's workers disabled; move turns them on.
• Compose: in each project's directory, sudo docker compose up -d, then sudo docker compose stop the worker services. Recreate containers started with docker run from docker inspect, with the same name, options, and restart policy.
• pm2: the pm2-<user>.service unit came with the files; enable it. It reads ~/.pm2/dump.pm2 at boot.
• Processes started by hand (in screen, tmux, or with nohup) don't survive a reboot there or here: give each a unit, and say so in the report.
• Cron: the crontabs are in place. Keep cron off until the move, so that no job runs on both machines:

sudo systemctl disable --now cron

Then run ~/droplet/copy again: it stops the apps and workers, brings what changed, and starts the apps, which also proves that it can.

Check#

This script fetches each page in the plan from the droplet (by its address, whatever DNS says) and from port 8000 here, as the proxy would, and compares the status, redirect, and title:

cat > ~/droplet/check <<'EOF'
#!/bin/bash
# check: fetch each site in ~/droplet/plan from the droplet and from this VM,
# and compare the status, the redirect, and the page title.
#   ~/droplet/check           compare with the droplet as it is now
#   ~/droplet/check --saved   compare with its answers from the last check
# It exits 1 if any page differs.
set -uo pipefail
here=$(cd "$(dirname "$0")" && pwd)
saved=$here/check.droplet
ip=$(ssh -F "$HOME/.ssh/config" -G droplet 2>/dev/null < /dev/null | awk '$1 == "hostname" {print $2}')
ip=$(getent ahostsv4 "$ip" | awk '{print $1; exit}')

# fetch URL [CURL_OPTIONS]: "<status> <redirect> | <title>"
fetch() {
  local body w
  body=$(mktemp)
  w=$(curl -sk -m 30 -o "$body" -w '%{http_code} %{redirect_url}' "$@")
  printf '%s | %s\n' "$w" "$(tr '\r\n\t' '   ' < "$body" | grep -oiE '<title[^>]*>[^<]*' | head -n 1 |
    sed -E 's/<[^>]*>//; s/^ +//; s/ +$//')"
  rm -f "$body"
}

if [ "${1:-}" != --saved ]; then : > "$saved.new"; fi
differ=0 pages=0
while read -r kind host paths; do
  [ "$kind" = site ] || continue
  for path in ${paths:-/}; do
    if [ "${1:-}" = --saved ]; then
      there=$(awk -v k="$host$path" '$1 == k {sub(/^[^ ]+ /, ""); print; exit}' "$saved")
    else
      there=$(fetch "https://$host$path" --resolve "$host:443:$ip")
      [[ $there == 000* ]] && there=$(fetch "http://$host$path" --resolve "$host:80:$ip")
      echo "$host$path $there" >> "$saved.new"
    fi
    # exe.dev's proxy speaks HTTPS to visitors and plain HTTP to port 8000.
    vm=$(fetch "http://localhost:8000$path" -H "Host: $host" -H "X-Forwarded-Proto: https" \
      -H "X-Forwarded-Host: $host" -H "X-Forwarded-For: 192.0.2.1")
    vm=${vm//http:\/\/localhost:8000/https:\/\/$host}
    pages=$((pages + 1))
    if [ "$there" = "$vm" ]; then
      printf 'same       %s%s: %s\n' "$host" "$path" "$vm"
    else
      differ=1
      printf 'DIFFERENT  %s%s\n  droplet: %s\n  this VM: %s\n' "$host" "$path" "$there" "$vm"
    fi
  done
done < "$here/plan"
[ "${1:-}" = --saved ] || mv "$saved.new" "$saved"
[ "$pages" -gt 0 ] || { echo "check: no site lines in $here/plan" >&2; exit 1; }
exit "$differ"
EOF
chmod +x ~/droplet/check
~/droplet/check

Expect same for every page. A difference names what's off: a redirect to http:// means the app didn't see X-Forwarded-Proto, a 502 that its backend isn't running here, a 421 or the wrong title that another server block answered.

Then compare the machines. Survey this VM, and list what runs on the droplet but not here, and what it listened on that nothing here does:

sudo bash ~/droplet/survey.sh > ~/droplet/survey.vm.txt
for s in survey.txt survey.vm.txt; do
  grep '^running:' ~/droplet/$s | tr ' ' '\n' | sort > ~/droplet/$s.running
  sed -n '/^## listening/,/^## /p' ~/droplet/$s | awk '{print $1, $NF}' | sort -u > ~/droplet/$s.listening
done
comm -23 ~/droplet/survey.txt.running ~/droplet/survey.vm.txt.running
comm -23 ~/droplet/survey.txt.listening ~/droplet/survey.vm.txt.listening

Each line is either deliberate (a worker, cron, an agent left behind, something exe.dev does instead) and goes in the report, or still to do.

Get ready for the move#

This script is the move, for the owner to run. It asks before it stops anything, and refuses to run without a terminal:

cat > ~/droplet/move <<'EOF'
#!/bin/bash
# move: the move. Stops the droplet's cron, workers, and apps; copies what
# changed since the rehearsal; starts the workers and cron here; and checks
# every site against the droplet's answers from just before.
#   ~/droplet/move        make the move
#   ~/droplet/move undo   back out, before your domains point here: stop the
#                         workers and cron here, and start the droplet again
set -euo pipefail
here=$(cd "$(dirname "$0")" && pwd)
cd /
ssh=(ssh -F "$HOME/.ssh/config" -o BatchMode=yes)
apps=() workers=() sites=()
while read -r kind arg rest; do
  case $kind in
    app) apps+=("$arg${rest:+ $rest}") ;;
    worker) workers+=("$arg${rest:+ $rest}") ;;
    site) sites+=("$arg") ;;
  esac
done < "$here/plan"

# droplet stop|start ENTRY...: units, or Compose directories with services.
droplet() {
  "${ssh[@]}" droplet "cd / && sudo -n bash -s -- $(printf '%q ' "$@")" <<'EOS'
action=$1; shift
for a in "$@"; do
  if [[ $a == /* ]]; then
    read -ra svcs <<< "${a#"${a%% *}"}"
    (cd "${a%% *}" && docker compose "$action" "${svcs[@]}") || echo "droplet: docker compose $action $a failed" >&2
  else
    systemctl "$action" "$a" || echo "droplet: systemctl $action $a failed" >&2
  fi
done
EOS
}

if [ "${1:-}" = undo ]; then
  for a in "${workers[@]}"; do
    if [[ $a == /* ]]; then
      read -ra svcs <<< "${a#"${a%% *}"}"
      (cd "${a%% *}" && sudo docker compose stop "${svcs[@]}")
    else
      sudo systemctl disable --now "$a"
    fi
  done
  sudo systemctl disable --now cron
  droplet start "${apps[@]}" "${workers[@]}" cron.service
  echo "The droplet is serving again, and this VM's workers and cron are off."
  exit 0
fi

[ -t 0 ] || { echo "move: run this yourself, in a terminal" >&2; exit 1; }
echo "This stops on the droplet: cron ${workers[*]} ${apps[*]}"
read -rp "Your sites there stay down until your domains point here. Go ahead? [y/N] " ok
[[ $ok == [yY]* ]] || exit 1

"$here/check" > /dev/null || true # the droplet's answers, for after
droplet stop cron.service "${workers[@]}" "${apps[@]}"
"$here/copy" --final
sudo systemctl enable --now cron
echo
"$here/check" --saved || echo "move: some pages differ from the droplet's; see above"

vm=$(hostname)
cat <<NEXT

Moved. Now point your domains at this VM. At your DNS provider, for each:

$(for s in "${sites[@]}"; do printf '  %-32s CNAME  %s.exe.xyz\n' "$s" "$vm"; done)

For a bare domain (example.com), use an ALIAS or a flattened CNAME instead,
or an A record to the address that \`host $vm.exe.xyz\` shows on your computer.
Then, on your computer:

$(for s in "${sites[@]}"; do echo "  ssh exe.dev domain add $vm $s"; done)
  ssh exe.dev share set-public $vm

Changed your mind before the DNS change? Run: ~/droplet/move undo
NEXT
EOF
chmod +x ~/droplet/move

Last, write ~/droplet/report.md for the owner, without secrets:

• What moved: the plan in plain words, the versions installed, and how long the last copy took.
• What changed for exe.dev: each keep file and why.
• What didn't move, and what to do instead: from the comparison above and anything you chose to leave.
• Before the move: anything the owner should decide or try first.
• The move: the domains, the DNS record each needs, and the commands from Make the move with this VM's name.

That's the rehearsal. Open https://my-server.exe.xyz/ and look around.

• ssh droplet says Permission denied (publickey): the key went to another user's authorized_keys. Run Let the VM in again with the User from ~/.ssh/config.
• sudo: a password is required: the user needs passwordless sudo. Log in as root instead; droplets allow it with SSH keys.
• copy: plan: ... holds this VM's own files too: list the directories inside it instead, such as /home/deploy for /home.
• rsync: command not found from the droplet: it needs rsync, which DigitalOcean's Ubuntu and Debian images have. Ask the owner to install it there.
• rsync ...: exit 23: some files couldn't be read or written; the lines above it name them.
• PostgreSQL: invalid locale name: generate it with sudo locale-gen, then copy again. extension ... is not available: install it here, such as postgresql-16-postgis-3.
• MySQL: Access denied for user 'root'@'localhost' on the droplet: put its root password in ~/droplet/mysql-password.
• MySQL: Unknown collation: the droplet runs MySQL and this VM MariaDB, or the other way round. Install the same one.
• A 502 here: the app behind the site isn't running. See systemctl status <unit> and journalctl -u <unit> -n 50, or sudo docker compose logs in its directory.
• Too many redirects at https://my-server.exe.xyz/: a redirect from HTTP to HTTPS is left in the web server's configuration or the app's.
• A virtualenv says python3.10: No such file or directory: install that Python from deadsnakes, where the virtualenv looks for it.
• Containers don't come back after a reboot: sudo systemctl enable --now docker.service.
• The disk filled up: see Make room, then run copy again; it picks up where it stopped.