Add systemd deployment for Linux servers

Browse files

scripts/deploy-server.sh user@host copies the checkout to the target,
builds it there (installing Rust with rustup if needed) and runs
packaging/linux/install.sh, which snapshots the database, swaps the
binary, health-checks it and rolls back on failure.

The unit restarts forever, keeps the web interface on loopback for a
proxy or tunnel, sandboxes the service and makes git fsync pushes so
power cuts can't corrupt repositories.

Co-Authored-By: Claude Opus 5.5 <[email protected]>

Sam Jeston committed 23 days ago

1 parent 9aa3c45 commit 102eb6122dc0324ca06d15163e5c2d5907724915

+3 −0 CLAUDE.md View file

		@@ -16,6 +16,7 @@ cargo clippy --all-targets # keep this warning-free
16	16	cargo fmt
17	17	scripts/smoke-test.sh [path/to/binary] # end-to-end test against a throwaway instance
18	18	docker compose up -d # production-style run (data in a named volume)
	19	+scripts/deploy-server.sh user@host # build on a Linux host over ssh and install/upgrade the systemd service
19	20	```
20	21
21	22	- The smoke test needs `git`, `ssh`, `ssh-keygen` and `curl`. It uses ports 38080 and 38022; override them with `HTTP_PORT` and `SSH_PORT`. It creates and deletes its own temp directory.
		@@ -54,6 +55,8 @@ docker compose up -d # production-style run (data in a named
54	55	\| `templates/` \| askama templates. See `templates/CLAUDE.md` \|
55	56	\| `static/` \| `style.css`, `app.js`, `favicon.svg`. Compiled into the binary with `include_str!` \|
56	57	\| `docs/` \| System documentation (start at `docs/README.md`) and the README screenshots in `docs/screenshots/` \|
	58	+\| `packaging/linux/` \| systemd unit and `install.sh` (snapshot DB, swap binary, health check, roll back). See `docs/operations.md` \|
	59	+\| `scripts/` \| `smoke-test.sh` (end-to-end test) and `deploy-server.sh` (build and install over ssh) \|
57	60
58	61	## How things fit together
59	62

+17 −0 README.md View file

		@@ -44,6 +44,23 @@ cargo build --release
44	44
45	45	By default the web interface listens on port `3000` and the SSH server on port `2222`. Everything is stored in `./data`.
46	46
	47	+### On a Linux server with systemd
	48	+
	49	+`scripts/deploy-server.sh` installs Openhub on a Linux machine as a systemd service, over SSH. It copies your checkout to the machine, builds it there (installing Rust with rustup the first time) and runs `packaging/linux/install.sh`. The machine needs `git`, `curl` and a C compiler, and your SSH user needs `sudo`.
	50	+
	51	+```sh
	52	+scripts/deploy-server.sh user@host
	53	+```
	54	+
	55	+The service runs as the `openhub` user, keeps its data in `/var/lib/openhub`, starts at boot and restarts whenever it stops. The web interface listens on `127.0.0.1:3000` for a reverse proxy or tunnel on the same machine. SSH listens on port `2222` on every interface. Put your settings in `/etc/default/openhub`, then run `sudo systemctl restart openhub`:
	56	+
	57	+```sh
	58	+OPENHUB_PUBLIC_URL=https://git.example.com
	59	+OPENHUB_DISABLE_REGISTRATION=true
	60	+```
	61	+
	62	+To upgrade, run the same command again. See [Operations → Running under systemd](docs/operations.md#running-under-systemd).
	63	+
47	64	## Pushing your first repository
48	65
49	66	1. Sign in and open Settings → SSH keys. Paste your public key, for example the contents of `~/.ssh/id_ed25519.pub`.

+37 −1 docs/operations.md View file

		@@ -11,6 +11,31 @@ This page covers running an Openhub instance day to day: first start, reverse pr
11	11	5. Set `OPENHUB_DISABLE_REGISTRATION=true` once everyone who needs an account has one, unless you intend to run an open instance.
12	12	6. Note the SSH host key fingerprint from the startup log (`SSH host key fingerprint: SHA256:…`) so users can verify it on first connect.
13	13
	14	+## Running under systemd
	15	+
	16	+`packaging/linux/` holds a systemd unit (`openhub.service`) and an installer (`install.sh`). `scripts/deploy-server.sh user@host` copies the checkout to the machine, builds it there and runs the installer over SSH (see the [README](../README.md#on-a-linux-server-with-systemd)).
	17	+
	18	+\| Item \| Where \|
	19	+\| --- \| --- \|
	20	+\| Binary \| `/usr/local/bin/openhub`. The previous one is kept as `openhub.prev` \|
	21	+\| Unit \| `/etc/systemd/system/openhub.service` \|
	22	+\| Settings \| `OPENHUB_*` variables in `/etc/default/openhub`. The first install creates it with commented examples \|
	23	+\| Data directory \| `/var/lib/openhub`, owned by the `openhub` system user \|
	24	+\| Logs \| `journalctl -u openhub` \|
	25	+\| Source and build cache \| `~/.cache/openhub-build` in the SSH user's home on the target \|
	26	+
	27	+The unit listens for the web interface on `127.0.0.1:3000` and for SSH on `0.0.0.0:2222`. Override either in `/etc/default/openhub`.
	28	+
	29	+Each run of `install.sh`:
	30	+
	31	+1. Stops the service and copies `openhub.db` into `backups/<timestamp>/` in the data directory, keeping the last five. Repositories aren't copied.
	32	+2. Swaps the binary, starts the service, and waits up to 20 seconds for the web interface to answer.
	33	+3. If it doesn't answer, puts the previous binary back. If the new version had already migrated the database, restore the snapshot as described in [Upgrades](#upgrades).
	34	+
	35	+Power cuts. The unit restarts the server whenever it exits and never stops retrying (`Restart=always`, `StartLimitIntervalSec=0`). It also sets git's `core.fsync=committed` through `GIT_CONFIG_*` variables, so a push's objects and refs are on disk before git reports success. SQLite runs in WAL mode with `synchronous=NORMAL`, so a power cut can lose the last few database writes but doesn't corrupt the database.
	36	+
	37	+Sandboxing. The unit uses `ProtectSystem=strict`, so the server can only write to `/var/lib/openhub`. If you move the data directory, also allow it with `ReadWritePaths=` in a drop-in (`sudo systemctl edit openhub`).
	38	+
14	39	## Logs
15	40
16	41	Openhub logs to stdout through `tracing`. Set the level with `RUST_LOG`:
		@@ -52,6 +77,17 @@ server {
52	77	}
53	78	```
54	79
	80	+Cloudflare Tunnel (an ingress rule in the `cloudflared` config, then `cloudflared tunnel route dns <tunnel> git.example.com`):
	81	+
	82	+```yaml
	83	+ingress:
	84	+ - hostname: git.example.com
	85	+ service: http://127.0.0.1:3000
	86	+ - service: http_status:404
	87	+```
	88	+
	89	+A tunnel carries only the web interface. Clients still need to reach the SSH port another way, for example on the local network or through a forwarded port. Set `OPENHUB_SSH_HOST` to the address they use.
	90	+
55	91	Then set `OPENHUB_PUBLIC_URL=https://git.example.com`. The cross-origin check accepts form posts whose `Origin` matches either the public URL or the `Host` header, so proxies that rewrite `Host` still work.
56	92
57	93	If you rate-limit at the proxy (recommended, see [Security](security.md#hardening-advice)), cover `POST /login`, `POST /register` and `///info/refs`.
		@@ -71,7 +107,7 @@ Everything lives in the data directory (see [Architecture](architecture.md#data-
71	107	## Upgrades
72	108
73	109	1. Back up the data directory.
74		-2. Build or pull the new version. From source, run `git pull && cargo build --release`. With Docker, run `docker compose build && docker compose up -d`.
	110	+2. Build or pull the new version. From source, run `git pull && cargo build --release`. With Docker, run `docker compose build && docker compose up -d`. Under systemd, run `scripts/deploy-server.sh user@host` again: it snapshots the database itself, and puts the old binary back if the new one fails to start (see [Running under systemd](#running-under-systemd)).
75	111	3. Start it. Pending database migrations run automatically at startup, before the servers listen.
76	112
77	113	Migrations only go forward. To roll back, restore the backup taken in step 1 and run the old version.

+123 −0 packaging/linux/install.sh added View file

		@@ -0,0 +1,123 @@
	1	+#!/usr/bin/env bash
	2	+# Installs or updates openhub as a systemd service. Run it as root on the
	3	+# target machine (scripts/deploy-server.sh builds openhub there and runs this
	4	+# for you):
	5	+#
	6	+# sudo ./install.sh ./openhub [./openhub.service]
	7	+#
	8	+# Each run stops the service, snapshots the database into backups/ in the data
	9	+# directory (keeping the last 5), swaps the binary (the old one stays as
	10	+# openhub.prev) and starts the service again. If the new binary fails its
	11	+# health check, the previous one is put back. Repositories aren't snapshotted:
	12	+# upgrades only migrate the database.
	13	+set -euo pipefail
	14	+
	15	+bin=${1:?usage: install.sh BINARY [UNIT_FILE]}
	16	+unit=${2:-}
	17	+dest=/usr/local/bin/openhub
	18	+unit_dest=/etc/systemd/system/openhub.service
	19	+defaults=/etc/default/openhub
	20	+keep_backups=5
	21	+
	22	+[ "$(id -u)" = 0 ] \|\| { echo "install.sh must run as root, e.g. with sudo." >&2; exit 1; }
	23	+
	24	+# setting NAME DEFAULT: the value the service sees, honouring /etc/default/openhub.
	25	+setting() {
	26	+ (
	27	+ [ -f "$defaults" ] && . "$defaults"
	28	+ printf '%s\n' "${!1:-$2}"
	29	+ )
	30	+}
	31	+
	32	+# put SRC: replace the installed binary without writing into a running executable.
	33	+put() {
	34	+ install -m 0755 "$1" "$dest.new"
	35	+ mv -f "$dest.new" "$dest"
	36	+}
	37	+
	38	+# get URL: succeeds when URL answers without an HTTP error.
	39	+get() {
	40	+ if command -v curl >/dev/null; then
	41	+ curl -fs --max-time 2 -o /dev/null "$1"
	42	+ else
	43	+ wget -q -T 2 -O /dev/null "$1"
	44	+ fi
	45	+}
	46	+
	47	+# Refuse a binary built for another architecture before touching the service.
	48	+new_version=$("$bin" --version) \|\| { echo "$bin doesn't run on this machine. Build it for $(uname -m)." >&2; exit 1; }
	49	+git_bin=$(setting OPENHUB_GIT_BIN git)
	50	+command -v "$git_bin" >/dev/null \|\| { echo "openhub needs git at runtime and can't find $git_bin. Install git first." >&2; exit 1; }
	51	+
	52	+data=$(setting OPENHUB_DATA_DIR /var/lib/openhub)
	53	+db=$data/openhub.db
	54	+addr=$(setting OPENHUB_HTTP_ADDR 127.0.0.1:3000)
	55	+case $addr in
	56	+ 0.0.0.0:) addr=127.0.0.1:${addr#:} ;;
	57	+ "[::]":) addr=[::1]:${addr##:} ;;
	58	+esac
	59	+url=http://$addr/
	60	+
	61	+id openhub >/dev/null 2>&1 \|\| useradd --system --user-group --home-dir /var/lib/openhub --no-create-home --shell /usr/sbin/nologin openhub
	62	+
	63	+if [ ! -f "$defaults" ]; then
	64	+ cat >"$defaults" <<'EOF'
	65	+# Settings for the openhub service; see "Configuration" in the openhub README.
	66	+# Restart after editing: sudo systemctl restart openhub
	67	+#OPENHUB_PUBLIC_URL=https://git.example.com
	68	+#OPENHUB_SSH_HOST=git.example.com
	69	+#OPENHUB_SSH_PORT=2222
	70	+#OPENHUB_DISABLE_REGISTRATION=true
	71	+EOF
	72	+ echo "Wrote $defaults. Set OPENHUB_PUBLIC_URL there, then run: sudo systemctl restart openhub"
	73	+fi
	74	+
	75	+if [ -n "$unit" ] && ! cmp -s "$unit" "$unit_dest"; then
	76	+ install -m 0644 "$unit" "$unit_dest"
	77	+ systemctl daemon-reload
	78	+fi
	79	+[ -f "$unit_dest" ] \|\| { echo "No openhub unit is installed yet. Pass the unit file as the second argument." >&2; exit 1; }
	80	+
	81	+systemctl stop openhub 2>/dev/null \|\| true
	82	+
	83	+snap=
	84	+if [ -f "$db" ]; then
	85	+ backups=$data/backups
	86	+ snap=$backups/$(date +%Y%m%d-%H%M%S)
	87	+ mkdir -p "$snap"
	88	+ # The service is stopped, so the files are consistent; -wal/-shm only exist after an unclean stop.
	89	+ for f in "$db" "$db-wal" "$db-shm"; do
	90	+ if [ -f "$f" ]; then cp -p "$f" "$snap/"; fi
	91	+ done
	92	+ chown -R openhub:openhub "$backups"
	93	+ ls -1d "$backups"/*/ \| head -n -"$keep_backups" \| xargs -r rm -rf --
	94	+ echo "Database snapshot: $snap"
	95	+fi
	96	+
	97	+if [ -f "$dest" ]; then cp -p "$dest" "$dest.prev"; fi
	98	+put "$bin"
	99	+systemctl enable --quiet openhub
	100	+started=$(date +%s)
	101	+systemctl start openhub
	102	+
	103	+for _ in $(seq 20); do
	104	+ sleep 1
	105	+ if get "$url"; then
	106	+ echo "$new_version is running ($url)."
	107	+ fingerprint=$(journalctl -u openhub --since "@$started" -o cat --no-pager 2>/dev/null \| grep -o 'SHA256:[A-Za-z0-9+/]*' \| head -n 1 \|\| true)
	108	+ if [ -n "$fingerprint" ]; then echo "SSH host key fingerprint: $fingerprint"; fi
	109	+ echo "Settings: $defaults"
	110	+ echo "Logs: journalctl -u openhub -f"
	111	+ exit 0
	112	+ fi
	113	+done
	114	+
	115	+echo "$new_version failed its health check at $url." >&2
	116	+journalctl -u openhub -n 20 --no-pager >&2 \|\| true
	117	+if [ -f "$dest.prev" ]; then
	118	+ put "$dest.prev"
	119	+ systemctl restart openhub
	120	+ echo "Restored the previous binary ($("$dest" --version))." >&2
	121	+ if [ -n "$snap" ]; then echo "If it can't open the database, stop the service and copy $snap back to $data." >&2; fi
	122	+fi
	123	+exit 1

+46 −0 packaging/linux/openhub.service added View file

		@@ -0,0 +1,46 @@
	1	+# systemd unit for openhub, installed by packaging/linux/install.sh
	2	+# (scripts/deploy-server.sh runs it for you). Serves the web interface on
	3	+# loopback for a TLS-terminating proxy or tunnel on the same machine, and SSH
	4	+# on every interface. Set OPENHUB_* variables (at least OPENHUB_PUBLIC_URL) in
	5	+# /etc/default/openhub, then restart the service.
	6	+[Unit]
	7	+Description=Openhub git server
	8	+After=network-online.target
	9	+Wants=network-online.target
	10	+# Keep retrying forever, e.g. while a disk is still mounting after a power cut.
	11	+StartLimitIntervalSec=0
	12	+
	13	+[Service]
	14	+User=openhub
	15	+Group=openhub
	16	+Environment=OPENHUB_DATA_DIR=/var/lib/openhub
	17	+Environment=OPENHUB_HTTP_ADDR=127.0.0.1:3000
	18	+Environment=OPENHUB_SSH_ADDR=0.0.0.0:2222
	19	+# Make git fsync objects and refs, so a power cut just after a push can't leave
	20	+# a ref pointing at an object that never reached the disk.
	21	+Environment=GIT_CONFIG_COUNT=1 GIT_CONFIG_KEY_0=core.fsync GIT_CONFIG_VALUE_0=committed
	22	+# The journal doesn't render colour codes.
	23	+Environment=NO_COLOR=1
	24	+EnvironmentFile=-/etc/default/openhub
	25	+ExecStart=/usr/local/bin/openhub
	26	+Restart=always
	27	+RestartSec=2
	28	+StateDirectory=openhub
	29	+StateDirectoryMode=0750
	30	+
	31	+NoNewPrivileges=yes
	32	+ProtectSystem=strict
	33	+ProtectHome=yes
	34	+PrivateTmp=yes
	35	+PrivateDevices=yes
	36	+ProtectKernelTunables=yes
	37	+ProtectKernelModules=yes
	38	+ProtectControlGroups=yes
	39	+RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
	40	+RestrictNamespaces=yes
	41	+LockPersonality=yes
	42	+MemoryDenyWriteExecute=yes
	43	+SystemCallArchitectures=native
	44	+
	45	+[Install]
	46	+WantedBy=multi-user.target

+44 −0 scripts/deploy-server.sh added View file

		@@ -0,0 +1,44 @@
	1	+#!/usr/bin/env bash
	2	+# Builds openhub on a Linux machine and installs or updates it there as a
	3	+# systemd service (packaging/linux), over ssh. It builds on the target instead
	4	+# of cross-compiling, because libgit2 and SQLite are compiled from C. The
	5	+# target needs a C compiler, curl and git. If it has no cargo, the first run
	6	+# installs Rust there with rustup. The ssh user needs sudo.
	7	+#
	8	+# scripts/deploy-server.sh user@host
	9	+#
	10	+# Uncommitted changes are deployed too. The source and build output are kept on
	11	+# the target in ~/.cache/openhub-build, so later builds only recompile openhub.
	12	+set -euo pipefail
	13	+
	14	+host=${1:?usage: scripts/deploy-server.sh user@host}
	15	+cd "$(dirname "$0")/.."
	16	+version=$(git describe --tags --always --dirty 2>/dev/null \|\| echo dev)
	17	+build=.cache/openhub-build # relative to the remote home directory
	18	+
	19	+echo "Uploading openhub $version to $host:~/$build/src"
	20	+# Tracked files plus untracked ones that aren't ignored, skipping deleted files.
	21	+# tar -m stamps them with the extraction time, so cargo always rebuilds openhub
	22	+# itself (whatever the clock skew between the machines) and reuses dependencies.
	23	+git ls-files -z --cached --others --exclude-standard \|
	24	+ while IFS= read -r -d '' f; do if [ -f "$f" ]; then printf '%s\0' "$f"; fi; done \|
	25	+ COPYFILE_DISABLE=1 tar --null -T - -czf - \|
	26	+ ssh "$host" "rm -rf $build/src && mkdir -p $build/src && tar -xzmf - -C $build/src"
	27	+
	28	+echo "Building on $host (the first build takes several minutes)"
	29	+ssh "$host" bash -s -- "$build" <<'EOF'
	30	+set -euo pipefail
	31	+build=$1
	32	+export PATH=$HOME/.cargo/bin:$PATH
	33	+if ! command -v cargo >/dev/null; then
	34	+ echo "Installing Rust with rustup"
	35	+ curl -sSf --proto '=https' --tlsv1.2 https://sh.rustup.rs \| sh -s -- -y --profile minimal --no-modify-path
	36	+fi
	37	+cd "$build/src"
	38	+CARGO_TARGET_DIR=$HOME/$build/target cargo build --release --locked
	39	+EOF
	40	+
	41	+# A terminal lets sudo ask for a password.
	42	+tty=
	43	+[ -t 0 ] && tty=-t
	44	+ssh $tty "$host" sudo bash "$build/src/packaging/linux/install.sh" "$build/target/release/openhub" "$build/src/packaging/linux/openhub.service"