Sam Jeston
History
| 1 | # Operations |
| 2 | |
| 3 | This page covers running an Openhub instance day to day: first start, reverse proxies, backups, upgrades, administration without a UI, and troubleshooting. The [project README](../README.md) has installation steps and the full [configuration table](../README.md#configuration). |
| 4 | |
| 5 | ## First start checklist |
| 6 | |
| 7 | 1. **Set `OPENHUB_PUBLIC_URL`** to the exact URL people will use, such as `https://git.example.com`. Links, HTTP clone URLs, push hints, the cross-origin check and the cookie `Secure` flag all depend on it. |
| 8 | 2. **Set the SSH host and port** with `OPENHUB_SSH_HOST` and `OPENHUB_SSH_PORT` if clients reach SSH on a different host or port from the one Openhub listens on. Docker port mapping is the usual reason. |
| 9 | 3. **Start the server and register the administrator account straight away.** The first account becomes site administrator (see [Security](security.md#known-gaps)). |
| 10 | 4. **Add an SSH key** in **Settings → SSH keys**, then check it with the command shown there (`ssh -T -p 2222 git@host`). |
| 11 | 5. **Set `OPENHUB_DISABLE_REGISTRATION=true`** once everyone who needs an account has one, unless you intend to run an open instance. |
| 12 | 6. **Note the SSH host key fingerprint** from the startup log (`SSH host key fingerprint: SHA256:…`) so users can verify it on first connect. |
| 13 | |
| 14 | ## Running under systemd |
| 15 | |
| 16 | `packaging/linux/` holds a systemd unit (`openhub.service`) and an installer (`install.sh`). `scripts/deploy-server.sh user@host` copies the checkout to the machine, builds it there and runs the installer over SSH (see the [README](../README.md#on-a-linux-server-with-systemd)). |
| 17 | |
| 18 | | Item | Where | |
| 19 | | --- | --- | |
| 20 | | Binary | `/usr/local/bin/openhub`. The previous one is kept as `openhub.prev` | |
| 21 | | Unit | `/etc/systemd/system/openhub.service` | |
| 22 | | Settings | `OPENHUB_*` variables in `/etc/default/openhub`. The first install creates it with commented examples | |
| 23 | | Data directory | `/var/lib/openhub`, owned by the `openhub` system user | |
| 24 | | Logs | `journalctl -u openhub` | |
| 25 | | Source and build cache | `~/.cache/openhub-build` in the SSH user's home on the target | |
| 26 | |
| 27 | The unit listens for the web interface on `127.0.0.1:3000` and for SSH on `0.0.0.0:2222`. Override either in `/etc/default/openhub`. |
| 28 | |
| 29 | Each run of `install.sh`: |
| 30 | |
| 31 | 1. Stops the service and copies `openhub.db` into `backups/<timestamp>/` in the data directory, keeping the last five. Repositories aren't copied. |
| 32 | 2. Swaps the binary, starts the service, and waits up to 20 seconds for the web interface to answer. |
| 33 | 3. If it doesn't answer, puts the previous binary back. If the new version had already migrated the database, restore the snapshot as described in [Upgrades](#upgrades). |
| 34 | |
| 35 | **Power cuts.** The unit restarts the server whenever it exits and never stops retrying (`Restart=always`, `StartLimitIntervalSec=0`). It also sets git's `core.fsync=committed` through `GIT_CONFIG_*` variables, so a push's objects and refs are on disk before git reports success. SQLite runs in WAL mode with `synchronous=NORMAL`, so a power cut can lose the last few database writes but doesn't corrupt the database. |
| 36 | |
| 37 | **Sandboxing.** The unit uses `ProtectSystem=strict`, so the server can only write to `/var/lib/openhub`. If you move the data directory, also allow it with `ReadWritePaths=` in a drop-in (`sudo systemctl edit openhub`). |
| 38 | |
| 39 | ## Logs |
| 40 | |
| 41 | Openhub logs to stdout through `tracing`. Set the level with `RUST_LOG`: |
| 42 | |
| 43 | | `RUST_LOG` | Shows | |
| 44 | | --- | --- | |
| 45 | | unset (`openhub=info`) | Startup, SSH sign-ins, git commands run over SSH, and errors | |
| 46 | | `openhub=debug` | Plus SSH session errors | |
| 47 | | `openhub=info,tower_http=debug` | Plus one line for every HTTP request and response | |
| 48 | |
| 49 | Internal errors (HTTP 500) are logged with their full error chain. The user only sees a generic message. |
| 50 | |
| 51 | ## Reverse proxy |
| 52 | |
| 53 | Openhub speaks plain HTTP. Put a TLS-terminating proxy in front of it for the web port only; SSH goes straight to the SSH port. |
| 54 | |
| 55 | **Caddy:** |
| 56 | |
| 57 | ``` |
| 58 | git.example.com { |
| 59 | reverse_proxy 127.0.0.1:3000 |
| 60 | } |
| 61 | ``` |
| 62 | |
| 63 | **nginx:** |
| 64 | |
| 65 | ```nginx |
| 66 | server { |
| 67 | listen 443 ssl; |
| 68 | server_name git.example.com; |
| 69 | # ssl_certificate … |
| 70 | |
| 71 | client_max_body_size 64m; # git fetch negotiation for large repositories |
| 72 | location / { |
| 73 | proxy_pass http://127.0.0.1:3000; |
| 74 | proxy_set_header Host $host; |
| 75 | proxy_buffering off; # stream clones and archive downloads |
| 76 | } |
| 77 | } |
| 78 | ``` |
| 79 | |
| 80 | **Cloudflare Tunnel** (an ingress rule in the `cloudflared` config, then `cloudflared tunnel route dns <tunnel> git.example.com`): |
| 81 | |
| 82 | ```yaml |
| 83 | ingress: |
| 84 | - hostname: git.example.com |
| 85 | service: http://127.0.0.1:3000 |
| 86 | - service: http_status:404 |
| 87 | ``` |
| 88 | |
| 89 | A tunnel carries only the web interface. Clients still need to reach the SSH port another way, for example on the local network or through a forwarded port. Set `OPENHUB_SSH_HOST` to the address they use. |
| 90 | |
| 91 | Then set `OPENHUB_PUBLIC_URL=https://git.example.com`. The cross-origin check accepts form posts whose `Origin` matches either the public URL or the `Host` header, so proxies that rewrite `Host` still work. |
| 92 | |
| 93 | If you rate-limit at the proxy (recommended, see [Security](security.md#hardening-advice)), cover `POST /login`, `POST /register` and `/*/*/info/refs`. |
| 94 | |
| 95 | ## Backups |
| 96 | |
| 97 | Everything lives in the data directory (see [Architecture](architecture.md#data-directory)): |
| 98 | |
| 99 | | Item | How to back it up | |
| 100 | | --- | --- | |
| 101 | | `openhub.db` | While running: `sqlite3 openhub.db ".backup backup.db"`, which takes a consistent snapshot. When stopped: copy the file. Don't copy a live database file without its `-wal` file | |
| 102 | | `repos/` | Copying bare repositories while a push is in progress can capture a half-written state. Stop the server, use a filesystem snapshot (ZFS, Btrfs or LVM), or accept that the latest push may need repeating | |
| 103 | | `ssh_host_ed25519_key` | Copy it. Without it, every client warns about a changed host key after a restore | |
| 104 | |
| 105 | **To restore:** stop Openhub, put all three items back in one data directory, and start it again. The database and `repos/` must come from the same point in time. A repository row without its directory, or a directory without its row, causes errors (see [Data model → Invariants](data-model.md#invariants)). |
| 106 | |
| 107 | ## Upgrades |
| 108 | |
| 109 | 1. **Back up** the data directory. |
| 110 | 2. **Build or pull the new version.** From source, run `git pull && cargo build --release`. With Docker, run `docker compose build && docker compose up -d`. Under systemd, run `scripts/deploy-server.sh user@host` again: it snapshots the database itself, and puts the old binary back if the new one fails to start (see [Running under systemd](#running-under-systemd)). |
| 111 | 3. **Start it.** Pending database migrations run automatically at startup, before the servers listen. |
| 112 | |
| 113 | Migrations only go forward. To roll back, restore the backup taken in step 1 and run the old version. |
| 114 | |
| 115 | ## Repository maintenance |
| 116 | |
| 117 | Openhub never runs `git gc`. `git receive-pack` runs `git gc --auto` after pushes by default, which keeps most repositories tidy. To compact a repository by hand, or to clean up objects left by failed web merges: |
| 118 | |
| 119 | ```sh |
| 120 | git --git-dir data/repos/<id>.git gc |
| 121 | ``` |
| 122 | |
| 123 | To map a repository to its ID: |
| 124 | |
| 125 | ```sh |
| 126 | sqlite3 -readonly data/openhub.db \ |
| 127 | "SELECT r.id FROM repositories r JOIN users u ON u.id = r.owner_id WHERE u.username = 'alice' AND r.name = 'project';" |
| 128 | ``` |
| 129 | |
| 130 | Run maintenance as the same user that runs Openhub, so file ownership stays correct. In Docker: |
| 131 | |
| 132 | ```sh |
| 133 | docker compose exec openhub git --git-dir /data/repos/<id>.git gc |
| 134 | ``` |
| 135 | |
| 136 | ## Administration without a UI |
| 137 | |
| 138 | Openhub has no admin pages. Do these tasks with `sqlite3` against `openhub.db`, after taking a backup. Changes take effect immediately, and the server doesn't need restarting. |
| 139 | |
| 140 | ```sql |
| 141 | -- Make someone a site administrator (or set to 0 to remove) |
| 142 | UPDATE users SET is_admin = 1 WHERE username = 'bob'; |
| 143 | |
| 144 | -- Sign a user out everywhere |
| 145 | DELETE FROM sessions WHERE user_id = (SELECT id FROM users WHERE username = 'bob'); |
| 146 | |
| 147 | -- Revoke all of a user's SSH keys (stops them pushing) |
| 148 | DELETE FROM ssh_keys WHERE user_id = (SELECT id FROM users WHERE username = 'bob'); |
| 149 | |
| 150 | -- Transfer a repository to another user (the new owner must not already have a repository with that name) |
| 151 | UPDATE repositories SET owner_id = (SELECT id FROM users WHERE username = 'carol') |
| 152 | WHERE id = 42; |
| 153 | ``` |
| 154 | |
| 155 | **Resetting a forgotten password.** There's no reset flow. You can set a new hash made by any Argon2id tool that writes the standard PHC string format. For example, with the `argon2` command-line tool (Debian package `argon2`): |
| 156 | |
| 157 | ```sh |
| 158 | printf '%s' 'new-password' | argon2 "$(openssl rand -hex 16)" -id -e |
| 159 | # → $argon2id$v=19$m=4096,t=3,p=1$… |
| 160 | ``` |
| 161 | |
| 162 | ```sql |
| 163 | UPDATE users SET password_hash = '$argon2id$v=19$…' WHERE username = 'bob'; |
| 164 | ``` |
| 165 | |
| 166 | **Deleting a user.** `DELETE FROM users …` removes their sessions, keys, repositories and collaborations, through cascades. It fails if they authored or merged pull requests, or wrote pull request events, in other users' repositories (see [Data model → Deleting rows](data-model.md#deleting-rows)). Deleting their rows also doesn't remove their repositories' directories under `repos/`. Delete each repository in the web interface first, while signed in as an administrator. |
| 167 | |
| 168 | ## Troubleshooting |
| 169 | |
| 170 | | Symptom | Likely cause | Fix | |
| 171 | | --- | --- | --- | |
| 172 | | Forms fail with **"Cross-origin request rejected"** | The browser's origin matches neither `OPENHUB_PUBLIC_URL` nor the `Host` header the server receives | Set the public URL to exactly what's in the address bar, including scheme and port | |
| 173 | | Startup fails with **"could not run `git`"** | git isn't installed or isn't on `PATH` | Install git, or set `OPENHUB_GIT_BIN` | |
| 174 | | Startup fails with a **migration checksum or "previously applied but has been modified"** error | An already-applied file in `migrations/` was edited | Restore the original file and put the change in a new migration | |
| 175 | | **"database is locked"** errors | Another process held a write lock for more than 5 seconds, often a `sqlite3` shell with an open transaction | Close the other connection. Open shells with `-readonly` for inspection | |
| 176 | | `git push` says **"Permission denied (publickey)"** | The key isn't registered, or ssh offered a different key | Add the key in Settings. Test with `ssh -i ~/.ssh/key -o IdentitiesOnly=yes -T -p 2222 git@host` | |
| 177 | | `git push` says **"ERROR: Repository not found."** | The path is wrong, or the account can't read the repository | Check `owner/name` in the remote URL, and that the key belongs to the expected account (`ssh -T` greets you by username) | |
| 178 | | `git push` says **"You do not have permission to push to …"** | The account can read but isn't the owner or a collaborator | The owner adds the account under repository **Settings → Collaborators** | |
| 179 | | ssh warns **"REMOTE HOST IDENTIFICATION HAS CHANGED"** | `ssh_host_ed25519_key` was lost or replaced | Restore the key from backup. If it really is gone, users remove the old entry with `ssh-keygen -R '[host]:2222'` | |
| 180 | | Push to `refs/pull/…` fails with **"deny updating a hidden ref"** | Expected: Openhub manages those refs | Push to a branch instead | |
| 181 | | `git push` over HTTP fails with **403** | HTTP is read-only by design | Use the SSH remote URL | |
| 182 | | Clone URLs show **localhost** or the wrong port | `OPENHUB_PUBLIC_URL`, `OPENHUB_SSH_HOST` or `OPENHUB_SSH_PORT` isn't set | Set them and restart | |
| 183 | | Large HTTP clones fail with **413** | A proxy limits request bodies below the 64 MB Openhub allows for fetch negotiation | Raise the proxy limit (`client_max_body_size` in nginx) | |
| 184 | | A repository page shows **"Something went wrong"** | An internal error, such as a missing `repos/<id>.git` directory | Check the log for `internal error:`. Restore the directory from backup, or delete the repository | |
| 185 | | A new repository unexpectedly shows **old content** | A previous repository with the same ID left its directory behind (deletion logs `could not remove`) | Delete the new repository, remove the stale `repos/<id>.git`, then create it again | |