Rotating the tokens behind 21 mirrors

On 12 October I rotated the credentials on both sides of the mirrors between Forgejo and GitHub. Twenty-one repositories. Each one has a push mirror from Forgejo out to GitHub, and the relay VPS keeps a pull mirror of each one coming back. The push side was still on the GitHub token from the migration. The pull side still had a Forgejo token embedded in the clone URLs. Three old tokens had also shown up in session transcripts that same day.

The rule was no gap. Create the new mirror, sync it, verify it, and only then delete the old one. Revoke last. While the old token is still valid, a mirror removed by mistake can be created again. After revocation, the only move is a new token and a repeat of that step.

The token shape

The push mirrors authenticate as a bot account that collaborates on the GitHub repositories. A fine-grained personal access token cannot reach repositories that account only collaborates on. I confirmed that on 12 October, so the replacement had to be a classic token.

The old classic token was missing the workflow scope. GitHub rejects a mirrored push that changes .github/workflows/ when that scope is absent. Four of the twenty-one mirrors had been failing for that reason.

The new token is a classic personal access token on that bot account, scopes repo and workflow, expiry 90 days. The master copy is in Proton Pass. Once Forgejo holds the token inside each push-mirror config, the workstation keeps no copy.

Forgejo needed two more tokens, created by hand and stored in Proton Pass with the GitHub one. An owner token with write:repository creates and deletes push mirrors. It is typed at a prompt and not left on disk. A second token, read:repository only, is what the relay VPS uses to pull. I checked those scope names against Forgejo 16.0.5 first. The mirror routes require the repo owner or a site admin. A collaborator token with only write access gets a 403.

Create, then prove it, then delete

Each new push mirror is HTTPS, aimed at the GitHub repository, interval 8h0m0s, sync_on_commit on, and no branch filter, so every branch goes across. List and create responses leave the password out. The script reads every token with a hidden prompt. The header and the JSON body sit in a mode 700 temporary directory that an exit trap removes. Tokens stay off the command line and out of any repository.

Create returns the new remote name, a successful sync returns 200, and delete returns 204. A failed create, a failed verify, or a response that does not parse stops before the delete. The old mirror stays, including when the process dies after the new mirror already exists.

A real run also stops, before any change, when a repository does not already have exactly one GitHub mirror. Finished and untouched repositories look the same to the script, one mirror either way, so a resume has to omit names that already finished. The dry run prints the plan, sends nothing, logs the addresses when the count is wrong, and exits non-zero at the end if any repository was off.

Forgejo 16 formats a unix time of zero in the server timezone, so a non-UTC host emits an offset timestamp and a check for 1970-01-01T00:00:00Z misses it. A sync counts only when last_error is empty and last_update is at or after the time taken before the create. Null, unparseable, zero, and older values stay pending, and the old mirror stays.

Before either script touched a live host I checked them against the Forgejo 16.0.5 source, rewrote what disagreed, and recorded them as safe to run. Both passed bash -n and shellcheck 0.11.0.

The call that returns 500

push_mirrors-sync is one call per repository. It syncs every push mirror on that repository, and the handler returns HTTP 500 when any one of them is rejected. The script logs that status and then reads the new mirror directly. Only the new mirror’s own check decides whether the old one is deleted.

The four repositories missing workflow on the old token hit this. The new mirror was created. The sync trigger returned 500 because the old mirror was rejected, so the new mirror was not pushed inside that call. Forgejo’s queue pushed each new mirror on its own, two to eight minutes later, with no error. Waiting 300 seconds did not cover that every time. The script stopped, and the old mirror was still there.

What worked was to leave it stopped, wait until the new mirror showed a last_update after its created time and an empty last_error, compare refs, and delete the old mirror by hand. The comparison is git ls-remote on Forgejo against git ls-remote on GitHub, pull-request refs excluded, both sides sorted. Identical output is the pass. A known-broken repository can be run alone with a 10 second wait, so the stop does not hold the others up. That wait is POLL_SECS, and it is overridable for this reason.

Every repository ended with exactly one GitHub mirror, no last_error set, and a ref diff that matched on all 21. The four that had been failing mirror again.

The line I keep is a count and a flag: how many mirrors, and whether each last_error is set. One mirror and a false flag is the pass. The error text stays off the terminal, because a mirror error that echoes the token is how one of these values landed in a transcript. A true flag gets read on the Forgejo mirror page in a browser.

The read-only token on the relay VPS

Each of the twenty-one clones on the relay VPS had the migration token embedded in its URL. The replacement is the read-only token, typed at a prompt on that host and stored at ~/.config/forgejo-mirror/credentials, mode 600, in a mode 700 directory. The clone URL has no token in it. Git’s store helper attaches the credential when a fetch runs.

The new line is written in the temporary directory and tested with ls-remote against the first repository before the store is installed. A bad token changes nothing. Each clone is then tested with ls-remote on its token-free URL before its helper or origin changes. A failed test leaves that clone alone. A later update failure keeps the token-free URL and the store entry that already passed, and the row says failed. Restoring the embedded URL would write the token into git config again. Any failed row exits non-zero, and a second run retests every row. The leftover count matches any URL that still carries userinfo, including a bare token with no username. Matching only user:secret@ would report zero while a bare token was still in a config, a log, or FETCH_HEAD.

I copied the script to the relay VPS by hand, ran it, and removed it from /tmp. It reported 21 clones ok and zero files still holding an embedded credential. SSH there uses the host alias, and the /etc/hosts entry for the Forgejo host has to be present first, because MagicDNS from that machine is unreliable. The pull timers fire at 04, 10, 16, and 22 hours, two of them on a separate schedule from the migration record. After a run the sync log has to be clear of fatal, error, and Authentication failed.

Revoke last

Nothing old was revoked until both checks were in. Each push mirror was the only GitHub mirror on its repository, with no error set, and the refs matched on all 21. The pull side was 21 clones ok and a leftover count of 0.

On 12 October I revoked the old classic GitHub token, the two unused fine-grained GitHub tokens on the same bot account, and the Forgejo token that had been created to make repositories during the migration. That last one was the token embedded in the outpost clone URLs until the pull-mirror script ran. It could write repositories and the user. Leftover token files were checked and cleared. After the revocation, one repository’s push-mirror sync still succeeded, and the outpost pull timers still succeeded. An old URL in sync.log or FETCH_HEAD is cleared after those counts are written down.

Three credentials remain. The GitHub token behind the push mirrors expires on 10 January 2027. The read-only pull-mirror token stays on the relay VPS, and the workstation token for ordinary Forgejo work stays too. The owner token from this run was deleted when the run finished. The next rotation creates a fresh one at the start, before that January expiry.

What I would do differently

I would run the four known-broken mirrors on their own before the rest. The sync-all call fails for the whole repository while GitHub is still rejecting the old mirror, and a short wait on those four keeps that timeout off the healthy ones. The runbook says that now.

I would also judge a new mirror by last_update against created, with last_error empty. The queue took up to eight minutes, and a five-minute poll was short. And I would keep error text out of the terminal from the first check. A flag is enough. The token belongs in Proton Pass and in the mirror config Forgejo stores.

Create, sync, verify, then delete is what held. On the four broken mirrors the trigger failed, the old mirror stayed, and the refs caught up before anything was deleted.