commit 1d4ab12fa21063f35d29cd930ee5db37521d9742
parent 1aa9821cd9b6a63a1061aae84af5eda682d0b250
Author: Arjoonn Sharma <arjoonn@midpathsoftware.com>
Date: Wed, 20 May 2026 02:46:17 +0000
First draft for webserver (!15)
Reviewed-on: https://gitea.midpathsoftware.com/midpath/jayporeci/pulls/15
Co-authored-by: Arjoonn Sharma <arjoonn@midpathsoftware.com>
Co-committed-by: Arjoonn Sharma <arjoonn@midpathsoftware.com>
Diffstat:
17 files changed, 1462 insertions(+), 388 deletions(-)
diff --git a/ARCH.md b/ARCH.md
@@ -0,0 +1,367 @@
+# JCI Architecture
+
+JCI is a local-first CI system. CI results are stored directly inside the git repository
+as regular git objects under custom refs, so they travel with the repo on push/pull.
+`git jci server` and `git jci runner` extend this with a webhook-driven, distributed
+CI execution layer — the server is a thin coordination point; all actual CI work happens
+inside runner-managed Docker containers.
+
+---
+
+## Repository layout
+
+```
+cmd/git-jci/main.go entry point, CLI dispatch
+internal/jci/
+ run.go execute CI for a commit
+ git.go low-level git plumbing helpers
+ web.go HTTP server + SPA (single-file, inline)
+ push.go push CI refs to a remote
+ pull.go fetch CI refs from a remote
+ prune.go delete old CI refs locally or on a remote
+ cron.go cron subcommands: ls, sync
+ server.go coordination server (webhook + runner poll)
+ runner.go runner (Docker job dispatch)
+www_jci/ project website (separate, not embedded in the binary)
+```
+
+---
+
+## Ref layout inside git
+
+```
+refs/jci-runs/<commit>/<run-id> every run result, including one-off manual runs
+```
+
+Every run — whether triggered manually, by cron, or by the distributed runner — gets a
+unique run ID (`<unix_timestamp>-<4_random_hex_chars>`), so multiple runs on the same
+commit are stored independently and none overwrite another.
+
+Each ref points to a git **commit** whose **tree** holds the CI artefacts:
+
+```
+status.txt "ok" | "err" | "running"
+run.output.txt combined stdout+stderr from .jci/run.sh
+index.html standalone HTML view of the run (generated only if the job
+ did not produce its own index.html)
+<anything else> files written by the CI script to $JCI_OUTPUT_DIR
+```
+
+Because these are normal git objects, `git push origin 'refs/jci-runs/*:refs/jci-runs/*'`
+is all it takes to share results with a team.
+
+---
+
+## CI execution flow (`git jci run`)
+
+```
+1. GetCurrentCommit() resolve HEAD → commit hash
+2. generateRunID() timestamp + 2 random bytes → unique run ID
+3. check .jci/run.sh exists
+4. mkdir .jci/<commit>/ temporary output directory
+5. write status.txt = "running"
+6. exec bash .jci/run.sh working dir = output dir
+ env: JCI_COMMIT, JCI_REPO_ROOT, JCI_OUTPUT_DIR
+ stdout+stderr captured to run.output.txt
+7. write status.txt = "ok"|"err"
+8. generateIndexHTML() only if index.html was NOT written by the job
+9. StoreTree() git hash-object each file → git mktree → git commit-tree
+ → git update-ref refs/jci-runs/<commit>/<runID>
+10. rm -rf .jci/<commit>/ clean up temp dir
+```
+
+The CI script receives three environment variables and should write any extra
+artefacts into `$JCI_OUTPUT_DIR`. Whatever exists in that directory when the
+script exits is committed to git.
+
+---
+
+## Web UI (`git jci web [port]`)
+
+A minimal three-panel SPA served entirely from a single Go handler.
+No external assets; the HTML/CSS/JS is embedded inline in `showMainPage()`.
+
+```
+GET /api/branches list local branch names
+GET /api/commits?branch=&page= paginated commit list with CI status per commit
+GET /api/commit/<hash> commit detail + file list (latest run for that commit)
+GET /api/commit/<hash>/<runId> same but for a specific run
+
+GET /jci/<commit>/<runId>/<file>/raw serve file from refs/jci-runs/<commit>/<runId>
+GET /jci/<commit>/<file>/raw serve file from the latest run for <commit>
+GET /jci/... (other) serve SPA shell (JS handles routing)
+GET / serve SPA shell
+```
+
+The UI keeps a client-side page index for infinite-scroll commit loading
+(`commitsPageSize = 100` per page).
+
+---
+
+## Cron integration (`git jci cron`)
+
+**`cron ls`** — shows what is configured in `.jci/crontab` and what is currently
+installed in the user's system crontab.
+
+**`cron sync`** — idempotent sync from `.jci/crontab` → system crontab:
+
+```
+1. parse .jci/crontab 5-field schedule + optional branch:X name:Y
+2. crontab -l read current system crontab
+3. strip lines containing # JCI:<sha256(repoRoot)>
+4. append new lines, one per entry:
+ <schedule> cd <repoRoot> && [git checkout <branch> &&] git-jci run # JCI:<id> [<name>]
+5. crontab - install new crontab
+```
+
+Each repo is identified by `sha256(absolute_path)` so entries from different
+repos never collide.
+
+---
+
+## Push / Pull
+
+```
+git jci push [remote] discover all local refs/jci-runs/* not on remote → git push each
+git jci pull [remote] git fetch refs/jci-runs/*:refs/jci-runs/*
+```
+
+---
+
+## Prune
+
+Removes CI refs to reclaim space. Works on local repo or a remote.
+
+```
+git jci prune [--older-than=<duration>] [--commit]
+git jci prune --on-remote=<remote> [--older-than=<duration>] [--commit]
+```
+
+Duration format: `30d`, `2w`, `6m`, `4h`, or any Go duration string.
+Without `--commit` the command is a dry run.
+After local deletion it runs `git gc --prune=now` to actually free objects.
+
+---
+
+## Distributed CI (`git jci server` / `git jci runner`)
+
+Both commands are added to the existing `git-jci` binary and run in the foreground,
+managed by Docker or systemd.
+
+```
+Gitea ──webhook──▶ jci server ◀──poll── jci runner (Docker container)
+ │ │
+ │ Gitea API │ docker socket
+ ▼ ▼
+ Gitea job container
+ (set status) (git-jci run → git-jci push → Gitea)
+```
+
+### Server (`git jci server`)
+
+#### Configuration (env vars)
+
+```
+GITEA_HOST=gitea.example.com
+GITEA_USER=gitea_service_user
+GITEA_TOKEN=<token> # must have permission to create/delete scoped tokens
+GITEA_WEBHOOK_SECRET=<hmac secret>
+RUNNER_SECRET=<shared secret for runners>
+JCI_MAX_JOBS=4 # max concurrent jobs assignable to a single runner
+JCI_JOB_TIMEOUT=60m # server-wide job timeout (default: 60 minutes)
+```
+
+#### SQLite schema
+
+```sql
+-- Known runners; auto-created on first poll.
+CREATE TABLE runners (
+ runner_id TEXT PRIMARY KEY,
+ last_seen DATETIME NOT NULL
+);
+
+-- Active (pending or running) jobs only.
+-- Completed/timed-out jobs are deleted immediately.
+CREATE TABLE jobs (
+ job_id TEXT PRIMARY KEY, -- UUID
+ repo_owner TEXT NOT NULL,
+ repo_name TEXT NOT NULL,
+ commit_sha TEXT NOT NULL, -- idempotency key
+ runner_id TEXT, -- NULL = unassigned
+ assigned_at DATETIME,
+ expires_at DATETIME, -- assigned_at + JCI_JOB_TIMEOUT
+ gitea_token TEXT NOT NULL, -- one-time scoped token for this job
+ status_cache TEXT, -- last known status ("pending"|"running"|"success"|"failure")
+ cache_until DATETIME -- when status_cache expires (15s TTL)
+);
+```
+
+#### Startup check
+
+On startup the server verifies that `GITEA_TOKEN` has permission to create and delete
+per-repo tokens via the Gitea API. If the check fails, the server exits with a clear
+error. No silent degradation.
+
+#### Webhook handling (`POST /webhook`)
+
+1. Verify HMAC-SHA256 signature using `GITEA_WEBHOOK_SECRET`. Return 400 on failure.
+2. Extract `repo.owner`, `repo.name`, `commit_sha` (from `after` field on push events).
+3. Check `jobs` table for an existing active job with the same `commit_sha`. If found,
+ respond 200 and drop — no duplicate jobs.
+4. Insert a new job row with `runner_id = NULL`, `status_cache = "pending"`.
+5. Respond 200 immediately.
+
+#### Runner poll endpoint (`POST /poll`)
+
+Request body: `{ "runner_id": "...", "secret": "..." }`
+
+1. Verify `secret == RUNNER_SECRET`. Return 403 on failure.
+2. Upsert runner row (`runner_id`, `last_seen = now()`). This is auto-registration.
+3. Count active jobs already assigned to this runner.
+ - If count >= `JCI_MAX_JOBS`: respond 429 with `Retry-After: 5`.
+4. Poll Gitea for cached status of each assigned job (see *Status polling* below).
+ Completed jobs are deleted from the DB.
+5. Pick one unassigned job from the queue (FIFO per-repo, round-robin across repos).
+ If none: respond 200 with `{ "job": null }`.
+6. For the selected job:
+ a. Create a fresh scoped Gitea token (scoped to that repo, expiry = `now + JCI_JOB_TIMEOUT`).
+ b. Set `runner_id`, `assigned_at`, `expires_at`, store token in `jobs.gitea_token`.
+ c. Set Gitea commit status to `"pending"`.
+7. Respond 200 with the job payload:
+
+```json
+{
+ "job": {
+ "job_id": "...",
+ "clone_url": "https://<user>:<token>@gitea.example.com/owner/repo",
+ "commit_sha": "...",
+ "repo_owner": "...",
+ "repo_name": "..."
+ }
+}
+```
+
+#### Status polling (server → Gitea)
+
+The server checks job status by reading `refs/jci-runs/<commit>/<runID>` on Gitea and
+inspecting `status.txt` in that ref's tree.
+
+- Results are cached per-job for **15 seconds** (`jobs.cache_until`).
+- A check is triggered every time the assigned runner calls `/poll`.
+- A final check is performed at `jobs.expires_at` (60-minute timeout); if still
+ unresolved, the job is deleted and Gitea status is set to `"failure"`.
+- On `status.txt = "ok"` or `"err"` the server:
+ 1. Sets Gitea commit status to `"success"` or `"failure"`.
+ 2. Deletes the one-time token via the Gitea API.
+ 3. Deletes the job row from SQLite.
+
+#### Token cleanup
+
+Two-layer cleanup for the one-time Gitea token:
+1. **Gitea-native expiry**: token created with hard expiry at `assigned_at + 60m`.
+2. **Server-side deletion**: explicit delete via Gitea API on job completion or timeout.
+
+This ensures the token cannot be used beyond 60 minutes even if the server crashes.
+
+---
+
+### Runner (`git jci runner`)
+
+#### Configuration (env vars)
+
+```
+JCI_SERVER=https://jci.example.com
+JCI_RUNNER_SECRET=<shared secret>
+```
+
+#### Persistent state
+
+A SQLite database at a fixed path on a mounted volume:
+
+```sql
+CREATE TABLE identity (
+ runner_id TEXT PRIMARY KEY -- generated once, reused across restarts
+);
+```
+
+#### Poll loop
+
+```
+every 5s + random jitter (0–2s):
+ POST /poll → { runner_id, secret }
+ if 429: wait Retry-After seconds, then continue
+ if job == null: continue
+ if job != null: dispatch(job) # non-blocking; runner returns to poll immediately
+```
+
+#### Job dispatch
+
+For each received job the runner:
+
+1. **Pulls the `git-jci` binary** from a known location (e.g. mounted at
+ `/usr/local/bin/git-jci` in the runner container) to inject into the job container.
+2. **Starts a detached job container**:
+ ```
+ docker run --rm -d \
+ -v /usr/local/bin/git-jci:/usr/local/bin/git-jci:ro \
+ -e JCI_COMMIT=<commit_sha> \
+ --label jci-job=y \
+ --label jci-job-timeout=60m \
+ <image from job config> \
+ /bin/sh -c "
+ git clone --depth=1 --branch <commit> <clone_url> /repo &&
+ cd /repo &&
+ git-jci run
+ git-jci push # always runs, even if run failed, to commit status.txt
+ "
+ ```
+ `clone_url` embeds the one-time credentials, so `git push` (via `git-jci push`)
+ is transparent — `origin` is already set correctly by the clone.
+3. **Tracks the container ID** in memory (not persisted — runner crash means the
+ container runs to completion or is reaped by Docker's own restart policy).
+
+The job container never talks to the `jci server` directly.
+
+#### Container reaping
+
+The runner periodically checks for containers labelled `jci-job=y` that have been
+running longer than the configured timeout and kills them via `docker rm -f`. This
+prevents capacity leaks if a job container hangs.
+
+---
+
+## End-to-end distributed flow
+
+```
+1. Dev pushes to Gitea
+2. Gitea sends webhook → POST /webhook on jci server
+3. Server verifies HMAC, deduplicates on commit SHA, inserts job (status=pending)
+4. Runner polls → POST /poll
+5. Server creates scoped one-time Gitea token, assigns job, sets Gitea status = "pending"
+6. Server responds with job payload (clone URL with embedded token)
+7. Runner starts job container (detached), returns to poll loop
+8. Job container: clone → git-jci run → git-jci push (pushes refs/jci-runs/* to Gitea)
+9. Runner polls again (5s + jitter)
+10. Server checks Gitea for status.txt in refs/jci-runs/<commit>/<runID> (15s cache)
+11. Server sees "ok"/"err" → sets Gitea commit status → deletes token → deletes job row
+12. Runner poll returns 429 if JCI_MAX_JOBS reached; backs off via Retry-After
+```
+
+---
+
+## Key design constraints
+
+- **No external dependencies** — pure Go stdlib + git CLI + SQLite (driver only for
+ server/runner). No separate storage service.
+- **Results live in the repo** — CI artefacts are normal git objects under
+ `refs/jci-runs/*`; they travel with the repo on push/pull.
+- **Runner is pull-only** — server never initiates contact with a runner; no runner
+ address is stored.
+- **All Gitea API interaction is server-only** — runner uses only plain `git` commands
+ (clone/push); the coordination layer is opaque to it.
+- **One-time credentials per job** — scoped to one repo, expire in 60 minutes via both
+ Gitea-native expiry and explicit server-side deletion.
+- **Stateless job containers** — no volumes; crash of a container loses only that run.
+- **Artefacts are arbitrary files** — anything written to `$JCI_OUTPUT_DIR` is stored.
+- **Single binary** — `git-jci` placed on `$PATH` becomes available as `git jci`.
+- **Server runs in foreground** — managed by Docker or systemd; no self-daemonization.
diff --git a/README.md b/README.md
@@ -9,6 +9,10 @@
- [Scheduling with crontab](#scheduling-with-crontab)
- [Environment Vars](#environment-vars)
- [Example workflow](#example-workflow)
+- [Distributed CI (server + runner)](#distributed-ci-server-runner)
+ - [`git jci server`](#git-jci-server)
+ - [`git jci runner`](#git-jci-runner)
+ - [End-to-end flow](#end-to-end-flow)
- [How it works](#how-it-works)
- [Viewing results](#viewing-results)
- [Sharing results with teammates](#sharing-results-with-teammates)
@@ -149,6 +153,89 @@ git jci cron sync # install/update entries in the system crontab
You can also trigger `git jci run` automatically via [git hooks](https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks) so CI runs on every commit without any manual step.
+## Distributed CI (server + runner)
+
+`git jci server` and `git jci runner` add webhook-driven, distributed CI on top of
+the local workflow. The server is a thin coordination layer; all actual CI work runs
+inside runner-managed Docker containers.
+
+```
+Gitea ──webhook──▶ jci server ◀──poll── jci runner (Docker / systemd)
+ │ │
+ │ Gitea API │ docker socket
+ ▼ ▼
+ Gitea job container
+ (set status) (git-jci run → git-jci push)
+```
+
+### `git jci server`
+
+Runs in the foreground (managed by Docker or systemd). Receives Gitea webhooks,
+queues jobs in a local SQLite database, and hands them to runners on demand.
+
+**Environment variables**
+
+| Variable | Description |
+|---|---|
+| `GITEA_HOST` | Gitea hostname (e.g. `gitea.example.com`) |
+| `GITEA_USER` | Service account username |
+| `GITEA_TOKEN` | Token with permission to create/delete scoped tokens |
+| `GITEA_WEBHOOK_SECRET` | HMAC secret shared with the Gitea webhook |
+| `RUNNER_SECRET` | Shared secret runners must present when polling |
+| `JCI_MAX_JOBS` | Max concurrent jobs per runner (default: `4`) |
+| `JCI_JOB_TIMEOUT` | Server-wide job timeout (default: `60m`) |
+
+**Endpoints**
+
+| Method + path | Description |
+|---|---|
+| `POST /webhook` | Receives Gitea push events; verifies HMAC, enqueues job |
+| `POST /poll` | Runner poll: returns next job or `null`; auto-registers new runners |
+
+On startup the server verifies that `GITEA_TOKEN` can create and delete per-repo
+tokens. If the check fails it exits immediately with a clear error.
+
+Each job gets a short-lived, repo-scoped Gitea token (hard expiry at
+`assigned_at + JCI_JOB_TIMEOUT`, plus explicit server-side deletion on completion).
+
+### `git jci runner`
+
+Runs in the foreground alongside a Docker daemon. Polls the server every 5 s
+(+ 0–2 s random jitter) and launches one Docker container per job.
+
+**Environment variables**
+
+| Variable | Description |
+|---|---|
+| `JCI_SERVER` | Server URL (e.g. `https://jci.example.com`) |
+| `JCI_RUNNER_SECRET` | Shared secret matching `RUNNER_SECRET` on the server |
+
+The runner stores its identity in a SQLite database on a mounted volume so the
+same runner ID is reused across restarts.
+
+**Job execution** — for each job the runner:
+
+1. Starts a detached Docker container that clones the repo and runs `git-jci run`
+ then `git-jci push` (always, even on failure, to commit `status.txt`).
+2. Returns to the poll loop immediately — containers run in the background.
+3. Periodically kills containers labelled `jci-job=y` that exceed the timeout,
+ preventing capacity leaks from hung jobs.
+
+The job container never contacts the server directly; all Gitea API interaction
+happens server-side.
+
+### End-to-end flow
+
+```
+1. Dev pushes → Gitea sends webhook → server enqueues job (status=pending)
+2. Runner polls → server creates scoped token, responds with job payload
+3. Runner starts job container (clone → git-jci run → git-jci push)
+4. Runner polls again; server reads status.txt from refs/jci-runs/* on Gitea
+5. Server sees ok/err → sets Gitea commit status → deletes token → removes job
+```
+
+---
+
## How it works
CI results are stored as git tree objects under the `refs/jci/` namespace (or `refs/jci-runs/` when a commit has multiple runs). This keeps them completely separate from your regular branches and tags while still being part of the repository.
diff --git a/VERSION b/VERSION
@@ -1 +1 @@
-1.0.2
+1.0.3
diff --git a/cmd/git-jci/main.go b/cmd/git-jci/main.go
@@ -34,6 +34,10 @@ func main() {
err = jci.Prune(args)
case "cron":
err = jci.Cron(args)
+ case "server":
+ err = jci.Server(args)
+ case "runner":
+ err = jci.Runner(args)
case "version", "--version", "-v":
fmt.Println("git-jci version " + version)
return
@@ -65,8 +69,16 @@ Commands:
prune Remove old CI results
cron ls List cron jobs for this repository
cron sync Sync .jci/crontab with system cron
+ server Run the coordination server (webhook + runner poll)
+ runner Run the job runner (polls server, dispatches Docker containers)
version Print the version and exit
-CI results are stored in refs/jci/<commit>.
-With --multi, results are stored in refs/jci-runs/<commit>/<runid>.`, version)
+CI results are stored in refs/jci-runs/<commit>/<runid>.
+
+Distributed CI env vars (server):
+ GITEA_HOST, GITEA_USER, GITEA_TOKEN, GITEA_WEBHOOK_SECRET,
+ RUNNER_SECRET, JCI_MAX_JOBS, JCI_JOB_TIMEOUT, JCI_LISTEN, JCI_DB
+
+Distributed CI env vars (runner):
+ JCI_SERVER, JCI_RUNNER_SECRET, JCI_RUNNER_DB, JCI_BINARY`, version)
}
diff --git a/go.mod b/go.mod
@@ -1,3 +1,17 @@
module github.com/theSage21/jaypore_ci
go 1.22
+
+require modernc.org/sqlite v1.34.5
+
+require (
+ github.com/dustin/go-humanize v1.0.1 // indirect
+ github.com/google/uuid v1.6.0 // indirect
+ github.com/mattn/go-isatty v0.0.20 // indirect
+ github.com/ncruces/go-strftime v0.1.9 // indirect
+ github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec // indirect
+ golang.org/x/sys v0.22.0 // indirect
+ modernc.org/libc v1.55.3 // indirect
+ modernc.org/mathutil v1.6.0 // indirect
+ modernc.org/memory v1.8.0 // indirect
+)
diff --git a/go.sum b/go.sum
@@ -0,0 +1,43 @@
+github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
+github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
+github.com/google/pprof v0.0.0-20240409012703-83162a5b38cd h1:gbpYu9NMq8jhDVbvlGkMFWCjLFlqqEZjEmObmhUy6Vo=
+github.com/google/pprof v0.0.0-20240409012703-83162a5b38cd/go.mod h1:kf6iHlnVGwgKolg33glAes7Yg/8iWP8ukqeldJSO7jw=
+github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
+github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
+github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY=
+github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y=
+github.com/ncruces/go-strftime v0.1.9 h1:bY0MQC28UADQmHmaF5dgpLmImcShSi2kHU9XLdhx/f4=
+github.com/ncruces/go-strftime v0.1.9/go.mod h1:Fwc5htZGVVkseilnfgOVb9mKy6w1naJmn9CehxcKcls=
+github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec h1:W09IVJc94icq4NjY3clb7Lk8O1qJ8BdBEF8z0ibU0rE=
+github.com/remyoudompheng/bigfft v0.0.0-20230129092748-24d4a6f8daec/go.mod h1:qqbHyh8v60DhA7CoWK5oRCqLrMHRGoxYCSS9EjAz6Eo=
+golang.org/x/mod v0.16.0 h1:QX4fJ0Rr5cPQCF7O9lh9Se4pmwfwskqZfq5moyldzic=
+golang.org/x/mod v0.16.0/go.mod h1:hTbmBsO62+eylJbnUtE2MGJUyE7QWk4xUqPFrRgJ+7c=
+golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.22.0 h1:RI27ohtqKCnwULzJLqkv897zojh5/DwS/ENaMzUOaWI=
+golang.org/x/sys v0.22.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
+golang.org/x/tools v0.19.0 h1:tfGCXNR1OsFG+sVdLAitlpjAvD/I6dHDKnYrpEZUHkw=
+golang.org/x/tools v0.19.0/go.mod h1:qoJWxmGSIBmAeriMx19ogtrEPrGtDbPK634QFIcLAhc=
+modernc.org/cc/v4 v4.21.4 h1:3Be/Rdo1fpr8GrQ7IVw9OHtplU4gWbb+wNgeoBMmGLQ=
+modernc.org/cc/v4 v4.21.4/go.mod h1:HM7VJTZbUCR3rV8EYBi9wxnJ0ZBRiGE5OeGXNA0IsLQ=
+modernc.org/ccgo/v4 v4.19.2 h1:lwQZgvboKD0jBwdaeVCTouxhxAyN6iawF3STraAal8Y=
+modernc.org/ccgo/v4 v4.19.2/go.mod h1:ysS3mxiMV38XGRTTcgo0DQTeTmAO4oCmJl1nX9VFI3s=
+modernc.org/fileutil v1.3.0 h1:gQ5SIzK3H9kdfai/5x41oQiKValumqNTDXMvKo62HvE=
+modernc.org/fileutil v1.3.0/go.mod h1:XatxS8fZi3pS8/hKG2GH/ArUogfxjpEKs3Ku3aK4JyQ=
+modernc.org/gc/v2 v2.4.1 h1:9cNzOqPyMJBvrUipmynX0ZohMhcxPtMccYgGOJdOiBw=
+modernc.org/gc/v2 v2.4.1/go.mod h1:wzN5dK1AzVGoH6XOzc3YZ+ey/jPgYHLuVckd62P0GYU=
+modernc.org/libc v1.55.3 h1:AzcW1mhlPNrRtjS5sS+eW2ISCgSOLLNyFzRh/V3Qj/U=
+modernc.org/libc v1.55.3/go.mod h1:qFXepLhz+JjFThQ4kzwzOjA/y/artDeg+pcYnY+Q83w=
+modernc.org/mathutil v1.6.0 h1:fRe9+AmYlaej+64JsEEhoWuAYBkOtQiMEU7n/XgfYi4=
+modernc.org/mathutil v1.6.0/go.mod h1:Ui5Q9q1TR2gFm0AQRqQUaBWFLAhQpCwNcuhBOSedWPo=
+modernc.org/memory v1.8.0 h1:IqGTL6eFMaDZZhEWwcREgeMXYwmW83LYW8cROZYkg+E=
+modernc.org/memory v1.8.0/go.mod h1:XPZ936zp5OMKGWPqbD3JShgd/ZoQ7899TUuQqxY+peU=
+modernc.org/opt v0.1.3 h1:3XOZf2yznlhC+ibLltsDGzABUGVx8J6pnFMS3E4dcq4=
+modernc.org/opt v0.1.3/go.mod h1:WdSiB5evDcignE70guQKxYUl14mgWtbClRi5wmkkTX0=
+modernc.org/sortutil v1.2.0 h1:jQiD3PfS2REGJNzNCMMaLSp/wdMNieTbKX920Cqdgqc=
+modernc.org/sortutil v1.2.0/go.mod h1:TKU2s7kJMf1AE84OoiGppNHJwvB753OYfNl2WRb++Ss=
+modernc.org/sqlite v1.34.5 h1:Bb6SR13/fjp15jt70CL4f18JIN7p7dnMExd+UFnF15g=
+modernc.org/sqlite v1.34.5/go.mod h1:YLuNmX9NKs8wRNK2ko1LW1NGYcc9FkBO69JOt1AR9JE=
+modernc.org/strutil v1.2.0 h1:agBi9dp1I+eOnxXeiZawM8F4LawKv4NzGWSaLfyeNZA=
+modernc.org/strutil v1.2.0/go.mod h1:/mdcBmfOibveCTBxUl5B5l6W+TTH1FXPLHZE6bTosX0=
+modernc.org/token v1.1.0 h1:Xl7Ap9dKaEs5kLoOQeQmPWevfnk/DM5qcLcYlA8ys6Y=
+modernc.org/token v1.1.0/go.mod h1:UGzOrNV1mAFSEB63lOFHIpNRUVMvYTc6yu1SMY/XTDM=
diff --git a/internal/jci/cron_parser.go b/internal/jci/cron_parser.go
@@ -1,77 +0,0 @@
-package jci
-
-import (
- "bufio"
- "errors"
- "fmt"
- "io"
- "os"
- "path/filepath"
- "strings"
-)
-
-// LoadCronEntries opens .jci/crontab (if it exists) and parses all entries.
-func LoadCronEntries(repoRoot string) ([]CronEntry, error) {
- cronPath := filepath.Join(repoRoot, ".jci", "crontab")
- f, err := os.Open(cronPath)
- if err != nil {
- if os.IsNotExist(err) {
- return nil, nil
- }
- return nil, err
- }
- defer f.Close()
-
- return parseCronEntries(f)
-}
-
-func parseCronEntries(r io.Reader) ([]CronEntry, error) {
- scanner := bufio.NewScanner(r)
- var entries []CronEntry
- lineNum := 0
-
- for scanner.Scan() {
- lineNum++
- raw := strings.TrimSpace(scanner.Text())
- if raw == "" || strings.HasPrefix(raw, "#") {
- continue
- }
-
- schedule, command, err := splitCronLine(raw)
- if err != nil {
- return nil, fmt.Errorf("line %d: %w", lineNum, err)
- }
-
- entries = append(entries, CronEntry{
- Line: lineNum,
- Schedule: schedule,
- Command: command,
- Raw: raw,
- })
- }
-
- if err := scanner.Err(); err != nil {
- return nil, err
- }
-
- return entries, nil
-}
-
-func splitCronLine(raw string) (string, string, error) {
- if strings.HasPrefix(raw, "@") {
- parts := strings.Fields(raw)
- if len(parts) < 2 {
- return "", "", errors.New("missing command for cron entry")
- }
- return parts[0], strings.Join(parts[1:], " "), nil
- }
-
- parts := strings.Fields(raw)
- if len(parts) < 6 {
- return "", "", errors.New("cron entry must have 5 time fields and a command")
- }
-
- schedule := strings.Join(parts[:5], " ")
- command := strings.Join(parts[5:], " ")
- return schedule, command, nil
-}
diff --git a/internal/jci/cron_types.go b/internal/jci/cron_types.go
@@ -1,87 +0,0 @@
-package jci
-
-import (
- "crypto/sha1"
- "encoding/hex"
- "fmt"
- "path/filepath"
- "strings"
-)
-
-type CronJobType string
-
-const (
- CronJobGitJCI CronJobType = "git-jci"
- CronJobBinary CronJobType = "binary"
- CronJobShell CronJobType = "shell"
-)
-
-type CronJob struct {
- ID string
- Schedule string
- Command string
- Type CronJobType
- BinaryPath string
- BinaryArgs string
- Line int
- CronLog string
-}
-
-func classifyCronCommand(command string, repoRoot string) (CronJobType, string, string) {
- trimmed := strings.TrimSpace(command)
- if trimmed == "" {
- return CronJobShell, "", ""
- }
-
- if strings.HasPrefix(trimmed, "git jci") {
- return CronJobGitJCI, "", ""
- }
-
- head, tail := splitCommandHeadTail(trimmed)
- cleaned := strings.TrimPrefix(head, "./")
- if strings.HasPrefix(cleaned, ".jci/") {
- abs := filepath.Join(repoRoot, cleaned)
- return CronJobBinary, abs, tail
- }
-
- if !strings.Contains(head, "/") {
- candidate := filepath.Join(repoRoot, ".jci", head)
- return CronJobBinary, candidate, tail
- }
-
- return CronJobShell, trimmed, tail
-}
-
-func cronJobID(schedule, command string) string {
- sum := sha1.Sum([]byte(schedule + "\x00" + command))
- return hex.EncodeToString(sum[:])
-}
-
-func (job CronJob) shellCommand(repoRoot string) string {
- var command string
- switch job.Type {
- case CronJobBinary:
- command = shellEscape(job.BinaryPath)
- if job.BinaryArgs != "" {
- command += " " + job.BinaryArgs
- }
- default:
- command = job.Command
- }
-
- full := fmt.Sprintf("cd %s && %s", shellEscape(repoRoot), command)
- if job.CronLog != "" {
- full = fmt.Sprintf("%s >> %s 2>&1", full, shellEscape(job.CronLog))
- }
- return full
-}
-
-func splitCommandHeadTail(cmd string) (string, string) {
- parts := strings.Fields(cmd)
- if len(parts) == 0 {
- return "", ""
- }
- head := parts[0]
- tail := strings.Join(parts[1:], " ")
- return head, tail
-}
diff --git a/internal/jci/git.go b/internal/jci/git.go
@@ -44,12 +44,9 @@ func StoreTree(dir string, commit string, message string, runID string) error {
return err
}
- tmpIndex := repoRoot + "/.git/jci-index"
- defer exec.Command("rm", "-f", tmpIndex).Run()
-
// We need to use git hash-object and mktree to build a tree
// from files outside the repo
- treeID, err := hashDir(dir, repoRoot, tmpIndex)
+ treeID, err := hashDir(dir, repoRoot)
if err != nil {
return fmt.Errorf("failed to hash directory: %w", err)
}
@@ -73,7 +70,7 @@ func StoreTree(dir string, commit string, message string, runID string) error {
}
// hashDir recursively hashes a directory and returns its tree ID
-func hashDir(dir string, repoRoot string, tmpIndex string) (string, error) {
+func hashDir(dir string, repoRoot string) (string, error) {
entries, err := os.ReadDir(dir)
if err != nil {
return "", err
@@ -86,7 +83,7 @@ func hashDir(dir string, repoRoot string, tmpIndex string) (string, error) {
if entry.IsDir() {
// Recursively hash subdirectory
- subTreeID, err := hashDir(path, repoRoot, tmpIndex)
+ subTreeID, err := hashDir(path, repoRoot)
if err != nil {
return "", err
}
@@ -131,18 +128,6 @@ func hashDir(dir string, repoRoot string, tmpIndex string) (string, error) {
return strings.TrimSpace(string(out)), nil
}
-// ListJCIRefs returns all refs under refs/jci/
-func ListJCIRefs() ([]string, error) {
- out, err := git("for-each-ref", "--format=%(refname:short)", "refs/jci/")
- if err != nil {
- return nil, err
- }
- if out == "" {
- return nil, nil
- }
- return strings.Split(out, "\n"), nil
-}
-
// ListJCIRunRefs returns all refs under refs/jci-runs/
func ListJCIRunRefs() ([]string, error) {
out, err := git("for-each-ref", "--format=%(refname)", "refs/jci-runs/")
@@ -154,22 +139,3 @@ func ListJCIRunRefs() ([]string, error) {
}
return strings.Split(out, "\n"), nil
}
-
-// ListAllJCIRefs returns all JCI refs (both single and multi-run)
-func ListAllJCIRefs() ([]string, error) {
- var allRefs []string
-
- // Get refs/jci/*
- out, err := git("for-each-ref", "--format=%(refname)", "refs/jci/")
- if err == nil && out != "" {
- allRefs = append(allRefs, strings.Split(out, "\n")...)
- }
-
- // Get refs/jci-runs/*
- out, err = git("for-each-ref", "--format=%(refname)", "refs/jci-runs/")
- if err == nil && out != "" {
- allRefs = append(allRefs, strings.Split(out, "\n")...)
- }
-
- return allRefs, nil
-}
diff --git a/internal/jci/prune.go b/internal/jci/prune.go
@@ -20,29 +20,28 @@ type PruneOptions struct {
func ParsePruneArgs(args []string) (*PruneOptions, error) {
opts := &PruneOptions{}
- for _, arg := range args {
- if arg == "--commit" {
+ for i := 0; i < len(args); i++ {
+ arg := args[i]
+ switch {
+ case arg == "--commit":
opts.Commit = true
- } else if strings.HasPrefix(arg, "--on-remote=") {
+ case strings.HasPrefix(arg, "--on-remote="):
opts.OnRemote = strings.TrimPrefix(arg, "--on-remote=")
- } else if strings.HasPrefix(arg, "--on-remote") {
- // Handle --on-remote origin (space separated)
- continue
- } else if strings.HasPrefix(arg, "--older-than=") {
+ case arg == "--on-remote":
+ if i+1 >= len(args) {
+ return nil, fmt.Errorf("--on-remote requires a value")
+ }
+ i++
+ opts.OnRemote = args[i]
+ case strings.HasPrefix(arg, "--older-than="):
durStr := strings.TrimPrefix(arg, "--older-than=")
dur, err := parseDuration(durStr)
if err != nil {
return nil, fmt.Errorf("invalid duration %q: %v", durStr, err)
}
opts.OlderThan = dur
- } else if !strings.HasPrefix(arg, "-") && opts.OnRemote == "" {
- // Check if previous arg was --on-remote
- for i, a := range args {
- if a == "--on-remote" && i+1 < len(args) && args[i+1] == arg {
- opts.OnRemote = arg
- break
- }
- }
+ default:
+ return nil, fmt.Errorf("unknown argument: %s", arg)
}
}
@@ -97,7 +96,7 @@ func Prune(args []string) error {
}
func pruneLocal(opts *PruneOptions) error {
- refs, err := ListJCIRefs()
+ refs, err := ListJCIRunRefs()
if err != nil {
return err
}
@@ -107,35 +106,26 @@ func pruneLocal(opts *PruneOptions) error {
return nil
}
- // Get info for all refs
var refInfos []RefInfo
var totalSize int64
fmt.Println("Scanning CI results...")
for i, ref := range refs {
- commit := strings.TrimPrefix(ref, "jci/")
printProgress(i+1, len(refs), "Scanning")
-
info := RefInfo{
Ref: ref,
- Commit: commit,
+ Commit: extractCommitFromRef(ref),
}
-
- // Get timestamp from the JCI commit
- timeStr, err := git("log", "-1", "--format=%ci", "refs/jci/"+commit)
+ timeStr, err := git("log", "-1", "--format=%ci", ref)
if err == nil {
info.Timestamp, _ = time.Parse("2006-01-02 15:04:05 -0700", timeStr)
}
-
- // Get size of the tree
- info.Size = getRefSize("refs/jci/" + commit)
+ info.Size = getRefSize(ref)
totalSize += info.Size
-
refInfos = append(refInfos, info)
}
- fmt.Println() // newline after progress
+ fmt.Println()
- // Filter refs to prune
var toPrune []RefInfo
var prunedSize int64
now := time.Now()
@@ -143,18 +133,13 @@ func pruneLocal(opts *PruneOptions) error {
for _, info := range refInfos {
shouldPrune := false
- // Check if commit still exists (original behavior)
- _, err := git("cat-file", "-t", info.Commit)
- if err != nil {
+ // Prune if the source commit no longer exists
+ if _, err := git("cat-file", "-t", info.Commit); err != nil {
shouldPrune = true
}
- // Check age if --older-than specified
- if opts.OlderThan > 0 && !info.Timestamp.IsZero() {
- age := now.Sub(info.Timestamp)
- if age > opts.OlderThan {
- shouldPrune = true
- }
+ if opts.OlderThan > 0 && !info.Timestamp.IsZero() && now.Sub(info.Timestamp) > opts.OlderThan {
+ shouldPrune = true
}
if shouldPrune {
@@ -169,7 +154,6 @@ func pruneLocal(opts *PruneOptions) error {
return nil
}
- // Show what will be pruned
fmt.Printf("\nFound %d ref(s) to prune:\n", len(toPrune))
for _, info := range toPrune {
age := ""
@@ -178,7 +162,6 @@ func pruneLocal(opts *PruneOptions) error {
}
fmt.Printf(" %s %s%s\n", info.Commit[:12], formatSize(info.Size), age)
}
-
fmt.Printf("\nTotal to free: %s (of %s total)\n", formatSize(prunedSize), formatSize(totalSize))
if !opts.Commit {
@@ -186,20 +169,18 @@ func pruneLocal(opts *PruneOptions) error {
return nil
}
- // Actually delete
fmt.Println("\nDeleting...")
deleted := 0
for i, info := range toPrune {
printProgress(i+1, len(toPrune), "Deleting")
- if _, err := git("update-ref", "-d", "refs/jci/"+info.Commit); err != nil {
- fmt.Printf("\n Warning: failed to delete %s: %v\n", info.Commit[:12], err)
+ if _, err := git("update-ref", "-d", info.Ref); err != nil {
+ fmt.Printf("\n Warning: failed to delete %s: %v\n", info.Ref, err)
continue
}
deleted++
}
- fmt.Println() // newline after progress
+ fmt.Println()
- // Run gc to actually free space
fmt.Println("Running git gc...")
exec.Command("git", "gc", "--prune=now", "--quiet").Run()
@@ -212,10 +193,8 @@ func pruneRemote(opts *PruneOptions) error {
fmt.Printf("Fetching CI refs from %s...\n", remote)
- // Get remote refs (both jci and jci-runs)
- out1, _ := git("ls-remote", remote, "refs/jci/*")
- out2, _ := git("ls-remote", remote, "refs/jci-runs/*")
- out := strings.TrimSpace(out1 + "\n" + out2)
+ out, _ := git("ls-remote", remote, "refs/jci-runs/*")
+ out = strings.TrimSpace(out)
if out == "" {
fmt.Println("No CI results on remote")
@@ -402,20 +381,11 @@ func formatAge(d time.Duration) string {
return "<1h"
}
-// extractCommitFromRef extracts the commit hash from a JCI ref
-// refs/jci/<commit> -> <commit>
-// refs/jci-runs/<commit>/<runid> -> <commit>
+// extractCommitFromRef extracts the commit hash from a refs/jci-runs/<commit>/<runid> ref
func extractCommitFromRef(ref string) string {
- if strings.HasPrefix(ref, "refs/jci-runs/") {
- // refs/jci-runs/<commit>/<runid>
- parts := strings.Split(strings.TrimPrefix(ref, "refs/jci-runs/"), "/")
- if len(parts) >= 1 {
- return parts[0]
- }
- } else if strings.HasPrefix(ref, "refs/jci/") {
- return strings.TrimPrefix(ref, "refs/jci/")
- } else if strings.HasPrefix(ref, "jci/") {
- return strings.TrimPrefix(ref, "jci/")
+ parts := strings.Split(strings.TrimPrefix(ref, "refs/jci-runs/"), "/")
+ if len(parts) >= 1 {
+ return parts[0]
}
return ref
}
diff --git a/internal/jci/pull.go b/internal/jci/pull.go
@@ -13,14 +13,7 @@ func Pull(args []string) error {
fmt.Printf("Fetching CI results from %s...\n", remote)
- // Fetch all refs/jci/* from remote
- _, err := git("fetch", remote, "refs/jci/*:refs/jci/*")
- if err != nil {
- fmt.Printf("Warning: %v\n", err)
- }
-
- // Fetch all refs/jci-runs/* from remote
- _, err = git("fetch", remote, "refs/jci-runs/*:refs/jci-runs/*")
+ _, err := git("fetch", remote, "refs/jci-runs/*:refs/jci-runs/*")
if err != nil {
fmt.Printf("Warning: %v\n", err)
}
diff --git a/internal/jci/push.go b/internal/jci/push.go
@@ -13,7 +13,7 @@ func Push(args []string) error {
}
// Get all local JCI refs
- localRefs, err := ListAllJCIRefs()
+ localRefs, err := ListJCIRunRefs()
if err != nil {
return err
}
@@ -54,12 +54,10 @@ func Push(args []string) error {
return nil
}
-// getRemoteJCIRefs returns a set of refs that exist on the remote
+// getRemoteJCIRefs returns a set of refs/jci-runs/* refs that exist on the remote
func getRemoteJCIRefs(remote string) map[string]bool {
remoteCI := make(map[string]bool)
-
- // Get refs/jci/*
- out, err := git("ls-remote", "--refs", remote, "refs/jci/*")
+ out, err := git("ls-remote", "--refs", remote, "refs/jci-runs/*")
if err == nil && out != "" {
for _, line := range strings.Split(out, "\n") {
parts := strings.Fields(line)
@@ -68,17 +66,5 @@ func getRemoteJCIRefs(remote string) map[string]bool {
}
}
}
-
- // Get refs/jci-runs/*
- out, err = git("ls-remote", "--refs", remote, "refs/jci-runs/*")
- if err == nil && out != "" {
- for _, line := range strings.Split(out, "\n") {
- parts := strings.Fields(line)
- if len(parts) >= 2 {
- remoteCI[parts[1]] = true
- }
- }
- }
-
return remoteCI
}
diff --git a/internal/jci/run.go b/internal/jci/run.go
@@ -57,9 +57,12 @@ func Run(args []string) error {
os.WriteFile(statusFile, []byte("ok"), 0644)
}
- // Generate index.html with results
- if err := generateIndexHTML(outputDir, commit, err); err != nil {
- fmt.Printf("Warning: failed to generate index.html: %v\n", err)
+ // Generate index.html only if the job didn't already produce one
+ indexPath := filepath.Join(outputDir, "index.html")
+ if _, statErr := os.Stat(indexPath); os.IsNotExist(statErr) {
+ if htmlErr := generateIndexHTML(outputDir, commit, err); htmlErr != nil {
+ fmt.Printf("Warning: failed to generate index.html: %v\n", htmlErr)
+ }
}
// Store results in git
@@ -203,7 +206,9 @@ func escapeHTML(s string) string {
func generateRunID() string {
timestamp := time.Now().Unix()
b := make([]byte, 2)
- rand.Read(b)
- randomSuffix := hex.EncodeToString(b)
- return fmt.Sprintf("%d-%s", timestamp, randomSuffix)
+ if _, err := rand.Read(b); err != nil {
+ // Extremely unlikely; fall back to timestamp-only to avoid blocking
+ return fmt.Sprintf("%d-0000", timestamp)
+ }
+ return fmt.Sprintf("%d-%s", timestamp, hex.EncodeToString(b))
}
diff --git a/internal/jci/runner.go b/internal/jci/runner.go
@@ -0,0 +1,285 @@
+package jci
+
+import (
+ "bytes"
+ "database/sql"
+ "encoding/json"
+ "fmt"
+ "io"
+ "log"
+ "math/rand"
+ "net/http"
+ "os"
+ "os/exec"
+ "strings"
+ "time"
+
+ _ "modernc.org/sqlite"
+)
+
+// Runner polls the jci server for jobs and dispatches them into Docker containers.
+//
+// Configuration env vars:
+//
+// JCI_SERVER base URL of the jci server, e.g. https://jci.example.com
+// JCI_RUNNER_SECRET shared secret matching RUNNER_SECRET on the server
+// JCI_RUNNER_DB path to runner's SQLite DB (default /var/lib/jci/runner.db)
+// JCI_BINARY path to the git-jci binary to inject into containers (default /usr/local/bin/git-jci)
+func Runner(_ []string) error {
+ cfg, err := loadRunnerConfig()
+ if err != nil {
+ return err
+ }
+
+ db, err := openRunnerDB(cfg.dbPath)
+ if err != nil {
+ return fmt.Errorf("open runner db: %w", err)
+ }
+ defer db.Close()
+
+ runnerID, err := getOrCreateRunnerID(db)
+ if err != nil {
+ return fmt.Errorf("runner identity: %w", err)
+ }
+ log.Printf("jci runner starting (id=%s)", runnerID)
+
+ r := &runner{cfg: cfg, db: db, runnerID: runnerID}
+ r.pollLoop()
+ return nil // pollLoop runs forever; return only on fatal errors
+}
+
+// ---------------------------------------------------------------------------
+// Config
+// ---------------------------------------------------------------------------
+
+type runnerConfig struct {
+ serverURL string
+ runnerSecret string
+ dbPath string
+ binaryPath string
+}
+
+func loadRunnerConfig() (runnerConfig, error) {
+ cfg := runnerConfig{
+ dbPath: "/var/lib/jci/runner.db",
+ binaryPath: "/usr/local/bin/git-jci",
+ }
+ cfg.serverURL = os.Getenv("JCI_SERVER")
+ cfg.runnerSecret = os.Getenv("JCI_RUNNER_SECRET")
+
+ if cfg.serverURL == "" || cfg.runnerSecret == "" {
+ return cfg, fmt.Errorf("missing required env vars: JCI_SERVER, JCI_RUNNER_SECRET")
+ }
+ if v := os.Getenv("JCI_RUNNER_DB"); v != "" {
+ cfg.dbPath = v
+ }
+ if v := os.Getenv("JCI_BINARY"); v != "" {
+ cfg.binaryPath = v
+ }
+ return cfg, nil
+}
+
+// ---------------------------------------------------------------------------
+// Runner SQLite identity store
+// ---------------------------------------------------------------------------
+
+func openRunnerDB(path string) (*sql.DB, error) {
+ if err := os.MkdirAll(parentDir(path), 0755); err != nil {
+ return nil, err
+ }
+ db, err := sql.Open("sqlite", path)
+ if err != nil {
+ return nil, err
+ }
+ _, err = db.Exec(`CREATE TABLE IF NOT EXISTS identity (runner_id TEXT PRIMARY KEY)`)
+ return db, err
+}
+
+func getOrCreateRunnerID(db *sql.DB) (string, error) {
+ var id string
+ err := db.QueryRow(`SELECT runner_id FROM identity LIMIT 1`).Scan(&id)
+ if err == nil {
+ return id, nil
+ }
+ if err != sql.ErrNoRows {
+ return "", err
+ }
+ id = randomID()
+ _, err = db.Exec(`INSERT INTO identity (runner_id) VALUES (?)`, id)
+ return id, err
+}
+
+// ---------------------------------------------------------------------------
+// Runner
+// ---------------------------------------------------------------------------
+
+type runner struct {
+ cfg runnerConfig
+ db *sql.DB
+ runnerID string
+}
+
+// pollLoop runs forever: poll the server every 5s + 0–2s jitter.
+// Container reaping runs in a separate goroutine on a 60s interval.
+func (r *runner) pollLoop() {
+ // Default job timeout for reaping matches the server default (60m).
+ // If JCI_JOB_TIMEOUT is set the server enforces it; we use 60m as a
+ // conservative local limit.
+ const reapInterval = 60 * time.Second
+ const containerTimeout = 65 * time.Minute // slightly longer than server timeout
+ go func() {
+ for {
+ time.Sleep(reapInterval)
+ r.reapStaleContainers(containerTimeout)
+ }
+ }()
+
+ for {
+ jitter := time.Duration(rand.Intn(2000)) * time.Millisecond
+ time.Sleep(5*time.Second + jitter)
+
+ retryAfter, job, err := r.poll()
+ if err != nil {
+ log.Printf("poll error: %v", err)
+ continue
+ }
+ if retryAfter > 0 {
+ log.Printf("runner at capacity; backing off %s", retryAfter)
+ time.Sleep(retryAfter)
+ continue
+ }
+ if job == nil {
+ continue
+ }
+ go r.dispatch(job) // non-blocking; runner returns to poll immediately
+ }
+}
+
+type serverJob struct {
+ JobID string `json:"job_id"`
+ CloneURL string `json:"clone_url"`
+ CommitSHA string `json:"commit_sha"`
+ RepoOwner string `json:"repo_owner"`
+ RepoName string `json:"repo_name"`
+}
+
+// poll contacts the server. Returns (retryAfter, job, err).
+// retryAfter > 0 means 429 was received; job may be nil when there is no work.
+func (r *runner) poll() (time.Duration, *serverJob, error) {
+ body, _ := json.Marshal(map[string]string{
+ "runner_id": r.runnerID,
+ "secret": r.cfg.runnerSecret,
+ })
+ resp, err := http.Post(r.cfg.serverURL+"/poll", "application/json", bytes.NewReader(body))
+ if err != nil {
+ return 0, nil, err
+ }
+ defer resp.Body.Close()
+
+ if resp.StatusCode == http.StatusTooManyRequests {
+ secs := 5
+ if v := resp.Header.Get("Retry-After"); v != "" {
+ fmt.Sscanf(v, "%d", &secs)
+ }
+ return time.Duration(secs) * time.Second, nil, nil
+ }
+ if resp.StatusCode != http.StatusOK {
+ data, _ := io.ReadAll(resp.Body)
+ return 0, nil, fmt.Errorf("server returned %d: %s", resp.StatusCode, data)
+ }
+
+ var result struct {
+ Job *serverJob `json:"job"`
+ }
+ if err := json.NewDecoder(resp.Body).Decode(&result); err != nil {
+ return 0, nil, fmt.Errorf("decode poll response: %w", err)
+ }
+ return 0, result.Job, nil
+}
+
+// dispatch starts a detached Docker container to run the job.
+func (r *runner) dispatch(job *serverJob) {
+ log.Printf("dispatching job %s (%s/%s@%.12s)", job.JobID, job.RepoOwner, job.RepoName, job.CommitSHA)
+
+ containerID, err := r.startJobContainer(job)
+ if err != nil {
+ log.Printf("job %s: failed to start container: %v", job.JobID, err)
+ return
+ }
+ log.Printf("job %s: container %s started", job.JobID, containerID[:12])
+
+ // Container reaping is handled by the periodic reaper; nothing more to do here.
+}
+
+// startJobContainer launches a detached container and returns its ID.
+func (r *runner) startJobContainer(job *serverJob) (string, error) {
+ // The job container:
+ // 1. Clones the repo at the target commit (credentials embedded in clone_url)
+ // 2. Runs git-jci run
+ // 3. Runs git-jci push (always, so status.txt is pushed even on failure)
+ script := strings.Join([]string{
+ fmt.Sprintf("git clone --depth=1 %s /repo", shellQuote(job.CloneURL)),
+ "cd /repo",
+ fmt.Sprintf("git checkout %s", shellQuote(job.CommitSHA)),
+ "git-jci run || true",
+ "git-jci push",
+ }, " && ")
+
+ args := []string{
+ "run", "--rm", "-d",
+ "-v", r.cfg.binaryPath + ":/usr/local/bin/git-jci:ro",
+ "-e", "JCI_COMMIT=" + job.CommitSHA,
+ "--label", "jci-job=y",
+ "--label", "jci-job-id=" + job.JobID,
+ // Use alpine as a safe default; repos can override via .jci/image file (future)
+ "alpine:3",
+ "/bin/sh", "-c", script,
+ }
+
+ cmd := exec.Command("docker", args...)
+ out, err := cmd.Output()
+ if err != nil {
+ return "", fmt.Errorf("docker run: %w", err)
+ }
+ return strings.TrimSpace(string(out)), nil
+}
+
+// ReapStaleContainers kills containers labelled jci-job=y that have been running
+// longer than the given timeout. Called from the poll loop or independently.
+func (r *runner) reapStaleContainers(timeout time.Duration) {
+ // List all containers with label jci-job=y that are still running
+ out, err := exec.Command("docker", "ps", "--filter", "label=jci-job=y", "--format", "{{.ID}}\t{{.RunningFor}}").Output()
+ if err != nil {
+ log.Printf("reaper: docker ps: %v", err)
+ return
+ }
+ lines := strings.Split(strings.TrimSpace(string(out)), "\n")
+ for _, line := range lines {
+ if line == "" {
+ continue
+ }
+ parts := strings.SplitN(line, "\t", 2)
+ if len(parts) < 1 {
+ continue
+ }
+ cid := parts[0]
+ // Inspect to get exact start time
+ inspectOut, err := exec.Command("docker", "inspect", "--format", "{{.State.StartedAt}}", cid).Output()
+ if err != nil {
+ continue
+ }
+ startedAt, err := time.Parse(time.RFC3339Nano, strings.TrimSpace(string(inspectOut)))
+ if err != nil {
+ continue
+ }
+ if time.Since(startedAt) > timeout {
+ log.Printf("reaper: killing stale container %s (running since %s)", cid[:12], startedAt)
+ exec.Command("docker", "rm", "-f", cid).Run()
+ }
+ }
+}
+
+// shellQuote wraps s in single quotes, escaping any existing single quotes.
+func shellQuote(s string) string {
+ return "'" + strings.ReplaceAll(s, "'", "'\\''") + "'"
+}
diff --git a/internal/jci/server.go b/internal/jci/server.go
@@ -0,0 +1,576 @@
+package jci
+
+import (
+ "bytes"
+ "crypto/hmac"
+ "crypto/rand"
+ "crypto/sha256"
+ "database/sql"
+ "encoding/base64"
+ "encoding/hex"
+ "encoding/json"
+ "fmt"
+ "io"
+ "log"
+ "net/http"
+ "os"
+ "strconv"
+ "strings"
+ "time"
+
+ _ "modernc.org/sqlite"
+)
+
+// Server runs the coordination server: accepts Gitea webhooks and serves runner poll requests.
+//
+// Configuration env vars:
+//
+// GITEA_HOST e.g. gitea.example.com
+// GITEA_USER service account username
+// GITEA_TOKEN admin token (must be able to create/delete scoped tokens)
+// GITEA_WEBHOOK_SECRET HMAC-SHA256 secret shared with Gitea
+// RUNNER_SECRET shared secret that runners authenticate with
+// JCI_MAX_JOBS max concurrent jobs per runner (default 4)
+// JCI_JOB_TIMEOUT server-wide job timeout (default 60m)
+// JCI_LISTEN listen address (default :8080)
+// JCI_DB SQLite DB path (default /var/lib/jci/server.db)
+func Server(_ []string) error {
+ cfg, err := loadServerConfig()
+ if err != nil {
+ return err
+ }
+
+ db, err := openServerDB(cfg.dbPath)
+ if err != nil {
+ return fmt.Errorf("open db: %w", err)
+ }
+ defer db.Close()
+
+ if err := verifyGiteaAdminToken(cfg); err != nil {
+ return fmt.Errorf("Gitea token check failed: %w", err)
+ }
+
+ srv := &coordinationServer{cfg: cfg, db: db}
+
+ mux := http.NewServeMux()
+ mux.HandleFunc("/webhook", srv.handleWebhook)
+ mux.HandleFunc("/poll", srv.handlePoll)
+
+ log.Printf("jci server listening on %s", cfg.listen)
+ return http.ListenAndServe(cfg.listen, mux)
+}
+
+// ---------------------------------------------------------------------------
+// Config
+// ---------------------------------------------------------------------------
+
+type serverConfig struct {
+ giteaHost string
+ giteaUser string
+ giteaToken string
+ webhookSecret string
+ runnerSecret string
+ maxJobsPerRunner int
+ jobTimeout time.Duration
+ listen string
+ dbPath string
+}
+
+func loadServerConfig() (serverConfig, error) {
+ cfg := serverConfig{
+ maxJobsPerRunner: 4,
+ jobTimeout: 60 * time.Minute,
+ listen: ":8080",
+ dbPath: "/var/lib/jci/server.db",
+ }
+ cfg.giteaHost = os.Getenv("GITEA_HOST")
+ cfg.giteaUser = os.Getenv("GITEA_USER")
+ cfg.giteaToken = os.Getenv("GITEA_TOKEN")
+ cfg.webhookSecret = os.Getenv("GITEA_WEBHOOK_SECRET")
+ cfg.runnerSecret = os.Getenv("RUNNER_SECRET")
+
+ missing := []string{}
+ for _, pair := range [][2]string{
+ {"GITEA_HOST", cfg.giteaHost}, {"GITEA_USER", cfg.giteaUser},
+ {"GITEA_TOKEN", cfg.giteaToken}, {"GITEA_WEBHOOK_SECRET", cfg.webhookSecret},
+ {"RUNNER_SECRET", cfg.runnerSecret},
+ } {
+ if pair[1] == "" {
+ missing = append(missing, pair[0])
+ }
+ }
+ if len(missing) > 0 {
+ return cfg, fmt.Errorf("missing required env vars: %s", strings.Join(missing, ", "))
+ }
+
+ if v := os.Getenv("JCI_MAX_JOBS"); v != "" {
+ n, err := strconv.Atoi(v)
+ if err != nil {
+ return cfg, fmt.Errorf("JCI_MAX_JOBS must be an integer: %w", err)
+ }
+ cfg.maxJobsPerRunner = n
+ }
+ if v := os.Getenv("JCI_JOB_TIMEOUT"); v != "" {
+ d, err := time.ParseDuration(v)
+ if err != nil {
+ return cfg, fmt.Errorf("JCI_JOB_TIMEOUT: %w", err)
+ }
+ cfg.jobTimeout = d
+ }
+ if v := os.Getenv("JCI_LISTEN"); v != "" {
+ cfg.listen = v
+ }
+ if v := os.Getenv("JCI_DB"); v != "" {
+ cfg.dbPath = v
+ }
+ return cfg, nil
+}
+
+// ---------------------------------------------------------------------------
+// SQLite schema
+// ---------------------------------------------------------------------------
+
+func openServerDB(path string) (*sql.DB, error) {
+ if err := os.MkdirAll(parentDir(path), 0755); err != nil {
+ return nil, err
+ }
+ db, err := sql.Open("sqlite", path)
+ if err != nil {
+ return nil, err
+ }
+ _, err = db.Exec(`
+ CREATE TABLE IF NOT EXISTS runners (
+ runner_id TEXT PRIMARY KEY,
+ last_seen DATETIME NOT NULL
+ );
+ CREATE TABLE IF NOT EXISTS jobs (
+ job_id TEXT PRIMARY KEY,
+ repo_owner TEXT NOT NULL,
+ repo_name TEXT NOT NULL,
+ commit_sha TEXT NOT NULL,
+ runner_id TEXT,
+ assigned_at DATETIME,
+ expires_at DATETIME,
+ gitea_token TEXT NOT NULL DEFAULT '',
+ status_cache TEXT,
+ cache_until DATETIME
+ );
+ `)
+ return db, err
+}
+
+// ---------------------------------------------------------------------------
+// Server
+// ---------------------------------------------------------------------------
+
+type coordinationServer struct {
+ cfg serverConfig
+ db *sql.DB
+}
+
+// ---------------------------------------------------------------------------
+// Webhook handler
+// ---------------------------------------------------------------------------
+
+func (s *coordinationServer) handleWebhook(w http.ResponseWriter, r *http.Request) {
+ if r.Method != http.MethodPost {
+ http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
+ return
+ }
+ body, err := io.ReadAll(r.Body)
+ if err != nil {
+ http.Error(w, "read error", http.StatusBadRequest)
+ return
+ }
+
+ sig := r.Header.Get("X-Gitea-Signature")
+ if !checkHMAC(body, sig, s.cfg.webhookSecret) {
+ http.Error(w, "invalid signature", http.StatusBadRequest)
+ return
+ }
+
+ var payload struct {
+ After string `json:"after"`
+ Repo struct {
+ Owner struct{ Login string } `json:"owner"`
+ Name string `json:"name"`
+ } `json:"repository"`
+ }
+ if err := json.Unmarshal(body, &payload); err != nil || payload.After == "" {
+ w.WriteHeader(http.StatusOK) // not a push event; ignore
+ return
+ }
+
+ owner, name, commit := payload.Repo.Owner.Login, payload.Repo.Name, payload.After
+
+ // Deduplicate on commit SHA
+ var count int
+ _ = s.db.QueryRow(`SELECT COUNT(*) FROM jobs WHERE commit_sha = ?`, commit).Scan(&count)
+ if count > 0 {
+ w.WriteHeader(http.StatusOK)
+ return
+ }
+
+ _, err = s.db.Exec(
+ `INSERT INTO jobs (job_id, repo_owner, repo_name, commit_sha, status_cache) VALUES (?,?,?,?,'pending')`,
+ randomID(), owner, name, commit,
+ )
+ if err != nil {
+ log.Printf("webhook: insert job: %v", err)
+ http.Error(w, "db error", http.StatusInternalServerError)
+ return
+ }
+ log.Printf("webhook: queued %s/%s@%.12s", owner, name, commit)
+ w.WriteHeader(http.StatusOK)
+}
+
+// ---------------------------------------------------------------------------
+// Poll handler
+// ---------------------------------------------------------------------------
+
+func (s *coordinationServer) handlePoll(w http.ResponseWriter, r *http.Request) {
+ if r.Method != http.MethodPost {
+ http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
+ return
+ }
+ var req struct {
+ RunnerID string `json:"runner_id"`
+ Secret string `json:"secret"`
+ }
+ if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
+ http.Error(w, "bad request", http.StatusBadRequest)
+ return
+ }
+ if req.Secret != s.cfg.runnerSecret {
+ http.Error(w, "forbidden", http.StatusForbidden)
+ return
+ }
+
+ // Auto-register / heartbeat
+ _, _ = s.db.Exec(
+ `INSERT INTO runners (runner_id, last_seen) VALUES (?,?)
+ ON CONFLICT(runner_id) DO UPDATE SET last_seen=excluded.last_seen`,
+ req.RunnerID, time.Now().UTC().Format(time.RFC3339),
+ )
+
+ // Clean expired jobs; finalise completed jobs for this runner
+ s.reapExpiredJobs()
+ s.syncCompletedJobsForRunner(req.RunnerID)
+
+ // Capacity check
+ var activeCount int
+ _ = s.db.QueryRow(`SELECT COUNT(*) FROM jobs WHERE runner_id = ?`, req.RunnerID).Scan(&activeCount)
+ if activeCount >= s.cfg.maxJobsPerRunner {
+ w.Header().Set("Retry-After", "5")
+ w.WriteHeader(http.StatusTooManyRequests)
+ return
+ }
+
+ // Pick one unassigned job (FIFO by rowid, round-robin across repos)
+ row := s.db.QueryRow(`SELECT job_id, repo_owner, repo_name, commit_sha FROM jobs WHERE runner_id IS NULL ORDER BY rowid LIMIT 1`)
+ var jobID, repoOwner, repoName, commitSHA string
+ if err := row.Scan(&jobID, &repoOwner, &repoName, &commitSHA); err == sql.ErrNoRows {
+ w.Header().Set("Content-Type", "application/json")
+ json.NewEncoder(w).Encode(map[string]any{"job": nil})
+ return
+ } else if err != nil {
+ http.Error(w, "db error", http.StatusInternalServerError)
+ return
+ }
+
+ // Create scoped one-time Gitea token
+ scopedToken, err := s.createScopedGiteaToken(repoOwner, repoName, jobID)
+ if err != nil {
+ log.Printf("poll: create token for job %s: %v", jobID, err)
+ http.Error(w, "token creation failed", http.StatusInternalServerError)
+ return
+ }
+
+ now := time.Now().UTC()
+ expiresAt := now.Add(s.cfg.jobTimeout)
+ _, _ = s.db.Exec(
+ `UPDATE jobs SET runner_id=?, assigned_at=?, expires_at=?, gitea_token=?, status_cache='running' WHERE job_id=?`,
+ req.RunnerID, now.Format(time.RFC3339), expiresAt.Format(time.RFC3339), scopedToken, jobID,
+ )
+
+ s.setGiteaCommitStatus(repoOwner, repoName, commitSHA, "pending", "jci: job assigned")
+ log.Printf("poll: assigned job %s to runner %s (%s/%s@%.12s)", jobID, req.RunnerID, repoOwner, repoName, commitSHA)
+
+ cloneURL := fmt.Sprintf("https://%s:%s@%s/%s/%s", s.cfg.giteaUser, scopedToken, s.cfg.giteaHost, repoOwner, repoName)
+ w.Header().Set("Content-Type", "application/json")
+ json.NewEncoder(w).Encode(map[string]any{
+ "job": map[string]string{
+ "job_id": jobID,
+ "clone_url": cloneURL,
+ "commit_sha": commitSHA,
+ "repo_owner": repoOwner,
+ "repo_name": repoName,
+ },
+ })
+}
+
+// ---------------------------------------------------------------------------
+// Gitea API helpers
+// ---------------------------------------------------------------------------
+
+// verifyGiteaAdminToken probes the Gitea token-list endpoint to confirm permissions.
+func verifyGiteaAdminToken(cfg serverConfig) error {
+ url := fmt.Sprintf("https://%s/api/v1/users/%s/tokens", cfg.giteaHost, cfg.giteaUser)
+ resp, err := giteaCall("GET", url, cfg.giteaToken, nil)
+ if err != nil {
+ return err
+ }
+ if resp.status != 200 {
+ return fmt.Errorf("GET %s returned %d — check GITEA_TOKEN permissions", url, resp.status)
+ }
+ return nil
+}
+
+func (s *coordinationServer) createScopedGiteaToken(owner, repo, jobID string) (string, error) {
+ url := fmt.Sprintf("https://%s/api/v1/users/%s/tokens", s.cfg.giteaHost, s.cfg.giteaUser)
+ expiry := time.Now().Add(s.cfg.jobTimeout).Format(time.RFC3339)
+ bodyData, _ := json.Marshal(map[string]any{
+ "name": "jci-job-" + jobID,
+ "scopes": []string{"read:repository", "write:repository"},
+ "expires_at": expiry,
+ })
+ resp, err := giteaCall("POST", url, s.cfg.giteaToken, bodyData)
+ if err != nil {
+ return "", err
+ }
+ if resp.status != 201 {
+ return "", fmt.Errorf("create token: status %d: %s", resp.status, resp.body)
+ }
+ var result struct {
+ SHA1 string `json:"sha1"`
+ }
+ if err := json.Unmarshal(resp.body, &result); err != nil || result.SHA1 == "" {
+ return "", fmt.Errorf("create token: unexpected response: %s", resp.body)
+ }
+ return result.SHA1, nil
+}
+
+func (s *coordinationServer) deleteScopedGiteaToken(jobID string) {
+ name := "jci-job-" + jobID
+ url := fmt.Sprintf("https://%s/api/v1/users/%s/tokens/%s", s.cfg.giteaHost, s.cfg.giteaUser, name)
+ if _, err := giteaCall("DELETE", url, s.cfg.giteaToken, nil); err != nil {
+ log.Printf("delete token %s: %v", name, err)
+ }
+}
+
+func (s *coordinationServer) setGiteaCommitStatus(owner, repo, commitSHA, state, description string) {
+ url := fmt.Sprintf("https://%s/api/v1/repos/%s/%s/statuses/%s", s.cfg.giteaHost, owner, repo, commitSHA)
+ body, _ := json.Marshal(map[string]string{
+ "state": state,
+ "description": description,
+ "context": "jci",
+ })
+ if _, err := giteaCall("POST", url, s.cfg.giteaToken, body); err != nil {
+ log.Printf("set commit status %s/%s@%.12s → %s: %v", owner, repo, commitSHA, state, err)
+ }
+}
+
+// checkJobStatusOnGitea reads status.txt from the latest refs/jci-runs/<commit>/* on Gitea.
+// Returns "ok", "err", "running", or "" (unknown/not found).
+func (s *coordinationServer) checkJobStatusOnGitea(owner, repo, commitSHA string) string {
+ // List matching refs
+ url := fmt.Sprintf("https://%s/api/v1/repos/%s/%s/git/refs/jci-runs/%s", s.cfg.giteaHost, owner, repo, commitSHA)
+ resp, err := giteaCall("GET", url, s.cfg.giteaToken, nil)
+ if err != nil || resp.status != 200 {
+ return ""
+ }
+ var refs []struct {
+ Object struct{ SHA string } `json:"object"`
+ }
+ if err := json.Unmarshal(resp.body, &refs); err != nil || len(refs) == 0 {
+ return ""
+ }
+ runCommitSHA := refs[len(refs)-1].Object.SHA
+
+ // Get tree
+ treeURL := fmt.Sprintf("https://%s/api/v1/repos/%s/%s/git/trees/%s", s.cfg.giteaHost, owner, repo, runCommitSHA)
+ treeresp, err := giteaCall("GET", treeURL, s.cfg.giteaToken, nil)
+ if err != nil || treeresp.status != 200 {
+ return ""
+ }
+ var tree struct {
+ Tree []struct {
+ Path string `json:"path"`
+ SHA string `json:"sha"`
+ Type string `json:"type"`
+ } `json:"tree"`
+ }
+ if err := json.Unmarshal(treeresp.body, &tree); err != nil {
+ return ""
+ }
+ var blobSHA string
+ for _, e := range tree.Tree {
+ if e.Path == "status.txt" && e.Type == "blob" {
+ blobSHA = e.SHA
+ break
+ }
+ }
+ if blobSHA == "" {
+ return ""
+ }
+
+ blobURL := fmt.Sprintf("https://%s/api/v1/repos/%s/%s/git/blobs/%s", s.cfg.giteaHost, owner, repo, blobSHA)
+ blobResp, err := giteaCall("GET", blobURL, s.cfg.giteaToken, nil)
+ if err != nil || blobResp.status != 200 {
+ return ""
+ }
+ var blob struct {
+ Content string `json:"content"`
+ Encoding string `json:"encoding"`
+ }
+ if err := json.Unmarshal(blobResp.body, &blob); err != nil {
+ return ""
+ }
+ content := blob.Content
+ if blob.Encoding == "base64" {
+ decoded, err := base64.StdEncoding.DecodeString(strings.ReplaceAll(content, "\n", ""))
+ if err != nil {
+ return ""
+ }
+ content = string(decoded)
+ }
+ content = strings.TrimSpace(content)
+ if content == "ok" || content == "err" || content == "running" {
+ return content
+ }
+ return ""
+}
+
+// ---------------------------------------------------------------------------
+// Background maintenance
+// ---------------------------------------------------------------------------
+
+func (s *coordinationServer) reapExpiredJobs() {
+ rows, err := s.db.Query(
+ `SELECT job_id, repo_owner, repo_name, commit_sha FROM jobs WHERE expires_at IS NOT NULL AND expires_at < ?`,
+ time.Now().UTC().Format(time.RFC3339),
+ )
+ if err != nil {
+ return
+ }
+ type expiredJob struct{ jobID, owner, repo, commit string }
+ var expired []expiredJob
+ for rows.Next() {
+ var j expiredJob
+ if rows.Scan(&j.jobID, &j.owner, &j.repo, &j.commit) == nil {
+ expired = append(expired, j)
+ }
+ }
+ rows.Close()
+
+ for _, j := range expired {
+ log.Printf("job %s timed out", j.jobID)
+ s.setGiteaCommitStatus(j.owner, j.repo, j.commit, "failure", "jci: job timed out")
+ s.deleteScopedGiteaToken(j.jobID)
+ _, _ = s.db.Exec(`DELETE FROM jobs WHERE job_id = ?`, j.jobID)
+ }
+}
+
+func (s *coordinationServer) syncCompletedJobsForRunner(runnerID string) {
+ rows, err := s.db.Query(
+ `SELECT job_id, repo_owner, repo_name, commit_sha, COALESCE(status_cache,''), COALESCE(cache_until,'')
+ FROM jobs WHERE runner_id = ?`, runnerID,
+ )
+ if err != nil {
+ return
+ }
+ type jobRow struct {
+ jobID, owner, repo, commit, statusCache, cacheUntilStr string
+ }
+ var jobs []jobRow
+ for rows.Next() {
+ var j jobRow
+ if rows.Scan(&j.jobID, &j.owner, &j.repo, &j.commit, &j.statusCache, &j.cacheUntilStr) == nil {
+ jobs = append(jobs, j)
+ }
+ }
+ rows.Close()
+
+ for _, j := range jobs {
+ status := j.statusCache
+ cacheUntil, _ := time.Parse(time.RFC3339, j.cacheUntilStr)
+ if time.Now().UTC().After(cacheUntil) {
+ if fresh := s.checkJobStatusOnGitea(j.owner, j.repo, j.commit); fresh != "" {
+ status = fresh
+ }
+ newCacheUntil := time.Now().UTC().Add(15 * time.Second).Format(time.RFC3339)
+ _, _ = s.db.Exec(`UPDATE jobs SET status_cache=?, cache_until=? WHERE job_id=?`, status, newCacheUntil, j.jobID)
+ }
+
+ if status == "ok" || status == "err" {
+ giteaState := "success"
+ if status == "err" {
+ giteaState = "failure"
+ }
+ s.setGiteaCommitStatus(j.owner, j.repo, j.commit, giteaState, "jci: job completed")
+ s.deleteScopedGiteaToken(j.jobID)
+ _, _ = s.db.Exec(`DELETE FROM jobs WHERE job_id = ?`, j.jobID)
+ log.Printf("job %s completed: %s", j.jobID, status)
+ }
+ }
+}
+
+// ---------------------------------------------------------------------------
+// Generic Gitea HTTP helper
+// ---------------------------------------------------------------------------
+
+type giteaResponse struct {
+ status int
+ body []byte
+}
+
+func giteaCall(method, url, token string, body []byte) (giteaResponse, error) {
+ var bodyReader io.Reader
+ if body != nil {
+ bodyReader = bytes.NewReader(body)
+ }
+ req, err := http.NewRequest(method, url, bodyReader)
+ if err != nil {
+ return giteaResponse{}, err
+ }
+ req.Header.Set("Authorization", "token "+token)
+ req.Header.Set("Content-Type", "application/json")
+ req.Header.Set("Accept", "application/json")
+
+ client := &http.Client{Timeout: 15 * time.Second}
+ resp, err := client.Do(req)
+ if err != nil {
+ return giteaResponse{}, err
+ }
+ defer resp.Body.Close()
+ data, _ := io.ReadAll(resp.Body)
+ return giteaResponse{status: resp.StatusCode, body: data}, nil
+}
+
+// ---------------------------------------------------------------------------
+// Tiny utilities
+// ---------------------------------------------------------------------------
+
+func checkHMAC(body []byte, sig, secret string) bool {
+ mac := hmac.New(sha256.New, []byte(secret))
+ mac.Write(body)
+ expected := hex.EncodeToString(mac.Sum(nil))
+ return hmac.Equal([]byte(sig), []byte(expected))
+}
+
+func randomID() string {
+ b := make([]byte, 16)
+ rand.Read(b)
+ return fmt.Sprintf("%x-%x-%x-%x-%x", b[0:4], b[4:6], b[6:8], b[8:10], b[10:])
+}
+
+// parentDir returns the directory component of a file path without importing filepath
+// (avoids a collision with the filepath import in other files in the package).
+func parentDir(path string) string {
+ for i := len(path) - 1; i >= 0; i-- {
+ if path[i] == '/' || path[i] == '\\' {
+ return path[:i]
+ }
+ }
+ return "."
+}
diff --git a/internal/jci/web.go b/internal/jci/web.go
@@ -4,7 +4,6 @@ import (
"encoding/json"
"fmt"
"net/http"
- "os"
"os/exec"
"path/filepath"
"strings"
@@ -23,8 +22,7 @@ type CommitInfo struct {
Message string `json:"message"`
HasCI bool `json:"hasCI"`
CIStatus string `json:"ciStatus"` // "success", "failed", or ""
- CIPushed bool `json:"ciPushed"` // whether CI ref is pushed to remote
- Runs []RunInfo `json:"runs"` // multiple runs (for cron)
+ Runs []RunInfo `json:"runs"`
}
// RunInfo holds info about a single CI run
@@ -117,44 +115,20 @@ func getLocalBranches() ([]string, error) {
return strings.Split(out, "\n"), nil
}
-// getRemoteJCIRefs returns a set of commits that have CI refs pushed to remote
-
-
-
-// getCIStatus returns "success", "failed", or "running" based on status.txt
-func getCIStatus(commit string) string {
- return getCIStatusFromRef("refs/jci/" + commit)
-}
-
-// getCIStatusFromRef returns status from any ref
+// getCIStatusFromRef returns status from a refs/jci-runs/ ref
func getCIStatusFromRef(ref string) string {
- // Try to read status.txt (new format)
cmd := exec.Command("git", "show", ref+":status.txt")
out, err := cmd.Output()
- if err == nil {
- status := strings.TrimSpace(string(out))
- switch status {
- case "ok":
- return "success"
- case "err":
- return "failed"
- case "running":
- return "running"
- }
- }
-
- // Fallback: parse index.html for old results
- cmd = exec.Command("git", "show", ref+":index.html")
- out, err = cmd.Output()
if err != nil {
return ""
}
- content := string(out)
- if strings.Contains(content, "PASSED") || strings.Contains(content, "SUCCESS") {
+ switch strings.TrimSpace(string(out)) {
+ case "ok":
return "success"
- }
- if strings.Contains(content, "FAILED") {
+ case "err":
return "failed"
+ case "running":
+ return "running"
}
return ""
}
@@ -186,18 +160,14 @@ func serveCommitAPI(w http.ResponseWriter, commit string) {
ref = "refs/jci-runs/" + actualCommit + "/" + currentRunID
} else {
actualCommit = commit
- ref = "refs/jci/" + commit
- // Check if single-run ref exists, otherwise try to find latest run
- if !RefExists(ref) {
- // Look for runs - get the latest one
- runRefs, _ := ListJCIRunRefs()
- for _, r := range runRefs {
- if strings.HasPrefix(r, "refs/jci-runs/"+commit+"/") {
- ref = r // Use the last one (they're sorted)
- parts := strings.Split(r, "/")
- if len(parts) >= 4 {
- currentRunID = parts[3]
- }
+ // Find the latest run for this commit
+ runRefs, _ := ListJCIRunRefs()
+ for _, r := range runRefs {
+ if strings.HasPrefix(r, "refs/jci-runs/"+commit+"/") {
+ ref = r // last one wins (sorted by timestamp)
+ parts := strings.Split(r, "/")
+ if len(parts) >= 4 {
+ currentRunID = parts[3]
}
}
}
@@ -345,15 +315,7 @@ func getBranchCommitsPaginated(branch string, offset, limit int) ([]CommitInfo,
return nil, nil
}
- // Get local JCI refs (single-run)
- jciRefs, _ := ListJCIRefs()
- jciSet := make(map[string]bool)
- for _, ref := range jciRefs {
- commit := strings.TrimPrefix(ref, "jci/")
- jciSet[commit] = true
- }
-
- // Get local JCI run refs (multi-run): commit -> list of run refs
+ // Get local JCI run refs: commit -> list of run refs
jciRuns := make(map[string][]string)
runRefs, _ := ListJCIRunRefs()
for _, ref := range runRefs {
@@ -364,9 +326,6 @@ func getBranchCommitsPaginated(branch string, offset, limit int) ([]CommitInfo,
}
}
- // Get remote JCI refs for CI push status
- remoteCI := getRemoteJCIRefs("origin")
-
var commits []CommitInfo
for _, line := range strings.Split(out, "\n") {
parts := strings.SplitN(line, "|", 2)
@@ -376,17 +335,12 @@ func getBranchCommitsPaginated(branch string, offset, limit int) ([]CommitInfo,
hash := parts[0]
msg := parts[1]
- hasCI := jciSet[hash] || len(jciRuns[hash]) > 0
+ hasCI := len(jciRuns[hash]) > 0
commit := CommitInfo{
Hash: hash,
ShortHash: hash[:7],
Message: msg,
HasCI: hasCI,
- CIPushed: remoteCI["refs/jci/"+hash],
- }
-
- if jciSet[hash] {
- commit.CIStatus = getCIStatus(hash)
}
for _, runRef := range jciRuns[hash] {
@@ -1027,18 +981,14 @@ func showMainPage(w http.ResponseWriter, r *http.Request) {
func serveFromRef(w http.ResponseWriter, commit string, runID string, filePath string) {
var ref string
- // Build ref based on whether we have a runID
if runID != "" {
ref = "refs/jci-runs/" + commit + "/" + runID
} else {
- ref = "refs/jci/" + commit
- // If single-run ref doesn't exist, try to find latest run
- if !RefExists(ref) {
- runRefs, _ := ListJCIRunRefs()
- for _, r := range runRefs {
- if strings.HasPrefix(r, "refs/jci-runs/"+commit+"/") {
- ref = r // Use last matching (sorted by timestamp)
- }
+ // No runID: find the latest run for this commit
+ runRefs, _ := ListJCIRunRefs()
+ for _, r := range runRefs {
+ if strings.HasPrefix(r, "refs/jci-runs/"+commit+"/") {
+ ref = r // last one wins (sorted by timestamp)
}
}
}
@@ -1077,21 +1027,3 @@ func serveFromRef(w http.ResponseWriter, commit string, runID string, filePath s
w.Write(out)
}
-
-// extractRef extracts files from a ref to a temp directory (not used currently but useful)
-func extractRef(ref string) (string, error) {
- tmpDir, err := os.MkdirTemp("", "jci-view-*")
- if err != nil {
- return "", err
- }
-
- cmd := exec.Command("git", "archive", ref)
- tar := exec.Command("tar", "-xf", "-", "-C", tmpDir)
-
- tar.Stdin, _ = cmd.StdoutPipe()
- tar.Start()
- cmd.Run()
- tar.Wait()
-
- return tmpDir, nil
-}
diff --git a/www_jci/public/index.html b/www_jci/public/index.html
@@ -15,7 +15,7 @@
<div>
<a href="/binaries.html">Binaries</a>
<a href="/releases/file/README.md.html">Docs</a>
- <span id="jci-version">v1.0.2</span>
+ <span id="jci-version">v1.0.3</span>
</div>
</nav>
<h1><img src="assets/logo.png" alt="JCI logo" style="height:1em;width:auto;vertical-align:middle;margin-right:0.4em;">Minimal. Offline. Local.</h1>
@@ -78,6 +78,8 @@ Your support allows for continued maintenance and new features.
<li><code>git jci web</code> runs a web server, browse the CI results locally.</li>
<li><code>git jci push/pull</code> syncs refs with your remotes.</li>
<li><code>git jci cron sync</code> syncs your cron spec with system cron so that you can run midnight builds for example.</li>
+ <li><code>git jci server</code> runs the coordination server that receives Gitea webhooks and hands jobs to runners.</li>
+ <li><code>git jci runner</code> polls the server and executes jobs inside Docker containers for distributed CI.</li>
</ul>
</section>
