Jaypore CI

ARCH.md (13771B)

---

 # JCI Architecture
 
 JCI is a local-first CI system. CI results are stored directly inside the git repository
 as regular git objects under custom refs, so they travel with the repo on push/pull.
 `git jci server` and `git jci runner` extend this with a webhook-driven, distributed
 CI execution layer — the server is a thin coordination point; all actual CI work happens
 inside runner-managed Docker containers.
 
 ---
 
 ## Repository layout
 
 ```
 cmd/git-jci/main.go   entry point, CLI dispatch
 internal/jci/
   run.go              execute CI for a commit
   git.go              low-level git plumbing helpers
   web.go              HTTP server + SPA (single-file, inline)
   push.go             push CI refs to a remote
   pull.go             fetch CI refs from a remote
   prune.go            delete old CI refs locally or on a remote
   cron.go             cron subcommands: ls, sync
   server.go           coordination server (webhook + runner poll)
   runner.go           runner (Docker job dispatch)
 www_jci/              project website (separate, not embedded in the binary)
 ```
 
 ---
 
 ## Ref layout inside git
 
 ```
 refs/jci-runs/<commit>/<run-id>   every run result, including one-off manual runs
 ```
 
 Every run — whether triggered manually, by cron, or by the distributed runner — gets a
 unique run ID (`<unix_timestamp>-<4_random_hex_chars>`), so multiple runs on the same
 commit are stored independently and none overwrite another.
 
 Each ref points to a git **commit** whose **tree** holds the CI artefacts:
 
 ```
 status.txt        "ok" | "err" | "running"
 run.output.txt    combined stdout+stderr from .jci/run.sh
 index.html        standalone HTML view of the run (generated only if the job
                   did not produce its own index.html)
 <anything else>   files written by the CI script to $JCI_OUTPUT_DIR
 ```
 
 Because these are normal git objects, `git push origin 'refs/jci-runs/*:refs/jci-runs/*'`
 is all it takes to share results with a team.
 
 ---
 
 ## CI execution flow (`git jci run`)
 
 ```
 1. GetCurrentCommit()          resolve HEAD → commit hash
 2. generateRunID()             timestamp + 2 random bytes → unique run ID
 3. check .jci/run.sh exists
 4. mkdir .jci/<commit>/        temporary output directory
 5. write status.txt = "running"
 6. exec bash .jci/run.sh       working dir = output dir
                                 env: JCI_COMMIT, JCI_REPO_ROOT, JCI_OUTPUT_DIR
                                 stdout+stderr captured to run.output.txt
 7. write status.txt = "ok"|"err"
 8. generateIndexHTML()         only if index.html was NOT written by the job
 9. StoreTree()                 git hash-object each file → git mktree → git commit-tree
                                 → git update-ref refs/jci-runs/<commit>/<runID>
 10. rm -rf .jci/<commit>/      clean up temp dir
 ```
 
 The CI script receives three environment variables and should write any extra
 artefacts into `$JCI_OUTPUT_DIR`. Whatever exists in that directory when the
 script exits is committed to git.
 
 ---
 
 ## Web UI (`git jci web [port]`)
 
 A minimal three-panel SPA served entirely from a single Go handler.
 No external assets; the HTML/CSS/JS is embedded inline in `showMainPage()`.
 
 ```
 GET /api/branches              list local branch names
 GET /api/commits?branch=&page= paginated commit list with CI status per commit
 GET /api/commit/<hash>         commit detail + file list (latest run for that commit)
 GET /api/commit/<hash>/<runId> same but for a specific run
 
 GET /jci/<commit>/<runId>/<file>/raw    serve file from refs/jci-runs/<commit>/<runId>
 GET /jci/<commit>/<file>/raw            serve file from the latest run for <commit>
 GET /jci/...  (other)                   serve SPA shell (JS handles routing)
 GET /                                   serve SPA shell
 ```
 
 The UI keeps a client-side page index for infinite-scroll commit loading
 (`commitsPageSize = 100` per page).
 
 ---
 
 ## Cron integration (`git jci cron`)
 
 **`cron ls`** — shows what is configured in `.jci/crontab` and what is currently
 installed in the user's system crontab.
 
 **`cron sync`** — idempotent sync from `.jci/crontab` → system crontab:
 
 ```
 1. parse .jci/crontab          5-field schedule + optional branch:X name:Y
 2. crontab -l                  read current system crontab
 3. strip lines containing      # JCI:<sha256(repoRoot)>
 4. append new lines, one per entry:
      <schedule>  cd <repoRoot> && [git checkout <branch> &&] git-jci run  # JCI:<id> [<name>]
 5. crontab -                   install new crontab
 ```
 
 Each repo is identified by `sha256(absolute_path)` so entries from different
 repos never collide.
 
 ---
 
 ## Push / Pull
 
 ```
 git jci push [remote]   discover all local refs/jci-runs/* not on remote → git push each
 git jci pull [remote]   git fetch refs/jci-runs/*:refs/jci-runs/*
 ```
 
 ---
 
 ## Prune
 
 Removes CI refs to reclaim space. Works on local repo or a remote.
 
 ```
 git jci prune [--older-than=<duration>] [--commit]
 git jci prune --on-remote=<remote> [--older-than=<duration>] [--commit]
 ```
 
 Duration format: `30d`, `2w`, `6m`, `4h`, or any Go duration string.
 Without `--commit` the command is a dry run.
 After local deletion it runs `git gc --prune=now` to actually free objects.
 
 ---
 
 ## Distributed CI (`git jci server` / `git jci runner`)
 
 Both commands are added to the existing `git-jci` binary and run in the foreground,
 managed by Docker or systemd.
 
 ```
 Gitea ──webhook──▶ jci server ◀──poll── jci runner (Docker container)
                        │                      │
                        │  Gitea API           │  docker socket
                        ▼                      ▼
                     Gitea                 job container
                   (set status)          (git-jci run → git-jci push → Gitea)
 ```
 
 ### Server (`git jci server`)
 
 #### Configuration (env vars)
 
 ```
 GITEA_HOST=gitea.example.com
 GITEA_USER=gitea_service_user
 GITEA_TOKEN=<token>              # must have permission to create/delete scoped tokens
 GITEA_WEBHOOK_SECRET=<hmac secret>
 RUNNER_SECRET=<shared secret for runners>
 JCI_MAX_JOBS=4                   # max concurrent jobs assignable to a single runner
 JCI_JOB_TIMEOUT=60m              # server-wide job timeout (default: 60 minutes)
 ```
 
 #### SQLite schema
 
 ```sql
 -- Known runners; auto-created on first poll.
 CREATE TABLE runners (
     runner_id   TEXT PRIMARY KEY,
     last_seen   DATETIME NOT NULL
 );
 
 -- Active (pending or running) jobs only.
 -- Completed/timed-out jobs are deleted immediately.
 CREATE TABLE jobs (
     job_id        TEXT PRIMARY KEY,   -- UUID
     repo_owner    TEXT NOT NULL,
     repo_name     TEXT NOT NULL,
     commit_sha    TEXT NOT NULL,      -- idempotency key
     runner_id     TEXT,               -- NULL = unassigned
     assigned_at   DATETIME,
     expires_at    DATETIME,           -- assigned_at + JCI_JOB_TIMEOUT
     gitea_token   TEXT NOT NULL,      -- one-time scoped token for this job
     status_cache  TEXT,               -- last known status ("pending"|"running"|"success"|"failure")
     cache_until   DATETIME            -- when status_cache expires (15s TTL)
 );
 ```
 
 #### Startup check
 
 On startup the server verifies that `GITEA_TOKEN` has permission to create and delete
 per-repo tokens via the Gitea API. If the check fails, the server exits with a clear
 error. No silent degradation.
 
 #### Webhook handling (`POST /webhook`)
 
 1. Verify HMAC-SHA256 signature using `GITEA_WEBHOOK_SECRET`. Return 400 on failure.
 2. Extract `repo.owner`, `repo.name`, `commit_sha` (from `after` field on push events).
 3. Check `jobs` table for an existing active job with the same `commit_sha`. If found,
    respond 200 and drop — no duplicate jobs.
 4. Insert a new job row with `runner_id = NULL`, `status_cache = "pending"`.
 5. Respond 200 immediately.
 
 #### Runner poll endpoint (`POST /poll`)
 
 Request body: `{ "runner_id": "...", "secret": "..." }`
 
 1. Verify `secret == RUNNER_SECRET`. Return 403 on failure.
 2. Upsert runner row (`runner_id`, `last_seen = now()`). This is auto-registration.
 3. Count active jobs already assigned to this runner.
    - If count >= `JCI_MAX_JOBS`: respond 429 with `Retry-After: 5`.
 4. Poll Gitea for cached status of each assigned job (see *Status polling* below).
    Completed jobs are deleted from the DB.
 5. Pick one unassigned job from the queue (FIFO per-repo, round-robin across repos).
    If none: respond 200 with `{ "job": null }`.
 6. For the selected job:
    a. Create a fresh scoped Gitea token (scoped to that repo, expiry = `now + JCI_JOB_TIMEOUT`) IF POSSIBLE. Some versions of gitea don't allow this at all.
    b. Set `runner_id`, `assigned_at`, `expires_at`, store token in `jobs.gitea_token`.
    c. Set Gitea commit status to `"pending"`.
 7. Respond 200 with the job payload:
 
 ```json
 {
   "job": {
     "job_id":     "...",
     "clone_url":  "https://<user>:<token>@gitea.example.com/owner/repo",
     "commit_sha": "...",
     "repo_owner": "...",
     "repo_name":  "..."
   }
 }
 ```
 
 #### Status polling (server → Gitea)
 
 The server checks job status by reading `refs/jci-runs/<commit>/<runID>` on Gitea and
 inspecting `status.txt` in that ref's tree.
 
 - Results are cached per-job for **15 seconds** (`jobs.cache_until`).
 - A check is triggered every time the assigned runner calls `/poll`.
 - A final check is performed at `jobs.expires_at` (60-minute timeout); if still
   unresolved, the job is deleted and Gitea status is set to `"failure"`.
 - On `status.txt = "ok"` or `"err"` the server:
   1. Sets Gitea commit status to `"success"` or `"failure"`.
   2. Deletes the one-time token via the Gitea API.
   3. Deletes the job row from SQLite.
 
 #### Token cleanup
 
 Two-layer cleanup for the one-time Gitea token:
 1. **Gitea-native expiry**: token created with hard expiry at `assigned_at + 60m`.
 2. **Server-side deletion**: explicit delete via Gitea API on job completion or timeout.
 
 This ensures the token cannot be used beyond 60 minutes even if the server crashes.
 
 ---
 
 ### Runner (`git jci runner`)
 
 #### Configuration (env vars)
 
 ```
 JCI_SERVER=https://jci.example.com
 JCI_RUNNER_SECRET=<shared secret>
 ```
 
 #### Persistent state
 
 A SQLite database at a fixed path on a mounted volume:
 
 ```sql
 CREATE TABLE identity (
     runner_id TEXT PRIMARY KEY   -- generated once, reused across restarts
 );
 ```
 
 #### Poll loop
 
 ```
 every 5s + random jitter (0–2s):
     POST /poll  →  { runner_id, secret }
     if 429: wait Retry-After seconds, then continue
     if job == null: continue
     if job != null: dispatch(job)   # non-blocking; runner returns to poll immediately
 ```
 
 #### Job dispatch
 
 For each received job the runner:
 
 1. **Pulls the `git-jci` binary** from a known location (e.g. mounted at
    `/usr/local/bin/git-jci` in the runner container) to inject into the job container.
 2. **Starts a detached job container**:
    ```
    docker run --rm -d \
      -v /usr/local/bin/git-jci:/usr/local/bin/git-jci:ro \
      -e JCI_COMMIT=<commit_sha> \
      --label jci-job=y \
      --label jci-job-timeout=60m \
      <image from job config> \
      /bin/sh -c "
        git clone --depth=1 --branch <commit> <clone_url> /repo &&
        cd /repo &&
        git-jci run
        git-jci push   # always runs, even if run failed, to commit status.txt
      "
    ```
    `clone_url` embeds the one-time credentials, so `git push` (via `git-jci push`)
    is transparent — `origin` is already set correctly by the clone.
 3. **Tracks the container ID** in memory (not persisted — runner crash means the
    container runs to completion or is reaped by Docker's own restart policy).
 
 The job container never talks to the `jci server` directly.
 
 #### Container reaping
 
 The runner periodically checks for containers labelled `jci-job=y` that have been
 running longer than the configured timeout and kills them via `docker rm -f`. This
 prevents capacity leaks if a job container hangs.
 
 ---
 
 ## End-to-end distributed flow
 
 ```
 1.  Dev pushes to Gitea
 2.  Gitea sends webhook → POST /webhook on jci server
 3.  Server verifies HMAC, deduplicates on commit SHA, inserts job (status=pending)
 4.  Runner polls → POST /poll
 5.  Server creates scoped one-time Gitea token, assigns job, sets Gitea status = "pending"
 6.  Server responds with job payload (clone URL with embedded token)
 7.  Runner starts job container (detached), returns to poll loop
 8.  Job container: clone → git-jci run → git-jci push (pushes refs/jci-runs/* to Gitea)
 9.  Runner polls again (5s + jitter)
 10. Server checks Gitea for status.txt in refs/jci-runs/<commit>/<runID> (15s cache)
 11. Server sees "ok"/"err" → sets Gitea commit status → deletes token → deletes job row
 12. Runner poll returns 429 if JCI_MAX_JOBS reached; backs off via Retry-After
 ```
 
 ---
 
 ## Key design constraints
 
 - **No external dependencies** — pure Go stdlib + git CLI + SQLite (driver only for
   server/runner). No separate storage service.
 - **Results live in the repo** — CI artefacts are normal git objects under
   `refs/jci-runs/*`; they travel with the repo on push/pull.
 - **Runner is pull-only** — server never initiates contact with a runner; no runner
   address is stored.
 - **All Gitea API interaction is server-only** — runner uses only plain `git` commands
   (clone/push); the coordination layer is opaque to it.
 - **One-time credentials per job** — scoped to one repo, expire in 60 minutes via both
   Gitea-native expiry and explicit server-side deletion.
 - **Stateless job containers** — no volumes; crash of a container loses only that run.
 - **Artefacts are arbitrary files** — anything written to `$JCI_OUTPUT_DIR` is stored.
 - **Single binary** — `git-jci` placed on `$PATH` becomes available as `git jci`.
 - **Server runs in foreground** — managed by Docker or systemd; no self-daemonization.