runner-test/README.md

# gitea-actions-compat-test

A self-asserting compatibility suite for Gitea Actions. Push this repo to a Gitea
instance with a registered runner, and `00-suite-runner.yml` runs ten independent
test workflows that prove Gitea's implementation of GitHub-style actions behaves
like GitHub's. The suite uses two strategies depending on what it tests:

- **Feature tests** (C1, C5, C6, C9): assert the feature works. A failure means
  the feature is broken.
- **Divergence probes** (C2-v4, C8, C10-v4): assert the *current state* of a
  known Gitea divergence. Both outcomes (divergence present or fixed) are PASS;
  the assertion message documents which state was found. A failure means
  something *unexpected* happened.
- **Infrastructure-dependent tests** (C3, C4, C7): fully verify when the
  prerequisite is met. When it isn't (no Docker, no cache server, TLS blocked),
  assertions are skipped with a `::warning::` documenting what was NOT verified.

A green run means "no unexpected failures." Read the per-test table and the
assertion messages in the logs to understand exactly what was verified vs.
what was skipped on your instance.

This README is the human-facing guide. It tells you what to set up before you
push, what each test verifies, where the docs disagree with the source code, and
which behaviors cannot be checked from inside a job (those live in the manual
eyeball checklist at the bottom).

---

## HARD prerequisites

Only the Gitea version check is a hard gate (the probe job fails if version <
1.22.0). All other prerequisites below are **informational**: the probe prints
best-effort notes about them but does NOT fail if they are missing.
Components that depend on them will conditionally skip with `::warning::`.

### Runner

- **A runner registered with the label used by `runs-on`.** The default label is
  `ubuntu-24.04`. If your instance uses a different label, set
  `RUNS_ON_LABEL` (or edit `runs-on:` in every workflow) to match, or the jobs
  will sit in `waiting` forever.

### Gitea version

- **Gitea >= 1.22.0.** This is the **minimum to PARSE/accept** these workflows.
  It is NOT a guarantee that every component passes. Several checks (notably the
  `upload-artifact@v4` regression and reusable-workflow secret passthrough) may
  still fail on 1.22.x and only turn green on later point releases. See the
  per-test table below for which components need a newer version.

  To be explicit about the two thresholds:

  | Threshold | Meaning |
  | --- | --- |
  | **1.22.0** | Minimum to PARSE/accept the workflow YAML. Below this, the runner rejects the suite outright. |
  | **higher, version-dependent** | Minimum to PASS all components. Each component that needs more lists its version note in the per-test table. |

### Storage and caches

- **Storage backend enabled.** The artifact and cache actions need object storage
  (local, MinIO, S3-compatible). Without it, uploads and downloads error out.
- **Cache server enabled.** `actions/cache@v3` relies on the Gitea cache server.
  Confirm it is reachable before expecting C3 to pass.
- **LFS enabled.** `test.bin` is an LFS object. The checkout LFS test (C1) and
  the fixtures depend on `git lfs` working on both the server and the runner.

### Action mirroring

- **Action mirroring allowed.** `ACTIONS_URL` and `DEFAULT_ACTIONS_URL` must NOT
  be set to a non-mirror strategy. The workflows reference actions by the
  `owner/repo@ref` shorthand (for example `actions/checkout@v4`), which Gitea
  resolves by mirroring the upstream action on first use. If mirroring is off or
  pointed at a custom non-mirror source, those `uses:` refs 404 and the run fails
  at fetch time. The mirror fetch is itself a behavior under test.

### Secrets

- **A repo secret `TEST_SECRET=placeholder`.** Define it in the repo settings.
  The reusable workflows that read it (`05-contexts` and `08-composite-reusable`)
  receive it through explicit forwarding (`secrets: inherit`) in the caller jobs,
  because Gitea's act_runner does not auto-forward caller secrets into reusable
  workflows by default (see act_runner #125 in the correction table). The
  placeholder value `placeholder` is what the assertion checks for, so do not put
  anything else there unless you also update the assertion.

---

## Warm-up: run `workflow_dispatch` once

After first setup, trigger the suite once via the **Run workflow** button
(`workflow_dispatch`) before drawing any conclusions from a `push` run. Action
mirroring is lazy. The first fetch of each `actions/*` ref can take a moment to
populate, and a cold run may show spurious fetch failures that vanish on the
second attempt. One warm-up dispatch lets the mirror settle.

---

## Per-test expectations

The suite is split into ten reusable workflows (`on: workflow_call`), each
invoked by a caller job (`c1` through `c10`) in `00-suite-runner.yml`. Each
component reports its own pass/fail, and `aggregate` fails the whole run if any
component is not `success` (a `skipped` or `cancelled` component counts as
failure).

| # | Workflow | What it verifies | Expected result | Min-version notes |
| --- | --- | --- | --- | --- |
| C1 | `01-checkout` | `actions/checkout@v4` variants: default checkout, `fetch-depth`, `lfs`, `ref` to a tag, `path`, sparse-checkout cone, and `persist-credentials` (both `true` and `false`, with a non-vacuous assertion that the embedded credential token differs between the two). | Pass on Gitea >= 1.22 with LFS enabled. The LFS sub-case is conditional on `git lfs` being present. | 1.22.0 to parse. LFS sub-case needs LFS enabled on both server and runner. |
| C2 | `02-artifacts` | Artifacts v3 and v4 round-trip, cross-job checksum comparison (producer writes, consumer verifies sha256). | v3 fully verified. v4 upload uses `continue-on-error: true`; if it fails (GHESNotSupportedError), the test asserts `regression-confirmed` (Gitea #31256). If it succeeds, full v4 checksum verification runs. | v4 needs a point release after the #31256 fix. v3 works on 1.22. |
| C3 | `03-cache` | Cache save/restore: a `seed` job writes a file and caches it (first run is a miss that saves), a `restore` job asserts an exact-key hit, then a miss-with-`restore-keys` fallback. | Fully verified when cache server is reachable. When cache server is down, all cache assertions are skipped with `::warning::`. | Cache server must be configured for meaningful verification. |
| C4 | `04-setup-runtime` | `setup-node`, `setup-python`, `setup-go`, `setup-java` (Temurin, Maven) in a matrix, each asserting the installed version. | Fully verified when setup actions can download runtimes. If TLS/network blocks downloads (self-signed cert), assertions are skipped with `::warning::`. | 1.22.0. Runtime downloads require internet access with valid TLS. |
| C5 | `05-contexts` | `github.sha` / `github.ref` / `github.run_id`, a job `env:` var round-tripping into a step, the auto-injected `GITHUB_TOKEN` (via `env:` mapping), the forwarded `TEST_SECRET` (skipped if empty), `needs` outputs consumed by a dependent job, and `if:` conditionals with `success()` / `always()`. `failure()` tested via step-level `continue-on-error` + `outcome`/`conclusion` assertion. | Pass. `TEST_SECRET` assertion is skipped with a message if the repo secret is unset. | 1.22.0 to parse. Secret forwarding depends on act_runner #125 (see correction table). |
| C6 | `06-workflow-commands` | `GITHUB_OUTPUT` (multi-line value, consumed via step output), `GITHUB_PATH` (added dir becomes callable), `GITHUB_ENV` (set in one step, read in the next), and `::warning::` / `::error::` / `::notice::` annotations plus `$GITHUB_STEP_SUMMARY`. | Pass. The annotation assertion is **emit-only**: it checks the command was accepted and the process did not crash. Annotation RENDERING in the UI is a manual checklist item. | 1.22.0. Rendering is UI-dependent (manual). |
| C7 | `07-services` | `postgres:16` and `redis:7` services with healthchecks, plus a `container: node:20` job. | Fully verified when Docker is available. On host-mode runners (no Docker), service/container assertions are skipped with `::warning::`. | 1.22.0. Docker on the runner is required for meaningful verification. |
| C8 | `08-composite-reusable` | A local composite action (`lib/composite-greet`) with input/output round-trip, and a reusable workflow that declares `inputs` and `secrets` on its `workflow_call` trigger. | Composite action fully verified. `inputs.who` and `TEST_SECRET` assertions document the *current state* of act_runner #125: PASS with "divergence-confirmed" if empty, PASS with value check if populated. | Version-dependent. See the note below. |
| C9 | `09-control-flow` | `timeout-minutes` at step level (step killed, outcome asserted as `failure` via `continue-on-error`), `continue-on-error` outcome/conclusion split, and `concurrency` as parse-and-accept only. | Pass. If step-level `timeout-minutes` is not enforced by the runner, a `::warning::` is emitted. Concurrency cancellation is manual-only. | 1.22.0. Step-level timeout-minutes may not be supported on all runner versions. |
| C10 | `10-gitea-divergences` | `environment:` non-blocking proof (fully verified), `upload-artifact@v4` regression re-confirm (asserts `regression-confirmed` if v4 fails, `v4-ok` if it succeeds), and problem matchers emitted (emit-only). | `environment:` non-blocking fully verified. v4 reconfirm asserts current state of #31256. Matcher rendering is manual-only. | 1.22.0 to parse. v4 needs the fix from #31256. |

---

## Correction table: where docs and source disagree

GitHub's public docs describe the canonical runtime. Gitea's `act` implementation
sometimes diverges. The table below records the known divergences with source
citations and the covering test. Keep this bounded: at most 10 rows.

| topic | doc-claim | source-truth | citation | covering-test |
| --- | --- | --- | --- | --- |
| `timeout-minutes` | Docs say it cancels a runaway job. | Source enforces it. `job_executor.go` around L370-380 applies the timeout and kills the job. | act source (`job_executor.go` L370-380) | C9 |
| `success` / `failure` / `cancelled` / `always` | Docs describe four status-check functions. | All four are implemented. `interpreter.go` around L621-640 defines them. | act source (`interpreter.go` L621-640) | C5 |
| `environment:` | Docs say it gates deployments behind protection rules. | Genuinely unsupported. There is no `environment` field on the Job struct in act, so the key is parsed but treated as a no-op. | act source (Job struct) | C10 |
| `upload-artifact@v4` | Docs frame v4 as a drop-in replacement for v3. | Not on Gitea 1.22. v4 triggers a GHES-style regression error. Reported fixed but may recur on certain configurations (see Gitea issue #36024 for a Nov 2025 recurrence on Docker-based runners). | Gitea issue #31256 | C2, C10 |
| Reusable-workflow secret inheritance | Docs say secrets auto-forward to called workflows. | They do not by default. You must pass `secrets: inherit` (or list them explicitly), otherwise the called workflow sees empty secrets. | act_runner #125 | C5, C8 |
| `continue-on-error` | Docs say the job reports success to dependents despite a tolerated step failure. | Source honors this. `workflow.go` `SetContinueOnError` around L216-227 sets the result. | act source (`workflow.go` L216-227) | C9 |
| Problem matchers | Docs say registered matchers render annotations in the UI. | Registration is accepted, but rendering depends on the Gitea UI and cannot be observed from inside the job. | act source (matcher registration path) | C6, C10 |
| `concurrency: cancel-in-progress` | Docs say an in-flight run is cancelled when a newer run starts. | Enforcement is server-side and not observable from within the run. The suite parses and accepts the key only. | Gitea server-side enforcement | C9 |

---

## reusable-workflow secret passthrough fix version

act_runner #125 is now CLOSED, fixed via act PR #41 ("Parse secret inputs in
reusable workflows"). This suite still tests the behavior empirically in C8
(and the forwarding path in C5) rather than assuming a version, because the
fix landed across several act and Gitea releases and older instances may
still be affected. If C8 fails on your instance, the most likely cause is
that you are running a version where the fix has not landed yet. Check the
act_runner #125 thread for the current status rather than gating on a
specific Gitea version number.

---

## Manual eyeball checklist (UI-only behaviors)

A workflow cannot observe its own rendered UI. These behaviors are emitted where
possible and left for a human to confirm in the Gitea web interface after a run.

- [ ] **Annotation rendering.** Open the run for `06-workflow-commands`. Confirm
      the `::warning::`, `::error::`, and `::notice::` messages appear as
      rendered annotations on the relevant steps (not just as raw text in the
      log).
- [ ] **Step summary rendering.** In the same run, confirm the content written to
      `$GITHUB_STEP_SUMMARY` appears in the run's summary view.
- [ ] **Run cancellation.** Trigger a run, then cancel it from the UI while it is
      executing. Confirm the run transitions to a cancelled state and that the
      `aggregate` job reflects it as a failure (not green).
- [ ] **Deployment environment gate.** Note that `environment:` is treated as a
      no-op by act (see the correction table). There is no protection-rule gate
      to enforce, so this item is about confirming that absence rather than
      testing a gate. If you need real environment gating, that is a known gap.

---

## Repository layout

```
.github/
  actionlint.yaml            narrowly-scoped lint config
  problem-matcher.json       problem matcher fixture
  workflows/
    00-suite-runner.yml      entry: probe -> c1..c10 -> aggregate
    01-checkout.yml          C1
    02-artifacts.yml         C2
    03-cache.yml             C3
    04-setup-runtime.yml     C4
    05-contexts.yml          C5
    06-workflow-commands.yml C6
    07-services.yml          C7
    08-composite-reusable.yml C8
    09-control-flow.yml      C9
    10-gitea-divergences.yml C10
lib/
  assert.sh                  self-asserting shell helpers
  composite-greet/action.yml local composite action (used by C8)
fixtures/                    sha256 of the LFS object, etc.
sparse-cone/                 layout for sparse-checkout tests
Makefile                     `make lint` runs actionlint
```

Numeric filename prefixes do NOT imply execution order. Ordering comes only from
`needs:`. The prefixes exist to give humans a stable reading order.

---

## Running the suite

1. Push the repo to your Gitea instance.
2. Confirm every HARD prerequisite above (runner label, version, storage, cache,
   LFS, mirroring, `TEST_SECRET`).
3. Trigger one warm-up `workflow_dispatch`.
4. Read the per-test results. `aggregate` is green only when the probe and every
   `c1`..`c10` are `success`.
5. Walk the manual eyeball checklist for the UI-only behaviors.