runner-test/README.md

gitea-actions-compat-test

A self-asserting compatibility suite for Gitea Actions. Push this repo to a Gitea instance with a registered runner, and 00-suite-runner.yml runs ten independent test workflows that prove Gitea's implementation of GitHub-style actions behaves like GitHub's. The suite uses two strategies depending on what it tests:

• Feature tests (C1, C5, C6, C9): assert the feature works. A failure means the feature is broken.
• Divergence probes (C2-v4, C8, C10-v4): assert the current state of a known Gitea divergence. Both outcomes (divergence present or fixed) are PASS; the assertion message documents which state was found. A failure means something unexpected happened.
• Infrastructure-dependent tests (C3, C4, C7): fully verify when the prerequisite is met. When it isn't (no Docker, no cache server, TLS blocked), assertions are skipped with a ::warning:: documenting what was NOT verified.

A green run means "no unexpected failures." Read the per-test table and the assertion messages in the logs to understand exactly what was verified vs. what was skipped on your instance.

This README is the human-facing guide. It tells you what to set up before you push, what each test verifies, where the docs disagree with the source code, and which behaviors cannot be checked from inside a job (those live in the manual eyeball checklist at the bottom).

---

HARD prerequisites

Only the Gitea version check is a hard gate (the probe job fails if version < 1.22.0). All other prerequisites below are informational: the probe prints best-effort notes about them but does NOT fail if they are missing. Components that depend on them will conditionally skip with ::warning::.

Runner

• A runner registered with the label used by runs-on. The default label is ubuntu-24.04. If your instance uses a different label, set RUNS_ON_LABEL (or edit runs-on: in every workflow) to match, or the jobs will sit in waiting forever.

Gitea version

• Gitea >= 1.22.0. This is the minimum to PARSE/accept these workflows. It is NOT a guarantee that every component passes. Several checks (notably the upload-artifact@v4 regression and reusable-workflow secret passthrough) may still fail on 1.22.x and only turn green on later point releases. See the per-test table below for which components need a newer version.

  To be explicit about the two thresholds:

  Threshold	Meaning
  1.22.0	Minimum to PARSE/accept the workflow YAML. Below this, the runner rejects the suite outright.
  higher, version-dependent	Minimum to PASS all components. Each component that needs more lists its version note in the per-test table.

Storage and caches

• Storage backend enabled. The artifact and cache actions need object storage (local, MinIO, S3-compatible). Without it, uploads and downloads error out.
• Cache server enabled. actions/cache@v3 relies on the Gitea cache server. Confirm it is reachable before expecting C3 to pass.
• LFS enabled. test.bin is an LFS object. The checkout LFS test (C1) and the fixtures depend on git lfs working on both the server and the runner.

Action mirroring

• Action mirroring allowed. ACTIONS_URL and DEFAULT_ACTIONS_URL must NOT be set to a non-mirror strategy. The workflows reference actions by the owner/repo@ref shorthand (for example actions/checkout@v4), which Gitea resolves by mirroring the upstream action on first use. If mirroring is off or pointed at a custom non-mirror source, those uses: refs 404 and the run fails at fetch time. The mirror fetch is itself a behavior under test.

Secrets

• A repo secret TEST_SECRET=placeholder. Define it in the repo settings. The reusable workflows that read it (05-contexts and 08-composite-reusable) receive it through explicit forwarding (secrets: inherit) in the caller jobs, because Gitea's act_runner does not auto-forward caller secrets into reusable workflows by default (see act_runner #125 in the correction table). The placeholder value placeholder is what the assertion checks for, so do not put anything else there unless you also update the assertion.

---

Warm-up: run workflow_dispatch once

After first setup, trigger the suite once via the Run workflow button (workflow_dispatch) before drawing any conclusions from a push run. Action mirroring is lazy. The first fetch of each actions/* ref can take a moment to populate, and a cold run may show spurious fetch failures that vanish on the second attempt. One warm-up dispatch lets the mirror settle.

---

Per-test expectations

The suite is split into ten reusable workflows (on: workflow_call), each invoked by a caller job (c1 through c10) in 00-suite-runner.yml. Each component reports its own pass/fail, and aggregate fails the whole run if any component is not success (a skipped or cancelled component counts as failure).

#	Workflow	What it verifies	Expected result	Min-version notes
C1	01-checkout	actions/checkout@v4 variants: default checkout, fetch-depth, lfs, ref to a tag, path, sparse-checkout cone, and persist-credentials (both true and false, with a non-vacuous assertion that the embedded credential token differs between the two).	Pass on Gitea >= 1.22 with LFS enabled. The LFS sub-case is conditional on git lfs being present.	1.22.0 to parse. LFS sub-case needs LFS enabled on both server and runner.
C2	02-artifacts	Artifacts v3 and v4 round-trip, cross-job checksum comparison (producer writes, consumer verifies sha256).	v3 fully verified. v4 upload uses continue-on-error: true; if it fails (GHESNotSupportedError), the test asserts regression-confirmed (Gitea #31256). If it succeeds, full v4 checksum verification runs.	v4 needs a point release after the #31256 fix. v3 works on 1.22.
C3	03-cache	Cache save/restore: a seed job writes a file and caches it (first run is a miss that saves), a restore job asserts an exact-key hit, then a miss-with-restore-keys fallback.	Fully verified when cache server is reachable. When cache server is down, all cache assertions are skipped with ::warning::.	Cache server must be configured for meaningful verification.
C4	04-setup-runtime	setup-node, setup-python, setup-go, setup-java (Temurin, Maven) in a matrix, each asserting the installed version.	Fully verified when setup actions can download runtimes. If TLS/network blocks downloads (self-signed cert), assertions are skipped with ::warning::.	1.22.0. Runtime downloads require internet access with valid TLS.
C5	05-contexts	github.sha / github.ref / github.run_id, a job env: var round-tripping into a step, the auto-injected GITHUB_TOKEN (via env: mapping), the forwarded TEST_SECRET (skipped if empty), needs outputs consumed by a dependent job, and if: conditionals with success() / always(). failure() tested via step-level continue-on-error + outcome/conclusion assertion.	Pass. TEST_SECRET assertion is skipped with a message if the repo secret is unset.	1.22.0 to parse. Secret forwarding depends on act_runner #125 (see correction table).
C6	06-workflow-commands	GITHUB_OUTPUT (multi-line value, consumed via step output), GITHUB_PATH (added dir becomes callable), GITHUB_ENV (set in one step, read in the next), and ::warning:: / ::error:: / ::notice:: annotations plus $GITHUB_STEP_SUMMARY.	Pass. The annotation assertion is emit-only: it checks the command was accepted and the process did not crash. Annotation RENDERING in the UI is a manual checklist item.	1.22.0. Rendering is UI-dependent (manual).
C7	07-services	postgres:16 and redis:7 services with healthchecks, plus a container: node:20 job.	Fully verified when Docker is available. On host-mode runners (no Docker), service/container assertions are skipped with ::warning::.	1.22.0. Docker on the runner is required for meaningful verification.
C8	08-composite-reusable	A local composite action (lib/composite-greet) with input/output round-trip, and a reusable workflow that declares inputs and secrets on its workflow_call trigger.	Composite action fully verified. inputs.who and TEST_SECRET assertions document the current state of act_runner #125: PASS with "divergence-confirmed" if empty, PASS with value check if populated.	Version-dependent. See the note below.
C9	09-control-flow	timeout-minutes at step level (step killed, outcome asserted as failure via continue-on-error), continue-on-error outcome/conclusion split, and concurrency as parse-and-accept only.	Pass. If step-level timeout-minutes is not enforced by the runner, a ::warning:: is emitted. Concurrency cancellation is manual-only.	1.22.0. Step-level timeout-minutes may not be supported on all runner versions.
C10	10-gitea-divergences	environment: non-blocking proof (fully verified), upload-artifact@v4 regression re-confirm (asserts regression-confirmed if v4 fails, v4-ok if it succeeds), and problem matchers emitted (emit-only).	environment: non-blocking fully verified. v4 reconfirm asserts current state of #31256. Matcher rendering is manual-only.	1.22.0 to parse. v4 needs the fix from #31256.

---

Correction table: where docs and source disagree

GitHub's public docs describe the canonical runtime. Gitea's act implementation sometimes diverges. The table below records the known divergences with source citations and the covering test. Keep this bounded: at most 10 rows.

topic	doc-claim	source-truth	citation	covering-test
timeout-minutes	Docs say it cancels a runaway job.	Source enforces it. job_executor.go around L370-380 applies the timeout and kills the job.	act source (job_executor.go L370-380)	C9
success / failure / cancelled / always	Docs describe four status-check functions.	All four are implemented. interpreter.go around L621-640 defines them.	act source (interpreter.go L621-640)	C5
environment:	Docs say it gates deployments behind protection rules.	Genuinely unsupported. There is no environment field on the Job struct in act, so the key is parsed but treated as a no-op.	act source (Job struct)	C10
upload-artifact@v4	Docs frame v4 as a drop-in replacement for v3.	Not on Gitea 1.22. v4 triggers a GHES-style regression error. Reported fixed but may recur on certain configurations (see Gitea issue #36024 for a Nov 2025 recurrence on Docker-based runners).	Gitea issue #31256	C2, C10
Reusable-workflow secret inheritance	Docs say secrets auto-forward to called workflows.	They do not by default. You must pass secrets: inherit (or list them explicitly), otherwise the called workflow sees empty secrets.	act_runner #125	C5, C8
continue-on-error	Docs say the job reports success to dependents despite a tolerated step failure.	Source honors this. workflow.go SetContinueOnError around L216-227 sets the result.	act source (workflow.go L216-227)	C9
Problem matchers	Docs say registered matchers render annotations in the UI.	Registration is accepted, but rendering depends on the Gitea UI and cannot be observed from inside the job.	act source (matcher registration path)	C6, C10
concurrency: cancel-in-progress	Docs say an in-flight run is cancelled when a newer run starts.	Enforcement is server-side and not observable from within the run. The suite parses and accepts the key only.	Gitea server-side enforcement	C9

---

reusable-workflow secret passthrough fix version

act_runner #125 is now CLOSED, fixed via act PR #41 ("Parse secret inputs in reusable workflows"). This suite still tests the behavior empirically in C8 (and the forwarding path in C5) rather than assuming a version, because the fix landed across several act and Gitea releases and older instances may still be affected. If C8 fails on your instance, the most likely cause is that you are running a version where the fix has not landed yet. Check the act_runner #125 thread for the current status rather than gating on a specific Gitea version number.

---

Manual eyeball checklist (UI-only behaviors)

A workflow cannot observe its own rendered UI. These behaviors are emitted where possible and left for a human to confirm in the Gitea web interface after a run.

• Annotation rendering. Open the run for 06-workflow-commands. Confirm the ::warning::, ::error::, and ::notice:: messages appear as rendered annotations on the relevant steps (not just as raw text in the log).
• Step summary rendering. In the same run, confirm the content written to $GITHUB_STEP_SUMMARY appears in the run's summary view.
• Run cancellation. Trigger a run, then cancel it from the UI while it is executing. Confirm the run transitions to a cancelled state and that the aggregate job reflects it as a failure (not green).
• Deployment environment gate. Note that environment: is treated as a no-op by act (see the correction table). There is no protection-rule gate to enforce, so this item is about confirming that absence rather than testing a gate. If you need real environment gating, that is a known gap.

---

Repository layout

.github/
  actionlint.yaml            narrowly-scoped lint config
  problem-matcher.json       problem matcher fixture
  workflows/
    00-suite-runner.yml      entry: probe -> c1..c10 -> aggregate
    01-checkout.yml          C1
    02-artifacts.yml         C2
    03-cache.yml             C3
    04-setup-runtime.yml     C4
    05-contexts.yml          C5
    06-workflow-commands.yml C6
    07-services.yml          C7
    08-composite-reusable.yml C8
    09-control-flow.yml      C9
    10-gitea-divergences.yml C10
lib/
  assert.sh                  self-asserting shell helpers
  composite-greet/action.yml local composite action (used by C8)
fixtures/                    sha256 of the LFS object, etc.
sparse-cone/                 layout for sparse-checkout tests
Makefile                     `make lint` runs actionlint

Numeric filename prefixes do NOT imply execution order. Ordering comes only from needs:. The prefixes exist to give humans a stable reading order.

---

Running the suite

1. Push the repo to your Gitea instance.
2. Confirm every HARD prerequisite above (runner label, version, storage, cache, LFS, mirroring, TEST_SECRET).
3. Trigger one warm-up workflow_dispatch.
4. Read the per-test results. aggregate is green only when the probe and every c1..c10 are success.
5. Walk the manual eyeball checklist for the UI-only behaviors.