agiledancers/docs/s6-validation.md

> Historical validation evidence from application commit `bd898e92d22f6c412e671908770f4d89c8ae12bc`. Commands retain their original paths; current tooling is documented in README.md.

# S6 implementation and operational evidence

Status: IN_REVIEW. Production `v1.0.0` is running on the Synology at
<http://192.168.2.28:8088>. Main publication, the real Gitea deployment and the
isolated NAS recovery/rollback acceptance checks passed. The evidence below
distinguishes real production actions from deliberately isolated fault tests.

## Confirmed environment

- Work is on `feature/S6_deployment`.
- Read-only authenticated inspection of the installed Gitea confirmed run fields
  `path=build.yaml@refs/heads/main`, `status=completed`, `conclusion=success`,
  `event=push`, `head_branch=main` and `head_sha`.
- No remote Git tags existed at the time of that read-only inspection.
- The maintainer created the dedicated DSM account `agiledancers-deploy` and
  confirmed SSH port 22. A separate Ed25519 deployment key was generated outside
  Git on the mini-pc and installed on the NAS.

Subsequent live checks confirmed key-based access after removing group/other
write permissions from the account home. The maintainer installed the dedicated
Docker sudo rule using `deploy/enable-docker.sh`; a subsequent noninteractive
check reports `linux/x86_64`. NAS model is **DS224+**, Compose is
**v2.20.1-6047-g6817716**, `timeout -k` and `curl` are available, and port **8088**
was free. The daemon allows HTTP registry access only through `127.0.0.0/8`.
Deployment therefore uses the NAS-local `127.0.0.1:3000` alias of the same Gitea
registry, without restarting Gitea/Kimai or changing the shared Docker daemon.

## NAS setup and real image preparation

`java ci/Ci.java configure-deployment` succeeded using separate owner-only local
pull/release tokens. It provisioned `/volume1/docker/agiledancers-production`,
the private IP/port configuration and registry pull login, verified the real NAS
preflight, and stored the three deployment secrets and five variables in Gitea.
No secret values are recorded here.

A real preparation probe exercised the production coordinator/API/SSH adapter
and NAS script, stopping explicitly at the tag-creation boundary. It retrieved
both images for verified main commit
`b1544357bf6ea5e1607e1de1fa4872b9510690a8` (Gitea API build run ID **35**), verified
revision/platform labels, selected registry digests and passed Compose v2.20
configuration validation. It released the deployment lock without creating a
Git tag, pending journal, successful release state or application containers.
The displayed candidate `v1.0.0` was not allocated as a release.

| Component | Retrieved digest (NAS registry alias `127.0.0.1:3000/wanjakrah/`) |
| --- | --- |
| Backend | `sha256:d46af705c527a8eaa506407206cfce927edbc4b774951cf13da8702b5cb8c56d` |
| Frontend | `sha256:d6c09ad839b929591878195c7403cbf0bbf24ac6e8d335ef90d72b7982c4ba31` |

Local probe evidence: `ci/artifacts/deployment/setup.log`, `nas-image-probe.log`
and `nas-image-probe.json` (ignored by Git). Retained configuration on the NAS:
`releases/image-probe-bbb2a980-e624-4331-adb6-b8fffd51fa5d/`. This is preparation
evidence, not a successful production release or Gitea dispatch run.

## Local checks

- `java ci/Ci.java test`: **17 passed**, including publication gates and process
  deadlines.
- `java ci/Ci.java release-test`: **32 passed**, including all version increments,
  input/main/manual gates, verified-build provenance, digest preparation failures,
  duplicate/stale release rejection, retry, version high-watermark after rollback,
  automatic recovery, failed recovery/first release, ambiguous state commits,
  lock contention, annotated-tag identity and registry aliases.
- Seven of those release tests execute the actual POSIX NAS script against
  disposable command adapters, covering image/platform checks, published endpoint
  requests, retained-configuration mismatch, bounded readiness failure, overlapping
  replacement fencing and atomic state/journal handling. These are simulated
  failure-path tests, separate from the real NAS preparation probe above.
- `sh -n` passed separately for `deploy/remote.sh`, `deploy/provision.sh` and
  `deploy/enable-docker.sh`.
- `mvn -N -Dformat.mode=check exec:exec@format`: passed with no differences.
- All workflow YAML parsed; assertions confirmed automatic builds, manual-only
  release/rollback triggers and the patch/minor/major form choices.
- `docker compose -f deploy/compose.yaml config --quiet` passed with representative
  digest references; real NAS Compose validation passed during preparation too.
- `git diff --check`: passed.

Backend/frontend application code is unchanged.

## Full Gitea feature build

[Feature build #36](http://192.168.2.28:3000/wanjakrah/agiledancers/actions/runs/36)
(API run ID **37**) completed successfully for implementation commit
`525e9f9335e4343c28f91664c149d24da9284904` on `feature/S6_deployment`. This exercises
the full existing verification path, including Maven verification, both helper
test suites, frontend checks/build, both Docker image builds and container E2E.
Image publication is skipped for the feature branch.

The subsequent evidence-only commit intentionally uses the accepted `[skip ci]`
marker; implementation code is unchanged from this successful run.

Implementation [PR #12](http://192.168.2.28:3000/wanjakrah/agiledancers/pulls/12)
was reviewed and merged through the protected main branch. Production and
post-merge acceptance evidence follow.

## Production deployment and native workflow guards

The main SHA is `b0d6592761cf3ceab6e1ffbf9ca50548584a0096`.

| Evidence | Gitea run | Result |
| --- | --- | --- |
| Main build/test/publication | [#37](http://192.168.2.28:3000/wanjakrah/agiledancers/actions/runs/37), API ID 38 | Success |
| First manual `new` / `patch` release | [#38](http://192.168.2.28:3000/wanjakrah/agiledancers/actions/runs/38), API ID 39 | Success; production `v1.0.0` |
| Deliberate duplicate new release for the same SHA | [#39](http://192.168.2.28:3000/wanjakrah/agiledancers/actions/runs/39), API ID 40 | Expected rejection, production unchanged |
| Manual deployment from a temporary validation branch | [#40](http://192.168.2.28:3000/wanjakrah/agiledancers/actions/runs/40), API ID 41 | Expected skip, production unchanged |

The original feature branch had been deleted after merge; its attempted dispatch
returned 404 and is not counted as branch-guard evidence. The actual skipped run
used `validation/s6-deploy-guard`, based on the existing documentation-only
`60496de` commit. That temporary branch was removed after verification.

The real annotated Git tag `v1.0.0` has tag-object SHA
`0b8df86a9de1729395baad35584e4accead29c44` and points directly to the main SHA above.
Gitea tag protection rule ID **1**, pattern `v*`, allows only `wanjakrah` (the
account owning the dedicated release token) to create/delete release tags.
The helper never moves or deletes them; this does not remove the trusted
maintainer's administrative ability to manage tags.

Production image digests, using the NAS-local `127.0.0.1:3000/wanjakrah/` registry:

| Component | Deployed digest |
| --- | --- |
| Backend | `sha256:1d6f5e35bb58341bfa7b6ed0b88e5bdebced60666ca289d5511613f35d9ee92e` |
| Frontend | `sha256:843a1ffcdbe708573ca55f51a04adee0d158f255f067225a20c6d41b382ce295` |

Both running containers report `v1.0.0` and the selected main SHA. Backend
readiness, frontend and proxied API checks passed in the workflow. A subsequent
read-only GET from the mini-pc to `http://192.168.2.28:8088/api/dance-schools`
returned HTTP success and `[]`. The successful state is recorded under
`releases/release-39-3b481acf-396d-41d6-96a6-6b7ef8710f18/`, with no pending
journal or deployment lock. As this is the first production release, its previous
successful production release is correctly `null`.

### Firewall and diagnostics

The initial readiness wait exposed an active DSM firewall blocking frontend-to-
backend traffic. The maintainer added these **All interfaces** allow rules above
deny rules: TCP **8088** from `192.168.2.0/24`, and TCP **8080** from the actual
production Docker subnet **172.18.0.0/16**. The same first deployment then passed;
there was no extra production version or forced bypass of readiness checks.

Downloaded and opened the actual workflow diagnostic ZIPs:

- `deployment-39-20261103135841995564629`: two files; recorded outcome `success`.
- `deployment-40-20261103141439174434089`: two files; recorded outcome `rejected`.

Both workflows request fourteen-day artifact retention, within the previously
confirmed server retention limits. The ZIPs and run-view metadata are retained
locally in `ci/artifacts/deployment/live-workflows/`. The older Gitea artifact
list API limitation documented under S4 was avoided by downloading the actual
run-page artifact archives.

## Isolated real NAS failure and rollback acceptance

**Twelve checks passed against real NAS Docker containers**, using the production
`Release` coordinator and remote shell script. They used only project
`agiledancers-s6-acceptance`, loopback endpoint `127.0.0.1:18089` and network
`172.20.0.0/16`. State/configuration was isolated under
`/volume1/docker/agiledancers-s6-acceptance`, with `-first` and `-recovery` sibling
directories for first-release and failed-recovery cases.

The adapter changed only the project/network/configuration namespace and injected
controlled faults. **Release tags and main-head selections in this test adapter
were virtual**: no additional real Git tags were created. Build verification and
image pulls used real Gitea/registry responses for verified main SHAs `b1544357...`
and `b0d65927...`. The missing-image case also used verified main SHA `c965f9b4...`.
Fault cases used a ten-second readiness window and healthy cases sixty seconds;
the production workflow retains its normal five-minute replacement bound.

1. A healthy initial isolated release passed the real container/readiness checks.
2. A distinct candidate whose backend deliberately exited with status 23 failed;
   the real previous image pair was restored, the attempt remained failed, and
   its virtual version remained allocated.
3. Retrying the same version/SHA with a healthy configuration succeeded without
   allocating another version.
4. Manual rollback restored the distinct previous SHA without another version;
   a second rollback restored the newer successful release.
5. Retry of an older version after rollback was rejected by the high-watermark.
6. A real missing registry image was rejected before version allocation or
   replacement.
7. An unverified SHA was rejected using real Gitea build metadata.
8. Duplicate and stale releases were rejected while running containers stayed
   unchanged.
9. A competing NAS lock owner excluded a second release attempt.
10. An injected lost state-commit response retained the real pending journal and
    lock, blocked the next attempt and was resolved with explicit manual recovery.
11. A real failed first release retained its journal/lock; manual stop/archive
    recovery left no fabricated successful state.
12. Both a candidate and its recovery configuration were deliberately broken;
    automatic recovery failed visibly and retained its journal/lock. Repairing
    the retained previous configuration and following manual recovery restored
    the recorded previous SHA.

Raw program, logs and `results.json` are retained in
`ci/artifacts/deployment/nas-acceptance/`. The test stack's containers and Docker
network were removed. Test state/history remains for inspection. Final independent
verification confirmed that production still ran the original `v1.0.0` SHA,
without a pending journal or lock, and that no acceptance containers/network
remained. The maintainer was told the temporary firewall rule for `172.20.0.0/16`
can be removed; the two production rules must remain.

These isolated tests prove the NAS replacement/recovery behavior, not a second
successful production release. Native Gitea rollback against a populated
production predecessor can be exercised after a normal subsequent release;
the same coordinator's actual distinct-version rollback is demonstrated above.

## Data boundary

The maintainer explicitly accepted redeployment data loss for the launching-
partner phase until S16 and confirmed the operator-communication policy during
refinement. The workflow and runbook display that limitation. No messages were
sent to the partner on the maintainer's behalf. S16 is not implemented, so saved-
data preservation and migration compatibility remain conditional S16 acceptance,
not a durability claim for this release.

## Required operational acceptance

- [x] Record NAS model/platform, Docker/Compose versions, private endpoint and
      production directory (no secret values).
- [x] Verify pinned SSH hostkey, key-only automation and separate registry pull
      credential. Release-token read access is verified; actual tag-writing
      acceptance awaits the first release.
- [x] Verify release-tag creation/protection with the dedicated token.
- [x] Merge workflows through the protected-branch process and identify the
      successful main build/publication run and SHA.
- [x] Demonstrate release form, first release tag and real Synology access;
      record workflow URL, version, SHA and both deployed digests.
- [x] Demonstrate rejected unverified/missing-image release and branch exclusion
      without modifying production.
- [x] Demonstrate duplicate/stale/concurrent release protection, retained versions
      after failure and exact-SHA retry without another version.
- [x] Demonstrate failed candidate recovery while the coordinator reports failure,
      with downloadable diagnostics.
- [x] Demonstrate manual rollback without a new version and retained image pairs.
- [x] Exercise first-release/recovery failure and interruption/manual recovery
      using isolated acceptance resources where appropriate.
- [x] Confirm fourteen-day artifact retention configuration and artifact downloads.
- [x] Confirm the maintainer's accepted pre-S16 data-loss/communication policy.
- [ ] Once S16 applies, verify data preservation and migration compatibility
      during deployment and rollback (deferred until S16 implementation, not a
      blocker for the explicitly accepted pre-S16 release).

Identify isolated acceptance resources explicitly. Simulated tests are not real
production evidence.