S6: record production deployment and recovery evidence #13

Merged
wanjakrah merged 3 commits from docs/S6-production-validation into main 2026-11-03 16:13:53 +01:00
5 changed files with 2329 additions and 41 deletions

View File

@@ -90,6 +90,28 @@ Docker-locatie en `DOCKER_CONFIG` in. Gebruik een eigen `docker-auth`-submap.
Alleen de frontend bindt aan de private IPv4; de backend krijgt geen hostpoort.
De Docker-projectnaam is vast `agiledancers-production`.
### DSM-firewall
De bevestigde productieomgeving gebruikt frontendpoort **8088** op
`192.168.2.28` en Docker-subnet **172.18.0.0/16**. Maak in
**Configuratiescherm → Beveiliging → Firewall → Regels bewerken** toestaan-regels
op **Alle interfaces**, boven eventuele weigerregels:
| Poort | Protocol | Bron | Actie |
| --- | --- | --- | --- |
| 8088 | TCP | `192.168.2.0`, masker `255.255.255.0` | Toestaan |
| 8080 | TCP | `172.18.0.0`, masker `255.255.0.0` | Toestaan |
De eerste regel geeft LAN-clients toegang; de tweede laat de frontend de backend
bereiken. Voeg voor bestaande VPN-clients zo nodig hun eigen bron-subnet toe
aan de frontendregel. Er wordt geen backendpoort op de NAS gepubliceerd en de
firewall blijft aan. Controleer bij het opnieuw aanmaken van het Docker-netwerk
het werkelijke subnet met `docker network inspect` en pas de regel aan als dat
subnet wijzigt. Gewone deployments vervangen containers en behouden het netwerk.
De tijdelijke S6-acceptatieregel voor `172.20.0.0/16` mag na de geïsoleerde tests
weg; dat testnetwerk en de testcontainers zijn opgeruimd.
Als Docker sudo vereist, laat een beheerder met `visudo` een afzonderlijke
sudoers-regel installeren en valideren, met het gecontroleerde Docker-pad:

View File

@@ -1,8 +1,9 @@
# S6 implementation and operational evidence
Status: IN_PROGRESS. Repository implementation and local validation do not yet
establish an operational Synology deployment. No production release tags have
been created by this implementation session.
Status: IN_REVIEW. Production `v1.0.0` is running on the Synology at
<http://192.168.2.28:8088>. Main publication, the real Gitea deployment and the
isolated NAS recovery/rollback acceptance checks passed. The evidence below
distinguishes real production actions from deliberately isolated fault tests.
## Confirmed environment
@@ -88,9 +89,129 @@ Image publication is skipped for the feature branch.
The subsequent evidence-only commit intentionally uses the accepted `[skip ci]`
marker; implementation code is unchanged from this successful run.
Main build/publication and dispatched production/recovery evidence remain to be
collected after review and the protected merge. Review PR:
[S6 #12](http://192.168.2.28:3000/wanjakrah/agiledancers/pulls/12).
Implementation [PR #12](http://192.168.2.28:3000/wanjakrah/agiledancers/pulls/12)
was reviewed and merged through the protected main branch. Production and
post-merge acceptance evidence follow.
## Production deployment and native workflow guards
The main SHA is `b0d6592761cf3ceab6e1ffbf9ca50548584a0096`.
| Evidence | Gitea run | Result |
| --- | --- | --- |
| Main build/test/publication | [#37](http://192.168.2.28:3000/wanjakrah/agiledancers/actions/runs/37), API ID 38 | Success |
| First manual `new` / `patch` release | [#38](http://192.168.2.28:3000/wanjakrah/agiledancers/actions/runs/38), API ID 39 | Success; production `v1.0.0` |
| Deliberate duplicate new release for the same SHA | [#39](http://192.168.2.28:3000/wanjakrah/agiledancers/actions/runs/39), API ID 40 | Expected rejection, production unchanged |
| Manual deployment from a temporary validation branch | [#40](http://192.168.2.28:3000/wanjakrah/agiledancers/actions/runs/40), API ID 41 | Expected skip, production unchanged |
The original feature branch had been deleted after merge; its attempted dispatch
returned 404 and is not counted as branch-guard evidence. The actual skipped run
used `validation/s6-deploy-guard`, based on the existing documentation-only
`60496de` commit. That temporary branch was removed after verification.
The real annotated Git tag `v1.0.0` has tag-object SHA
`0b8df86a9de1729395baad35584e4accead29c44` and points directly to the main SHA above.
Gitea tag protection rule ID **1**, pattern `v*`, allows only `wanjakrah` (the
account owning the dedicated release token) to create/delete release tags.
The helper never moves or deletes them; this does not remove the trusted
maintainer's administrative ability to manage tags.
Production image digests, using the NAS-local `127.0.0.1:3000/wanjakrah/` registry:
| Component | Deployed digest |
| --- | --- |
| Backend | `sha256:1d6f5e35bb58341bfa7b6ed0b88e5bdebced60666ca289d5511613f35d9ee92e` |
| Frontend | `sha256:843a1ffcdbe708573ca55f51a04adee0d158f255f067225a20c6d41b382ce295` |
Both running containers report `v1.0.0` and the selected main SHA. Backend
readiness, frontend and proxied API checks passed in the workflow. A subsequent
read-only GET from the mini-pc to `http://192.168.2.28:8088/api/dance-schools`
returned HTTP success and `[]`. The successful state is recorded under
`releases/release-39-3b481acf-396d-41d6-96a6-6b7ef8710f18/`, with no pending
journal or deployment lock. As this is the first production release, its previous
successful production release is correctly `null`.
### Firewall and diagnostics
The initial readiness wait exposed an active DSM firewall blocking frontend-to-
backend traffic. The maintainer added these **All interfaces** allow rules above
deny rules: TCP **8088** from `192.168.2.0/24`, and TCP **8080** from the actual
production Docker subnet **172.18.0.0/16**. The same first deployment then passed;
there was no extra production version or forced bypass of readiness checks.
Downloaded and opened the actual workflow diagnostic ZIPs:
- `deployment-39-20261103135841995564629`: two files; recorded outcome `success`.
- `deployment-40-20261103141439174434089`: two files; recorded outcome `rejected`.
Both workflows request fourteen-day artifact retention, within the previously
confirmed server retention limits. The ZIPs and run-view metadata are retained
locally in `ci/artifacts/deployment/live-workflows/`. The older Gitea artifact
list API limitation documented under S4 was avoided by downloading the actual
run-page artifact archives.
## Isolated real NAS failure and rollback acceptance
**Twelve checks passed against real NAS Docker containers**, using the production
`Release` coordinator and remote shell script. They used only project
`agiledancers-s6-acceptance`, loopback endpoint `127.0.0.1:18089` and network
`172.20.0.0/16`. State/configuration was isolated under
`/volume1/docker/agiledancers-s6-acceptance`, with `-first` and `-recovery` sibling
directories for first-release and failed-recovery cases.
The adapter changed only the project/network/configuration namespace and injected
controlled faults. **Release tags and main-head selections in this test adapter
were virtual**: no additional real Git tags were created. Build verification and
image pulls used real Gitea/registry responses for verified main SHAs `b1544357...`
and `b0d65927...`. The missing-image case also used verified main SHA `c965f9b4...`.
Fault cases used a ten-second readiness window and healthy cases sixty seconds;
the production workflow retains its normal five-minute replacement bound.
1. A healthy initial isolated release passed the real container/readiness checks.
2. A distinct candidate whose backend deliberately exited with status 23 failed;
the real previous image pair was restored, the attempt remained failed, and
its virtual version remained allocated.
3. Retrying the same version/SHA with a healthy configuration succeeded without
allocating another version.
4. Manual rollback restored the distinct previous SHA without another version;
a second rollback restored the newer successful release.
5. Retry of an older version after rollback was rejected by the high-watermark.
6. A real missing registry image was rejected before version allocation or
replacement.
7. An unverified SHA was rejected using real Gitea build metadata.
8. Duplicate and stale releases were rejected while running containers stayed
unchanged.
9. A competing NAS lock owner excluded a second release attempt.
10. An injected lost state-commit response retained the real pending journal and
lock, blocked the next attempt and was resolved with explicit manual recovery.
11. A real failed first release retained its journal/lock; manual stop/archive
recovery left no fabricated successful state.
12. Both a candidate and its recovery configuration were deliberately broken;
automatic recovery failed visibly and retained its journal/lock. Repairing
the retained previous configuration and following manual recovery restored
the recorded previous SHA.
Raw program, logs and `results.json` are retained in
`ci/artifacts/deployment/nas-acceptance/`. The test stack's containers and Docker
network were removed. Test state/history remains for inspection. Final independent
verification confirmed that production still ran the original `v1.0.0` SHA,
without a pending journal or lock, and that no acceptance containers/network
remained. The maintainer was told the temporary firewall rule for `172.20.0.0/16`
can be removed; the two production rules must remain.
These isolated tests prove the NAS replacement/recovery behavior, not a second
successful production release. Native Gitea rollback against a populated
production predecessor can be exercised after a normal subsequent release;
the same coordinator's actual distinct-version rollback is demonstrated above.
## Data boundary
The maintainer explicitly accepted redeployment data loss for the launching-
partner phase until S16 and confirmed the operator-communication policy during
refinement. The workflow and runbook display that limitation. No messages were
sent to the partner on the maintainer's behalf. S16 is not implemented, so saved-
data preservation and migration compatibility remain conditional S16 acceptance,
not a durability claim for this release.
## Required operational acceptance
@@ -99,24 +220,25 @@ collected after review and the protected merge. Review PR:
- [x] Verify pinned SSH hostkey, key-only automation and separate registry pull
credential. Release-token read access is verified; actual tag-writing
acceptance awaits the first release.
- [ ] Verify release-tag creation/protection with the dedicated token.
- [ ] Merge workflows through the protected-branch process and identify the
- [x] Verify release-tag creation/protection with the dedicated token.
- [x] Merge workflows through the protected-branch process and identify the
successful main build/publication run and SHA.
- [ ] Demonstrate release form, first release tag and real Synology access;
- [x] Demonstrate release form, first release tag and real Synology access;
record workflow URL, version, SHA and both deployed digests.
- [ ] Demonstrate rejected unverified/missing-image release and branch exclusion
- [x] Demonstrate rejected unverified/missing-image release and branch exclusion
without modifying production.
- [ ] Demonstrate duplicate/stale/concurrent release protection, retained tags
- [x] Demonstrate duplicate/stale/concurrent release protection, retained versions
after failure and exact-SHA retry without another version.
- [ ] Demonstrate failed candidate recovery while the workflow reports failure,
- [x] Demonstrate failed candidate recovery while the coordinator reports failure,
with downloadable diagnostics.
- [ ] Demonstrate manual rollback without a new version and retained image pairs.
- [ ] Exercise first-release/recovery failure and interruption/manual recovery
- [x] Demonstrate manual rollback without a new version and retained image pairs.
- [x] Exercise first-release/recovery failure and interruption/manual recovery
using isolated acceptance resources where appropriate.
- [ ] Confirm fourteen-day artifact retention.
- [ ] Confirm operator communication of the accepted pre-S16 data-loss policy.
- [x] Confirm fourteen-day artifact retention configuration and artifact downloads.
- [x] Confirm the maintainer's accepted pre-S16 data-loss/communication policy.
- [ ] Once S16 applies, verify data preservation and migration compatibility
during deployment and rollback (conditional on S16 implementation).
during deployment and rollback (deferred until S16 implementation, not a
blocker for the explicitly accepted pre-S16 release).
Identify isolated acceptance resources explicitly. Simulated tests are not real
production evidence.

View File

@@ -1,6 +1,6 @@
# S18 — Scheduling dance groups across a show
Status: IN_REVIEW
Status: IMPLEMENTED
As a dance school manager,
I want OR-Tools to schedule groups and order performances using participation,

View File

@@ -1,6 +1,6 @@
# S6 — Manual versioned production deployment through Gitea
Status: IN_PROGRESS
Status: IMPLEMENTED
Agreed after the grill-me interview; shared understanding confirmed.
@@ -141,44 +141,48 @@ does not restore lost data.
## Acceptance criteria
- [ ] Successful main builds still automatically verify and publish matching SHA
- [x] Successful main builds still automatically verify and publish matching SHA
images; publishing does not automatically change production.
- [ ] Gitea's manual workflow form provides new/retry mode and the appropriate
- [x] Gitea's manual workflow form provides new/retry mode and the appropriate
release-type or existing-tag inputs; invalid combinations fail clearly.
- [ ] The first release and subsequent patch/minor/major releases allocate the
- [x] The first release and subsequent patch/minor/major releases allocate the
agreed versions and immutable annotated remote Git tags on verified SHAs.
- [ ] Duplicate new releases, tag conflicts, concurrent attempts, stale main
- [x] Duplicate new releases, tag conflicts, concurrent attempts, stale main
snapshots and older pending deployments cannot overwrite release identity
or replace a newer successful release.
- [ ] An unverified release or missing image is rejected before production changes;
- [x] An unverified release or missing image is rejected before production changes;
branch builds cannot deploy production.
- [ ] A real Synology deployment serves the frontend and proxied API over the
- [x] A real Synology deployment serves the frontend and proxied API over the
private endpoint, with both containers running the selected image pair.
- [ ] Readiness, frontend, proxied read-only API and image-identity checks establish
- [x] Readiness, frontend, proxied read-only API and image-identity checks establish
success within five minutes or trigger recovery.
- [ ] A deliberately failed release restores the previous successful version,
- [x] A deliberately failed release restores the previous successful version,
while the workflow reports failure. Failed first deployment and failed
recovery produce actionable diagnostics and manual recovery instructions.
- [ ] A failed release can be retried with its existing tag/SHA without allocating
- [x] A failed release can be retried with its existing tag/SHA without allocating
another version or superseding a newer successful release.
- [ ] Manual rollback demonstrably restores the previous successful release with
- [x] Manual rollback demonstrably restores the previous successful release with
no new version, and required retained images are available.
- [ ] The temporary pre-S16 data-loss policy is visible in workflow documentation
- [x] The temporary pre-S16 data-loss policy is visible in workflow documentation
and the runbook. Once S16 applies, demonstrate saved data surviving both
deployment and rollback, including the migration compatibility requirement.
- [ ] Server setup and operational recovery documentation are usable; diagnostics
The latter is conditional S16 acceptance and is not applicable yet.
- [x] Server setup and operational recovery documentation are usable; diagnostics
and release outcomes are retained for fourteen days without secrets.
## Implementation handoff
## Implementation and evidence
This specification records the confirmed design. Implementing workflows, release
automation, Compose configuration, Synology provisioning and collecting actual
deployment/recovery evidence are subsequent work. Configuration alone does not
This specification records the confirmed design. The evidence combines automated
behavior tests, real Gitea production release/guard runs and isolated real NAS
fault tests. Their distinct scopes and the virtual-tag boundary of the isolated
tests are documented in `docs/s6-validation.md`. Configuration alone does not
complete S6.
Implementation is in progress on `feature/S6_deployment`: workflows, Compose,
release/retry/rollback helpers, failure-path tests and provisioning/runbook are
provided. The dedicated NAS account, registry login and Gitea deployment secrets
are provisioned. Real image preparation passed without a release or container
replacement. See `docs/s6-validation.md`; workflow execution and production
deployment/recovery evidence remain required after the protected main merge.
Implementation PR #12 is merged. Production `v1.0.0` runs at
`http://192.168.2.28:8088` following a successful main build and real manual Gitea
deployment. Native duplicate/branch guards and downloadable artifacts were
verified. Twelve isolated real NAS checks cover recovery, retry, distinct-version
rollback, failure guards and manual recovery; they use separate state/containers
and virtual test tags, without changing production. The test resources have been
removed. See `docs/s6-validation.md` for run links, digests, evidence boundaries
and the conditional S16 data-persistence acceptance. S6 is ready for review.

File diff suppressed because it is too large Load Diff