From c9db5287568377d567569feda9618c8b3536444f Mon Sep 17 00:00:00 2001 From: Wanja Krah Date: Tue, 3 Nov 2026 13:30:47 +0100 Subject: [PATCH 1/3] refinement of spec S6 --- .../S6_Deployment-gitea-action-to-production | 3 - ...6_deployment-gitea-action-to-production.md | 177 ++++++++++++++++++ 2 files changed, 177 insertions(+), 3 deletions(-) delete mode 100644 specs/S6_Deployment-gitea-action-to-production create mode 100644 specs/refined/S6_deployment-gitea-action-to-production.md diff --git a/specs/S6_Deployment-gitea-action-to-production b/specs/S6_Deployment-gitea-action-to-production deleted file mode 100644 index cefc1bc..0000000 --- a/specs/S6_Deployment-gitea-action-to-production +++ /dev/null @@ -1,3 +0,0 @@ -As developer -I want a succeeded branch on main to be deployed to a production environment -so that the latest succesfull build is always running and accissasible at the Synology server \ No newline at end of file diff --git a/specs/refined/S6_deployment-gitea-action-to-production.md b/specs/refined/S6_deployment-gitea-action-to-production.md new file mode 100644 index 0000000..1bf322d --- /dev/null +++ b/specs/refined/S6_deployment-gitea-action-to-production.md @@ -0,0 +1,177 @@ +# S6 — Manual versioned production deployment through Gitea + +Status: REFINED + +Agreed after the grill-me interview; shared understanding confirmed. + +## User story + +As a developer, I want to manually release a verified main-branch build from +Gitea to the Synology, choosing a semantic version increment, so that I control +when the launching partner receives a new production version and can recover +the previous working version when necessary. + +This replaces the original automatic-deployment story. Build, test and image +publication remain automatic; production deployment requires a deliberate action. + +## Existing foundation and scope + +- Extend the implemented S4 pipeline, which verifies builds and publishes matching + frontend/backend images with immutable full commit SHA tags for successful main + builds. Deploy those existing images without rebuilding them. +- Target one production Docker Compose stack on the Synology (`linux/amd64`). + The existing Gitea runner triggers deployment over SSH. +- Keep one active backend. A brief restart interruption is acceptable, including + interruption of a scheduling calculation; the user may restart the calculation. +- Production is private to the LAN or existing VPN. A configurable Synology + hostname/IP and port provide access without an application login. +- Publish only the frontend port. Keep the backend reachable within the Compose + network; route browser API requests through the frontend proxy. +- Actual server addresses, ports, paths and account configuration must be + established and documented during implementation, without committing secrets. + +## Manual release interface + +Provide a separate **Deploy production** workflow using Gitea's native manual +workflow trigger (`workflow_dispatch`). Its Run workflow popup offers: + +- **New release** or **Retry release** mode. +- For a new release: a release-type choice of **patch**, **minor** or **major**. +- For a retry: the existing release version tag. + +Provide a manual rollback action for the previous successful production release. +The native Gitea button need not be renamed or embedded into an existing build +run. Gitea 1.25.4 supports the required modal and choice inputs. + +### New release eligibility and ordering + +- Run the release workflow from main. Branch builds cannot deploy production. +- Pin the current main SHA at workflow start. Require successful build checks and + image publication for that exact SHA; missing or failed verification leaves + production unchanged. Both images must be retrievable before replacement. +- Recheck main immediately before changing production. If it has advanced, abort + without changing production. Once replacement starts, finish deployment or + recovery using the pinned SHA; later commits require another manual release. +- Serialize deployment, recovery, rollback, version allocation and tag creation. + Prevent stale runs from replacing newer successful production releases. +- Reject a new release for an already release-tagged commit and direct the + operator to retry or rollback instead. Repeated clicks must not create extra + versions or deploy out of order. + +## Semantic versioning and Git tags + +- Use one shared application version for frontend and backend, represented by + an immutable annotated Git tag `vMAJOR.MINOR.PATCH` on the verified SHA. +- The first release is `v1.0.0`. Inspect remote tags before initialization and + fail visibly on conflicting existing history instead of overwriting it. +- Subsequent releases increment the highest existing release version tag, + including tags belonging to failed deployments. A rollback does not lower the + version baseline. Patch increments PATCH; minor increments MINOR and resets + PATCH; major increments MAJOR and resets MINOR and PATCH. +- For example, from `v1.2.3`, the choices produce `v1.2.4`, `v1.3.0` or `v2.0.0`. +- After checking eligibility and image availability, create the tag before + changing production. This establishes tag-writing access before replacement. + Never move a release tag or reuse its version for different code. +- Retain a tag if deployment subsequently fails or aborts. Record the outcome so + a tag is not mistaken for proof of successful production deployment. +- A retry reuses the existing tag and exact SHA without allocating a version. + Require successful build verification and matching available images. Retrying + an older release must not supersede a newer successful release. Retrying an + existing release is an explicit exception to the current-main requirement. +- Rollback restores an existing release and creates no new version. It is an + explicit exception to the prohibition on replacing newer code with older code. +- Continue using SHA-tagged container images; the Git release tag and deployment + record map the semantic version to that SHA and image pair. +- Maven/npm version fields and automated changelog generation are outside S6. + Do not introduce a version-bump commit and rebuild into the release path. + +## Deployment checks and recovery + +- Retrieve the candidate images and validate prerequisites before replacing the + running release. Frontend and backend must use the same selected commit SHA. +- Within five minutes of starting the replacement, require all of: + - Backend readiness succeeds. + - The frontend page is served. + - A read-only API request succeeds through the frontend proxy. + - Both running containers match the selected SHA/image pair. +- Checks must not create or modify user data. Log the selected SHA/version and + final outcome. +- A failed candidate automatically restores the previous successful image pair + and checks its health. The workflow remains failed even if recovery succeeds. +- Retain references to the current and previous successful releases and ensure + their images remain available. A manual rollback targets that previous release; + fail before modifying production if its required images cannot be retrieved. +- If the first deployment fails with no previous release, or recovery itself + fails, stop with a clear failure report and manual recovery instructions. + Preserve diagnostics; do not enter an endless deployment/recovery loop. +- Record successful production release state separately from allocated Git tags, + so failed attempts do not become the rollback target. + +## Data policy and S16/S17 boundaries + +S16 is **not a prerequisite** for S6. Until S16 is implemented, application state +is held in memory and may be lost during deployment, rollback or recovery. The +launching-partner phase explicitly accepts this limitation. Restoring old images +does not restore lost data. + +- State the temporary data-loss limitation in the workflow description and + deployment runbook. The operator is responsible for informing the launching + partner; do not add a further approval step to each deployment. +- After S16 is implemented, deployments and rollbacks must preserve saved data. + Keep persistent storage outside replaceable containers and do not delete it + during deployment or recovery. +- After S16, ordinary release and rollback paths require database compatibility + with the previous application version. Incompatible migrations require a + separately planned maintenance procedure; image rollback cannot undo them. +- Database backup and restore remain separate S17 work. S6 does not promise + recovery of data lost before persistence or through storage loss. + +## Credentials and operational documentation + +- Use a dedicated Synology deployment account with SSH key authentication and a + pinned server host key. Document the narrowly required deployment permissions. +- Use a separate registry credential with pull access for production and a + credential with the required repository permission for release-tag creation. +- Store deployment secrets in Gitea Actions secrets, provision server-side + credentials securely where needed, and keep secret values out of Git and logs. +- Document initial provisioning, configuration, release/retry/rollback operation, + credential setup, health checks, release records and manual recovery. +- Retain deployment diagnostics for fourteen days, including failed deployment + and recovery outcomes, without exposing credentials. + +## Acceptance criteria + +- [ ] Successful main builds still automatically verify and publish matching SHA + images; publishing does not automatically change production. +- [ ] Gitea's manual workflow form provides new/retry mode and the appropriate + release-type or existing-tag inputs; invalid combinations fail clearly. +- [ ] The first release and subsequent patch/minor/major releases allocate the + agreed versions and immutable annotated remote Git tags on verified SHAs. +- [ ] Duplicate new releases, tag conflicts, concurrent attempts, stale main + snapshots and older pending deployments cannot overwrite release identity + or replace a newer successful release. +- [ ] An unverified release or missing image is rejected before production changes; + branch builds cannot deploy production. +- [ ] A real Synology deployment serves the frontend and proxied API over the + private endpoint, with both containers running the selected image pair. +- [ ] Readiness, frontend, proxied read-only API and image-identity checks establish + success within five minutes or trigger recovery. +- [ ] A deliberately failed release restores the previous successful version, + while the workflow reports failure. Failed first deployment and failed + recovery produce actionable diagnostics and manual recovery instructions. +- [ ] A failed release can be retried with its existing tag/SHA without allocating + another version or superseding a newer successful release. +- [ ] Manual rollback demonstrably restores the previous successful release with + no new version, and required retained images are available. +- [ ] The temporary pre-S16 data-loss policy is visible in workflow documentation + and the runbook. Once S16 applies, demonstrate saved data surviving both + deployment and rollback, including the migration compatibility requirement. +- [ ] Server setup and operational recovery documentation are usable; diagnostics + and release outcomes are retained for fourteen days without secrets. + +## Implementation handoff + +This specification records the confirmed design. Implementing workflows, release +automation, Compose configuration, Synology provisioning and collecting actual +deployment/recovery evidence are subsequent work. Configuration alone does not +complete S6. From 525e9f9335e4343c28f91664c149d24da9284904 Mon Sep 17 00:00:00 2001 From: Wanja Krah Date: Tue, 3 Nov 2026 14:16:52 +0100 Subject: [PATCH 2/3] Implement manual versioned Synology deployments for S6 --- .gitea/workflows/deploy.yaml | 68 +++ .gitea/workflows/rollback.yaml | 42 ++ README.md | 5 + ci/Ci.java | 5 +- ci/Pipeline.java | 1 + ci/Release.java | 234 ++++++++ ci/ReleaseApi.java | 124 ++++ ci/ReleaseRemote.java | 119 ++++ ci/ReleaseSetup.java | 107 ++++ ci/ReleaseShellTests.java | 197 +++++++ ci/ReleaseState.java | 80 +++ ci/ReleaseTests.java | 549 ++++++++++++++++++ ci/ReleaseVersion.java | 39 ++ deploy/compose.yaml | 34 ++ deploy/enable-docker.sh | 28 + deploy/production.env.example | 9 + deploy/provision.sh | 38 ++ deploy/remote.sh | 186 ++++++ docs/gitea-ci.md | 4 +- docs/production-deployment.md | 281 +++++++++ docs/s6-validation.md | 107 ++++ ...6_deployment-gitea-action-to-production.md | 9 +- 22 files changed, 2263 insertions(+), 3 deletions(-) create mode 100644 .gitea/workflows/deploy.yaml create mode 100644 .gitea/workflows/rollback.yaml create mode 100644 ci/Release.java create mode 100644 ci/ReleaseApi.java create mode 100644 ci/ReleaseRemote.java create mode 100644 ci/ReleaseSetup.java create mode 100644 ci/ReleaseShellTests.java create mode 100644 ci/ReleaseState.java create mode 100644 ci/ReleaseTests.java create mode 100644 ci/ReleaseVersion.java create mode 100644 deploy/compose.yaml create mode 100644 deploy/enable-docker.sh create mode 100644 deploy/production.env.example create mode 100644 deploy/provision.sh create mode 100644 deploy/remote.sh create mode 100644 docs/production-deployment.md create mode 100644 docs/s6-validation.md rename specs/{refined => in_progress}/S6_deployment-gitea-action-to-production.md (95%) diff --git a/.gitea/workflows/deploy.yaml b/.gitea/workflows/deploy.yaml new file mode 100644 index 0000000..1c167c5 --- /dev/null +++ b/.gitea/workflows/deploy.yaml @@ -0,0 +1,68 @@ +name: Deploy production +on: + workflow_dispatch: + inputs: + mode: + description: 'Release mode — deployment can erase data until S16 is implemented' + required: true + type: choice + default: new + options: + - new + - retry + increment: + description: 'New release only: semantic version increment (first release is v1.0.0)' + required: true + type: choice + default: patch + options: + - patch + - minor + - major + version: + description: 'Retry only: existing version tag, e.g. v1.2.3; leave empty for new release' + required: false + type: string + +# Gitea 1.25 ignores concurrency. The helper acquires a NAS-wide mkdir lock. +jobs: + deploy: + if: gitea.ref == 'refs/heads/main' + runs-on: agiledancers-mini-pc + env: + CI: 'true' + GITEA_SERVER_URL: ${{ gitea.server_url }} + CI_REPOSITORY: ${{ gitea.repository }} + CI_COMMIT: ${{ gitea.sha }} + CI_REF: ${{ gitea.ref }} + CI_EVENT: ${{ gitea.event_name }} + CI_RUN_ID: ${{ gitea.run_id }} + steps: + - name: Check out trusted main workflow revision + uses: https://github.com/actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 + with: + persist-credentials: false + - name: Identify workflow attempt + run: java ci/Ci.java attempt + - name: Release verified images — data can be lost until S16 + env: + RELEASE_MODE: ${{ inputs.mode }} + RELEASE_INCREMENT: ${{ inputs.increment }} + RELEASE_VERSION: ${{ inputs.version }} + RELEASE_TOKEN: ${{ secrets.RELEASE_TOKEN }} + DEPLOY_HOST: ${{ vars.DEPLOY_HOST }} + DEPLOY_USER: ${{ vars.DEPLOY_USER }} + DEPLOY_SSH_PORT: ${{ vars.DEPLOY_SSH_PORT }} + DEPLOY_DIRECTORY: ${{ vars.DEPLOY_DIRECTORY }} + DEPLOY_REGISTRY: ${{ vars.DEPLOY_REGISTRY }} + DEPLOY_SSH_KEY: ${{ secrets.DEPLOY_SSH_KEY }} + DEPLOY_KNOWN_HOSTS: ${{ secrets.DEPLOY_KNOWN_HOSTS }} + run: java ci/Ci.java deploy + - name: Retain deployment diagnostics for fourteen days + if: always() + uses: https://github.com/actions/upload-artifact@ff15f0306b3f739f7b6fd43fb5d26cd321bd4de5 + with: + name: deployment-${{ gitea.run_id }}-${{ env.CI_RUN_ATTEMPT }} + path: ci/artifacts/deployment/ + if-no-files-found: warn + retention-days: 14 diff --git a/.gitea/workflows/rollback.yaml b/.gitea/workflows/rollback.yaml new file mode 100644 index 0000000..1f3721d --- /dev/null +++ b/.gitea/workflows/rollback.yaml @@ -0,0 +1,42 @@ +name: Rollback production +on: + workflow_dispatch: + +jobs: + rollback: + if: gitea.ref == 'refs/heads/main' + runs-on: agiledancers-mini-pc + env: + CI: 'true' + GITEA_SERVER_URL: ${{ gitea.server_url }} + CI_REPOSITORY: ${{ gitea.repository }} + CI_COMMIT: ${{ gitea.sha }} + CI_REF: ${{ gitea.ref }} + CI_EVENT: ${{ gitea.event_name }} + CI_RUN_ID: ${{ gitea.run_id }} + steps: + - name: Check out trusted main workflow revision + uses: https://github.com/actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 + with: + persist-credentials: false + - name: Identify workflow attempt + run: java ci/Ci.java attempt + - name: Restore previous successful release — data can be lost until S16 + env: + RELEASE_TOKEN: ${{ secrets.RELEASE_TOKEN }} + DEPLOY_HOST: ${{ vars.DEPLOY_HOST }} + DEPLOY_USER: ${{ vars.DEPLOY_USER }} + DEPLOY_SSH_PORT: ${{ vars.DEPLOY_SSH_PORT }} + DEPLOY_DIRECTORY: ${{ vars.DEPLOY_DIRECTORY }} + DEPLOY_REGISTRY: ${{ vars.DEPLOY_REGISTRY }} + DEPLOY_SSH_KEY: ${{ secrets.DEPLOY_SSH_KEY }} + DEPLOY_KNOWN_HOSTS: ${{ secrets.DEPLOY_KNOWN_HOSTS }} + run: java ci/Ci.java rollback + - name: Retain deployment diagnostics for fourteen days + if: always() + uses: https://github.com/actions/upload-artifact@ff15f0306b3f739f7b6fd43fb5d26cd321bd4de5 + with: + name: rollback-${{ gitea.run_id }}-${{ env.CI_RUN_ATTEMPT }} + path: ci/artifacts/deployment/ + if-no-files-found: warn + retention-days: 14 diff --git a/README.md b/README.md index c8ea59f..b8d8e5e 100644 --- a/README.md +++ b/README.md @@ -266,6 +266,11 @@ Only successful `main` runs publish images to the Synology registry. See image names, diagnostics and recovery, and [validation evidence](docs/s4-validation.md) for the current operational status. +Production releases and rollback use separate manual Gitea workflows. See the +[deployment runbook](docs/production-deployment.md) for Synology provisioning, +semantic version tags, release/retry/rollback and the accepted data-loss limitation +until S16, and [S6 evidence](docs/s6-validation.md) for deployment readiness. + ### Java formatting (S8) Maven is the authoritative formatting path. The root build runs the actual diff --git a/ci/Ci.java b/ci/Ci.java index c08c052..48e3c97 100644 --- a/ci/Ci.java +++ b/ci/Ci.java @@ -37,6 +37,9 @@ public class Ci { case "install-runner" -> Setup.installRunner(root, env, commands, args.length == 2 && args[1].equals("--prepare-only")); case "test" -> CiTests.run(); + case "deploy", "rollback" -> Release.execute(root, env, args[0]); + case "release-test" -> ReleaseTests.run(); + case "configure-deployment" -> ReleaseSetup.configure(root, env); default -> throw new IllegalArgumentException(usage()); } } catch (Commands.Failure e) { @@ -54,6 +57,6 @@ public class Ci { static String usage() { return "Usage: java ci/Ci.java verify|publish|cleanup|attempt|test|configure-gitea|configure-docker" - + " |install-runner [--prepare-only]"; + + " |install-runner [--prepare-only]|deploy|rollback|release-test|configure-deployment"; } } diff --git a/ci/Pipeline.java b/ci/Pipeline.java index 447ca7e..6a3b7ef 100644 --- a/ci/Pipeline.java +++ b/ci/Pipeline.java @@ -101,6 +101,7 @@ class Pipeline { run("node", "--version"); run("mvn", "-B", "-ntp", "-Dformat.mode=check", "clean", "verify"); CiTests.run(); + ReleaseTests.run(); Path frontend = root.resolve("agiledancers-fe"); verifyFrontend(); String platform = env.get("TARGET_PLATFORM"); diff --git a/ci/Release.java b/ci/Release.java new file mode 100644 index 0000000..043fef8 --- /dev/null +++ b/ci/Release.java @@ -0,0 +1,234 @@ +import java.net.URI; +import java.nio.file.Files; +import java.nio.file.Path; +import java.time.Duration; +import java.time.Instant; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.UUID; + +/** Manual release coordinator. All mutation decisions are made under the NAS lock. */ +class Release { + final Path root, artifacts; + final Map env; + final ReleaseApi api; + final ReleaseRemote remote; + final String mode, increment, retry, pinned, attempt; + ReleaseState.Deployment selected; + boolean locked, pending; + + static void execute(Path root, Map env, String operation) throws Exception { + validate(env, operation); + String attempt = "release-" + env.get("CI_RUN_ID") + "-" + UUID.randomUUID(); + Path artifacts = root.resolve("ci/artifacts/deployment"); + Files.createDirectories(artifacts); + var commands = new Commands(Duration.ofMinutes(15)); + commands.log = artifacts.resolve(attempt + ".log"); + Files.writeString(commands.log, ""); + try (var remote = new ReleaseRemote(root, env, commands, attempt)) { + new Release(root, env, new ReleaseApi(env), remote, operation, attempt).run(); + } + } + + static void validate(Map env, String operation) { + Pipeline.check("refs/heads/main".equals(env.get("CI_REF")), "Production workflows must run from main"); + Pipeline.check( + "workflow_dispatch".equals(env.get("CI_EVENT")), "Production requires a manual workflow dispatch"); + Pipeline.check(Setup.required(env, "CI_RUN_ID").matches("[0-9]+"), "Invalid Gitea run ID"); + ReleaseApi.sha(env.get("CI_COMMIT")); + Pipeline.check(List.of("deploy", "rollback").contains(operation), "Unknown release operation"); + if (operation.equals("rollback")) return; + String mode = Setup.required(env, "RELEASE_MODE"); + String increment = env.getOrDefault("RELEASE_INCREMENT", ""); + String retry = env.getOrDefault("RELEASE_VERSION", ""); + if (mode.equals("new")) { + Pipeline.check(List.of("patch", "minor", "major").contains(increment), "Choose patch, minor or major"); + Pipeline.check(retry.isBlank(), "New release cannot specify an existing version"); + } else { + Pipeline.check(mode.equals("retry"), "Choose new or retry release mode"); + ReleaseVersion.parse(retry); + // Gitea shows all inputs; the new-release increment is ignored for retries. + } + } + + Release(Path root, Map env, ReleaseApi api, ReleaseRemote remote, String operation, String attempt) + throws Exception { + this.root = root; + this.env = env; + this.api = api; + this.remote = remote; + this.attempt = attempt; + artifacts = root.resolve("ci/artifacts/deployment"); + Files.createDirectories(artifacts); + mode = operation.equals("rollback") ? "rollback" : env.get("RELEASE_MODE"); + increment = env.getOrDefault("RELEASE_INCREMENT", ""); + retry = env.getOrDefault("RELEASE_VERSION", ""); + pinned = ReleaseApi.sha(env.get("CI_COMMIT")); + } + + void run() throws Exception { + String outcome = "rejected"; + try { + remote.acquire(); + locked = true; + remote.call("preflight", null); + ReleaseState before = ReleaseState.parse(remote.call("state", null)); + selected = select(before); + System.out.println("Selected " + selected.version() + " at " + selected.sha()); + // Validate the recovery pair before any replacement too. + if (before.current() != null) ensure(before.current()); + if (mode.equals("new")) { + api.create(selected.version(), selected.sha(), attempt); + Pipeline.check( + selected.sha().equals(api.head()), + "Main advanced; release tagged but production unchanged. Retry explicitly."); + } + var journal = new LinkedHashMap(); + journal.put("attempt", attempt); + journal.put("mode", mode); + journal.put("candidate", selected.json()); + journal.put("before", Json.parse(before.current() == null ? "{}" : before.json())); + remote.call("pending", Json.write(journal)); + pending = true; + outcome = "replacement-started"; + try { + remote.call("apply", null, selected.directory()); + } catch (Exception failure) { + diagnose(selected); + outcome = recover(before); + throw new IllegalArgumentException( + "Deployment failed: " + outcome + ". See deployment diagnostics and recovery runbook."); + } + // A lost reply here is ambiguous: retain journal and lock, never guess + // whether a state commit happened and launch an overlapping recovery. + outcome = "healthy-state-commit-pending"; + remote.call("commit", before.success(selected).json()); + outcome = "success"; + record(outcome); + remote.call("finish", null); + pending = false; + System.out.println("Production " + selected.version() + " is healthy (" + selected.sha() + ")"); + } finally { + try { + record(outcome); + } finally { + if (locked && !pending) remote.release(); + else if (locked) + System.err.println("NAS lock and pending journal retained. Follow docs/production-deployment.md for" + + " recovery."); + } + } + } + + ReleaseState.Deployment select(ReleaseState before) throws Exception { + if (mode.equals("rollback")) { + Pipeline.check(before.previous() != null, "No previous successful production release is available"); + ensure(before.previous()); + return before.previous(); + } + List tags = api.tags(); + ReleaseVersion version; + String commit; + if (mode.equals("new")) { + Pipeline.check( + pinned.equals(api.head()), "Main advanced since this dispatch; start a new release workflow"); + Pipeline.check( + tags.stream().noneMatch(t -> t.sha().equals(pinned)), + "This commit already has a release tag; use retry or rollback"); + version = ReleaseVersion.next( + tags.stream().map(ReleaseApi.Tag::version).toList(), increment); + commit = pinned; + } else { + version = ReleaseVersion.parse(retry); + var tag = tags.stream() + .filter(t -> t.version().equals(version)) + .findFirst() + .orElseThrow(() -> new IllegalArgumentException("Retry release tag does not exist")); + api.annotated(tag); + commit = tag.sha(); + } + Pipeline.check( + before.highest() == null || version.compareTo(before.highest()) >= (mode.equals("new") ? 1 : 0), + "A newer release has already succeeded; use explicit rollback for older versions"); + String build = api.verified(commit); + remote.call("stage", Files.readString(root.resolve("deploy/compose.yaml"))); + String prepared = remote.call("prepare", null, version.toString(), commit, prefix()); + return new ReleaseState.Deployment( + version, commit, digest(prepared, "BACKEND"), digest(prepared, "FRONTEND"), attempt, build); + } + + String prefix() { + var uri = URI.create(api.server); + String authority = uri.getRawAuthority(); + String registry = env.getOrDefault("DEPLOY_REGISTRY", "").strip(); + if (registry.isEmpty()) registry = authority; + int port = uri.getPort() >= 0 ? uri.getPort() : uri.getScheme().equals("https") ? 443 : 80; + Pipeline.check( + registry.equals(authority) || registry.equals("127.0.0.1:" + port), + "Deployment registry must be the existing Gitea registry or its NAS loopback alias"); + String owner = Setup.required(env, "CI_REPOSITORY").split("/")[0].toLowerCase(java.util.Locale.ROOT); + return registry + "/" + owner + "/agiledancers-"; + } + + void ensure(ReleaseState.Deployment release) throws Exception { + remote.call( + "ensure", + null, + release.directory(), + release.version().toString(), + release.sha(), + release.backend(), + release.frontend()); + } + + String digest(String response, String role) { + String value = response.lines() + .filter(line -> line.startsWith("S6_" + role + "=")) + .map(line -> line.substring(("S6_" + role + "=").length())) + .reduce((a, b) -> { + throw new IllegalArgumentException("Ambiguous prepared image digest"); + }) + .orElseThrow(() -> new IllegalArgumentException("Missing prepared image digest")); + Pipeline.check( + value.startsWith(prefix() + role.toLowerCase(java.util.Locale.ROOT) + "@sha256:"), + "Unexpected image repository"); + return value; + } + + String recover(ReleaseState before) throws Exception { + if (before.current() == null) return "first-release-failed-manual-recovery-required"; + try { + remote.call("apply", null, before.current().directory()); + } catch (Exception failure) { + diagnose(before.current()); + return "recovery-failed-manual-recovery-required"; + } + record("failed-previous-release-restored"); + remote.call("finish", null); + pending = false; + return "previous-release-restored"; + } + + void diagnose(ReleaseState.Deployment release) { + try { + Files.writeString( + artifacts.resolve(attempt + "-" + release.directory() + "-containers.log"), + remote.call("diagnose", null, release.directory())); + } catch (Exception ignored) { + System.err.println("Container diagnostics unavailable; use the NAS recovery runbook"); + } + } + + void record(String outcome) throws Exception { + var data = new LinkedHashMap(); + data.put("attempt", attempt); + data.put("mode", mode); + data.put("outcome", outcome); + data.put("time", Instant.now().toString()); + data.put("release", selected == null ? null : selected.json()); + String json = Json.write(data) + "\n"; + Files.writeString(artifacts.resolve(attempt + ".json"), json); + if (locked) remote.call("event", json); + } +} diff --git a/ci/ReleaseApi.java b/ci/ReleaseApi.java new file mode 100644 index 0000000..2971e18 --- /dev/null +++ b/ci/ReleaseApi.java @@ -0,0 +1,124 @@ +import java.net.URI; +import java.util.ArrayList; +import java.util.List; +import java.util.Map; +import java.util.Objects; + +/** Gitea 1.25 API adapter. Never include response bodies or tokens in errors. */ +class ReleaseApi { + record Tag(ReleaseVersion version, String sha, String object) {} + + final String server, token, base; + + ReleaseApi(Map env) { + server = Setup.required(env, "GITEA_SERVER_URL"); + var uri = URI.create(server); + Pipeline.check( + List.of("http", "https").contains(uri.getScheme()) + && uri.getUserInfo() == null + && uri.getQuery() == null + && uri.getFragment() == null + && (uri.getPath().isEmpty() || uri.getPath().equals("/")), + "Gitea must use a server-root URL"); + String repository = Setup.required(env, "CI_REPOSITORY"); + Pipeline.check(repository.matches("[a-zA-Z0-9_.-]+/[a-zA-Z0-9_.-]+"), "Invalid repository name"); + base = "/api/v1/repos/" + repository; + token = Setup.required(env, "RELEASE_TOKEN"); + } + + Object request(String method, String path, Object body) throws Exception { + return Setup.api(server, token, method, base + path, body); + } + + String head() throws Exception { + return sha( + Json.object(Json.object(request("GET", "/branches/main", null)).get("commit")) + .get("id")); + } + + static String sha(Object value) { + String result = Objects.toString(value, ""); + Pipeline.check(result.matches("[0-9a-f]{40}"), "Invalid release commit SHA"); + return result; + } + + List tags() throws Exception { + List tags = new ArrayList<>(); + var seen = new java.util.HashSet(); + for (int page = 1; ; page++) { + var entries = (List) request("GET", "/tags?limit=50&page=" + page, null); + if (entries.isEmpty()) return tags; + for (Object entry : entries) { + var tag = Json.object(entry); + String name = Objects.toString(tag.get("name"), ""); + Pipeline.check( + seen.add(name), + "Remote tag pagination repeated a tag; cannot establish a safe version baseline"); + // Unknown version-looking tags require explicit reconciliation, not a second baseline. + if (!name.matches("v?[0-9].*")) continue; + ReleaseVersion version = ReleaseVersion.parse(name); + String commit = sha(Json.object(tag.get("commit")).get("sha")); + String object = sha(tag.get("id")); + tags.add(new Tag(version, commit, object)); + } + } + } + + void annotated(Tag tag) throws Exception { + var data = Json.object(request("GET", "/git/tags/" + tag.object(), null)); + Pipeline.check( + tag.version().toString().equals(data.get("tag")) + && tag.sha().equals(Json.object(data.get("object")).get("sha")), + "Release tag must be annotated and point directly to its recorded commit"); + } + + void create(ReleaseVersion version, String commit, String run) throws Exception { + request( + "POST", + "/tags", + Map.of( + "tag_name", + version.toString(), + "target", + commit, + "message", + "Agiledancers " + version + "\nCommit: " + commit + "\nDeployment attempt: " + run)); + Tag tag = tags().stream() + .filter(t -> t.version().equals(version)) + .findFirst() + .orElseThrow(); + Pipeline.check(tag.sha().equals(commit), "Created release tag does not match selected commit"); + annotated(tag); + } + + String verified(String commit) throws Exception { + // A successful build.yaml main-push run includes verification AND successful image publication. + // Do not accept another workflow, a branch build, or merely an existing image/tag. + var seen = new java.util.HashSet(); + for (int page = 1; ; page++) { + var response = Json.object(request( + "GET", + "/actions/runs?head_sha=" + commit + "&event=push&branch=main&status=success&limit=50&page=" + page, + null)); + var runs = (List) response.get("workflow_runs"); + if (runs.isEmpty()) break; + boolean fresh = false; + for (Object entry : runs) { + var run = Json.object(entry); + fresh |= seen.add(Objects.toString(run.get("id"))); + if (commit.equals(run.get("head_sha")) + && "push".equals(run.get("event")) + && List.of("main", "refs/heads/main").contains(Objects.toString(run.get("head_branch"))) + && "completed".equals(run.get("status")) + && "success".equals(run.get("conclusion")) + && "build.yaml@refs/heads/main".equals(run.get("path"))) { + return Objects.toString(run.get("id")); + } + } + // Older Gitea installations may ignore some list filters/pagination. + // Validate each entry locally and stop if a page makes no progress. + if (!fresh) break; + } + throw new IllegalArgumentException("No successful build.yaml main-push run verifies and publishes this SHA"); + } +} diff --git a/ci/ReleaseRemote.java b/ci/ReleaseRemote.java new file mode 100644 index 0000000..e493d9c --- /dev/null +++ b/ci/ReleaseRemote.java @@ -0,0 +1,119 @@ +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.attribute.PosixFilePermissions; +import java.time.Duration; +import java.util.ArrayList; +import java.util.List; +import java.util.Map; + +/** Argument-safe SSH adapter; server-side mkdir lock is shared by every runner. */ +class ReleaseRemote implements AutoCloseable { + final Path root, credentials; + final Map env; + final Commands commands; + final String owner, directory, host, user, port, script; + + ReleaseRemote(Path root, Map env, Commands commands, String owner) throws Exception { + this.root = root; + this.env = env; + this.commands = commands; + this.owner = owner; + directory = Setup.required(env, "DEPLOY_DIRECTORY"); + host = Setup.required(env, "DEPLOY_HOST"); + user = Setup.required(env, "DEPLOY_USER"); + port = env.getOrDefault("DEPLOY_SSH_PORT", "22"); + Pipeline.check( + directory.matches("/[a-zA-Z0-9_./-]+") && !directory.contains("..") && !directory.equals("/"), + "Use an absolute dedicated deployment directory without spaces or parent traversal"); + Pipeline.check( + host.matches("[a-zA-Z0-9][a-zA-Z0-9.-]*") + && user.matches("[a-zA-Z0-9_][a-zA-Z0-9_-]*") + && port.matches("[0-9]{1,5}") + && Integer.parseInt(port) > 0 + && Integer.parseInt(port) <= 65535, + "Invalid deployment SSH host, user or port"); + script = Files.readString(root.resolve("deploy/remote.sh")); + credentials = Files.createTempDirectory("agiledancers-deploy-ssh-"); + try { + writePrivate("identity", Setup.required(env, "DEPLOY_SSH_KEY") + "\n"); + writePrivate("known_hosts", Setup.required(env, "DEPLOY_KNOWN_HOSTS") + "\n"); + } catch (Exception e) { + close(); + throw e; + } + } + + void writePrivate(String name, String value) throws Exception { + Path file = credentials.resolve(name); + Files.createFile(file, PosixFilePermissions.asFileAttribute(PosixFilePermissions.fromString("rw-------"))); + Files.writeString(file, value); + } + + static String quote(String value) { + return "'" + value.replace("'", "'\"'\"'") + "'"; + } + + String call(String action, String input, String... arguments) throws Exception { + int seconds = switch (action) { + case "apply" -> 300; + case "prepare", "ensure" -> 600; + default -> 60; + }; + var parameters = new ArrayList<>(List.of(action, directory, owner)); + parameters.addAll(List.of(arguments)); + return executeScript(script, input, seconds, parameters); + } + + String executeScript(String source, String input, int seconds, List parameters) throws Exception { + String remote = "timeout -k 5 " + seconds + " sh -c " + quote(source) + " --"; + for (String argument : parameters) remote += " " + quote(argument); + var args = new ArrayList<>(List.of( + "ssh", + "-F", + "/dev/null", + "-T", + "-o", + "BatchMode=yes", + "-o", + "StrictHostKeyChecking=yes", + "-o", + "IdentitiesOnly=yes", + "-o", + "IdentityAgent=none", + "-o", + "ForwardAgent=no", + "-o", + "ConnectTimeout=10", + "-o", + "ServerAliveInterval=10", + "-o", + "ServerAliveCountMax=3", + "-o", + "LogLevel=ERROR", + "-o", + "UserKnownHostsFile=" + credentials.resolve("known_hosts"), + "-o", + "GlobalKnownHostsFile=/dev/null", + "-i", + credentials.resolve("identity").toString(), + "-p", + port, + user + "@" + host, + remote)); + commands.renew(Duration.ofSeconds(seconds + 30)); + return commands.execute(args, root, env, input, false).checked(); + } + + void acquire() throws Exception { + call("lock", null); + } + + void release() throws Exception { + call("unlock", null); + } + + @Override + public void close() throws Exception { + Pipeline.deleteTree(credentials); + } +} diff --git a/ci/ReleaseSetup.java b/ci/ReleaseSetup.java new file mode 100644 index 0000000..412966d --- /dev/null +++ b/ci/ReleaseSetup.java @@ -0,0 +1,107 @@ +import java.net.URI; +import java.nio.file.Files; +import java.nio.file.Path; +import java.time.Duration; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +/** One-time provisioning; secrets stay in owner-only files or Gitea secret storage. */ +class ReleaseSetup { + static void configure(Path root, Map input) throws Exception { + var env = new HashMap<>(input); + env.putIfAbsent("GITEA_SERVER_URL", "http://192.168.2.28:3000"); + env.putIfAbsent("CI_REPOSITORY", "wanjakrah/agiledancers"); + env.putIfAbsent("DEPLOY_HOST", "192.168.2.28"); + env.putIfAbsent("DEPLOY_USER", "agiledancers-deploy"); + env.putIfAbsent("DEPLOY_SSH_PORT", "22"); + env.putIfAbsent("DEPLOY_DIRECTORY", "/volume1/docker/agiledancers-production"); + env.putIfAbsent("DEPLOY_BIND_IP", "192.168.2.28"); + env.putIfAbsent("DEPLOY_FRONTEND_PORT", "8088"); + env.putIfAbsent("DEPLOY_REGISTRY", "127.0.0.1:3000"); + Path config = Setup.home().resolve(".config/agiledancers-runner"); + env.putIfAbsent("GITEA_ADMIN_TOKEN_FILE", config.resolve("admin-token").toString()); + env.putIfAbsent("RELEASE_TOKEN_FILE", config.resolve("s6-release-token").toString()); + env.putIfAbsent("PULL_TOKEN_FILE", config.resolve("s6-pull-token").toString()); + env.putIfAbsent("DEPLOY_SSH_KEY_FILE", config.resolve("s6-deploy-key").toString()); + String admin = Setup.secret(env, "GITEA_ADMIN_TOKEN_FILE"); + String pull = Setup.secret(env, "PULL_TOKEN_FILE"); + env.put("RELEASE_TOKEN", Setup.secret(env, "RELEASE_TOKEN_FILE")); + env.put("DEPLOY_SSH_KEY", Setup.secret(env, "DEPLOY_SSH_KEY_FILE")); + String bind = env.get("DEPLOY_BIND_IP"), port = env.get("DEPLOY_FRONTEND_PORT"); + Pipeline.check( + bind.matches("[0-9.]+") + && port.matches("[0-9]{4,5}") + && Integer.parseInt(port) >= 1024 + && Integer.parseInt(port) <= 65535, + "Invalid private bind address or port"); + var commands = new Commands(Duration.ofMinutes(15)); + Path diagnostics = root.resolve("ci/artifacts/deployment"); + Files.createDirectories(diagnostics); + commands.log = diagnostics.resolve("setup.log"); + Files.writeString(commands.log, ""); + String host = env.get("DEPLOY_HOST"); + String lookup = env.get("DEPLOY_SSH_PORT").equals("22") ? host : "[" + host + "]:" + env.get("DEPLOY_SSH_PORT"); + env.put( + "DEPLOY_KNOWN_HOSTS", + commands.execute( + List.of( + "ssh-keygen", + "-F", + lookup, + "-f", + Setup.home().resolve(".ssh/known_hosts").toString()), + root, + env, + null, + false) + .checked()); + Pipeline.check(!env.get("DEPLOY_KNOWN_HOSTS").isBlank(), "A previously trusted NAS hostkey is required"); + var api = new ReleaseApi(env); + api.verified(api.head()); + var uri = URI.create(api.server); + String registry = env.get("DEPLOY_REGISTRY"); + int registryPort = uri.getPort() >= 0 ? uri.getPort() : uri.getScheme().equals("https") ? 443 : 80; + Pipeline.check( + registry.equals(uri.getRawAuthority()) || registry.equals("127.0.0.1:" + registryPort), + "Deployment registry must be the Gitea registry or its NAS loopback alias"); + String registryUser = + env.getOrDefault("PULL_USER", env.get("CI_REPOSITORY").split("/")[0]); + Pipeline.check(registryUser.matches("[a-zA-Z0-9_.-]+"), "Invalid registry user"); + try (var remote = new ReleaseRemote(root, env, commands, "setup")) { + System.out.println(remote.executeScript( + Files.readString(root.resolve("deploy/provision.sh")), + pull + "\n", + 300, + List.of(remote.directory, registryUser, registry, bind, port))); + remote.acquire(); + try { + remote.call("preflight", null); + } finally { + remote.release(); + } + } + // Provision credentials only after successful NAS login and release-read verification. + for (String secret : List.of("RELEASE_TOKEN", "DEPLOY_SSH_KEY", "DEPLOY_KNOWN_HOSTS")) + Setup.api( + api.server, admin, "PUT", api.base + "/actions/secrets/" + secret, Map.of("data", env.get(secret))); + for (String variable : + List.of("DEPLOY_HOST", "DEPLOY_USER", "DEPLOY_SSH_PORT", "DEPLOY_DIRECTORY", "DEPLOY_REGISTRY")) + variable(api, admin, variable, env.get(variable)); + System.out.println( + "Production configuration and Gitea deployment secrets/variables installed; no release was created."); + } + + static void variable(ReleaseApi api, String admin, String name, String value) throws Exception { + String path = api.base + "/actions/variables/" + name; + boolean exists; + try { + Setup.api(api.server, admin, "GET", path, null); + exists = true; + } catch (Setup.ApiFailure error) { + if (error.status != 404) throw error; + exists = false; + } + Setup.api(api.server, admin, exists ? "PUT" : "POST", path, Map.of("value", value)); + } +} diff --git a/ci/ReleaseShellTests.java b/ci/ReleaseShellTests.java new file mode 100644 index 0000000..25cfbe4 --- /dev/null +++ b/ci/ReleaseShellTests.java @@ -0,0 +1,197 @@ +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.attribute.PosixFilePermissions; +import java.time.Duration; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +/** Runs the actual NAS script with disposable command adapters, no Docker daemon. */ +class ReleaseShellTests { + static void run() throws Exception { + ReleaseTests.test("actual NAS script prepares digest images and checks the published endpoint", () -> { + try (var f = new Fixture()) { + f.prepare(); + f.call("pending", "{}", "one").checked(); + CiTests.equal("S6_HEALTHY", f.call("apply", null, "one", "one").checked()); + String calls = Files.readString(f.directory.resolve("calls")); + CiTests.equal(true, calls.contains("http://backend:8080/q/health/ready")); + CiTests.equal(true, calls.contains("http://127.0.0.1:8088/api/dance-schools")); + CiTests.equal(false, calls.contains(" down")); + CiTests.equal(false, calls.contains("prune")); + } + }); + ReleaseTests.test("actual NAS script rejects wrong image revision before replacement", () -> { + try (var f = new Fixture()) { + f.env.put("WRONG_REVISION", "1"); + CiTests.failure(f::prepare); + CiTests.equal(false, Files.exists(f.directory.resolve("live.env"))); + } + }); + ReleaseTests.test("actual NAS script rejects unavailable image before replacement", () -> { + try (var f = new Fixture()) { + f.env.put("FAIL_PULL", "1"); + CiTests.failure(f::prepare); + CiTests.equal(false, Files.exists(f.directory.resolve("live.env"))); + } + }); + ReleaseTests.test("actual NAS script rejects a changed retained release record", () -> { + try (var f = new Fixture()) { + f.prepare(); + CiTests.failure(() -> f.call( + "ensure", null, "one", "one", "v9.0.0", ReleaseTests.SHA, f.backend(), f.frontend()) + .checked()); + f.call("ensure", null, "one", "one", "v1.0.0", ReleaseTests.SHA, f.backend(), f.frontend()) + .checked(); + } + }); + ReleaseTests.test("actual NAS replacement timeout is bounded and leaves recovery journal", () -> { + try (var f = new Fixture()) { + f.prepare(); + f.call("pending", "{}", "one").checked(); + f.env.put("FAIL_HEALTH", "1"); + CiTests.equal(1, f.call("apply", null, "one", "one", "1").code()); + CiTests.equal(true, Files.exists(f.directory.resolve("pending.json"))); + // Termination releases only the operation fence, not the deployment lock. + CiTests.equal(false, Files.exists(f.directory.resolve("deploy.lock/operation"))); + CiTests.equal(true, Files.exists(f.directory.resolve("deploy.lock/owner"))); + f.env.remove("FAIL_HEALTH"); + f.call("apply", null, "one", "one").checked(); + } + }); + ReleaseTests.test("actual NAS script fences overlapping remote replacements", () -> { + try (var f = new Fixture()) { + f.prepare(); + Files.createDirectory(f.directory.resolve("deploy.lock/operation")); + CiTests.failure(() -> f.call("apply", null, "one", "one").checked()); + CiTests.equal(false, Files.exists(f.directory.resolve("live.env"))); + } + }); + ReleaseTests.test("actual NAS script stores state atomically and archives resolved journal", () -> { + try (var f = new Fixture()) { + f.call("lock", null, "one").checked(); + f.call("pending", "{\"attempt\":\"one\"}", "one").checked(); + f.call("commit", "{\"current\":\"one\"}", "one").checked(); + CiTests.equal( + "{\"current\":\"one\"}", f.call("state", null, "one").checked()); + f.call("event", "{\"outcome\":\"success\"}", "one").checked(); + f.call("finish", null, "one").checked(); + f.call("unlock", null, "one").checked(); + CiTests.equal(true, Files.exists(f.directory.resolve("history/one.json"))); + CiTests.equal(false, Files.exists(f.directory.resolve("pending.json"))); + } + }); + } + + static class Fixture implements AutoCloseable { + final Path directory = Files.createTempDirectory("nas-script-test-"); + final Map env = new HashMap<>(System.getenv()); + final String script = Files.readString(Path.of("deploy/remote.sh")); + final Commands commands = new Commands(Duration.ofMinutes(1)); + + Fixture() throws Exception { + Files.createDirectory(directory.resolve("bin")); + executable("docker", DOCKER); + executable( + "curl", + "#!/bin/sh\nprintf 'curl %s\\n' \"$*\" >> \"$FIXTURE/calls\"\n[ \"${FAIL_HEALTH:-0}\" = 0 ]\n"); + env.put("PATH", directory.resolve("bin") + ":" + env.get("PATH")); + env.put("FIXTURE", directory.toString()); + Files.createDirectory(directory.resolve("docker-auth")); + Files.writeString(directory.resolve("docker-auth/config.json"), "{}"); + Files.writeString( + directory.resolve("production.env"), + "BIND_IP=127.0.0.1\nFRONTEND_PORT=8088\nDOCKER_SUDO=0\nDOCKER_BIN=" + + directory.resolve("bin/docker") + "\nDOCKER_CONFIG=" + directory.resolve("docker-auth") + + "\n"); + } + + void executable(String name, String body) throws Exception { + Path path = directory.resolve("bin").resolve(name); + Files.writeString(path, body); + Files.setPosixFilePermissions(path, PosixFilePermissions.fromString("rwx------")); + } + + Commands.Result call(String action, String input, String owner, String... args) throws Exception { + var command = new ArrayList<>( + List.of("timeout", "-k", "1", "10", "sh", "-c", script, "--", action, directory.toString(), owner)); + command.addAll(List.of(args)); + commands.renew(Duration.ofSeconds(15)); + return commands.execute(command, directory, env, input, false); + } + + String backend() { + return "nas:3000/owner/agiledancers-backend@sha256:" + "1".repeat(64); + } + + String frontend() { + return "nas:3000/owner/agiledancers-frontend@sha256:" + "2".repeat(64); + } + + void prepare() throws Exception { + call("lock", null, "one").checked(); + call("preflight", null, "one").checked(); + call("stage", Files.readString(Path.of("deploy/compose.yaml")), "one") + .checked(); + String result = call("prepare", null, "one", "v1.0.0", ReleaseTests.SHA, "nas:3000/owner/agiledancers-") + .checked(); + CiTests.equal(true, result.contains("S6_BACKEND=" + backend())); + CiTests.equal(true, result.contains("S6_FRONTEND=" + frontend())); + } + + @Override + public void close() throws Exception { + Pipeline.deleteTree(directory); + } + } + + static final String DOCKER = """ + #!/bin/sh + set -eu + printf 'docker %s\n' "$*" >> "$FIXTURE/calls" + [ "$1" != --config ] || shift 2 + case "$1" in + info) echo linux/x86_64;; + pull) [ "${FAIL_PULL:-0}" = 0 ];; + image) + format=$4; image=$5 + case "$image" in *backend*) role=backend; digit=1;; *) role=frontend; digit=2;; esac + case "$format" in + *revision*) + if [ "${WRONG_REVISION:-0}" = 1 ]; then echo wrong; else printf '%040d\n' 0 | tr 0 a; fi;; + *Architecture*) echo linux/amd64;; + *RepoDigests*) + printf '%s@sha256:' "${image%:*}"; printf '%064d\n' 0 | tr 0 "$digit";; + *Id*) echo "sha256:$role";; + *) exit 1;; + esac;; + compose) + shift + [ "$1" != version ] || exit 0 + shift 4 + [ "$1" = --env-file ] || exit 1 + file=$2 + shift 4 + case "$1" in + config) test -f "$file";; + up) cp "$file" "$FIXTURE/live.env";; + ps) eval 'role=${'"$#"'}'; echo "$role";; + logs) echo 'sample container diagnostics';; + *) exit 1;; + esac;; + inspect) + . "$FIXTURE/live.env" + format=$3; id=$4 + case "$format" in + *State.Running*) echo true;; + *Image*) echo "sha256:$id";; + *revision*) echo "$RELEASE_SHA";; + *release*) echo "$RELEASE_VERSION";; + *) exit 1;; + esac;; + exec) [ "${FAIL_HEALTH:-0}" = 0 ];; + *) exit 1;; + esac + """; +} diff --git a/ci/ReleaseState.java b/ci/ReleaseState.java new file mode 100644 index 0000000..3b526e2 --- /dev/null +++ b/ci/ReleaseState.java @@ -0,0 +1,80 @@ +import java.util.LinkedHashMap; +import java.util.Map; +import java.util.Objects; + +/** Successful deployment state is separate from immutable allocated version tags. */ +record ReleaseState(ReleaseState.Deployment current, ReleaseState.Deployment previous, ReleaseVersion highest) { + record Deployment( + ReleaseVersion version, String sha, String backend, String frontend, String directory, String build) { + Deployment { + ReleaseApi.sha(sha); + Pipeline.check(directory.matches("[a-zA-Z0-9-]+"), "Invalid retained release directory"); + for (String image : new String[] {backend, frontend}) + Pipeline.check(image.matches("[a-zA-Z0-9.:/_-]+@sha256:[0-9a-f]{64}"), "Invalid retained image digest"); + } + + Map json() { + return Map.of( + "version", + version.toString(), + "sha", + sha, + "backend", + backend, + "frontend", + frontend, + "directory", + directory, + "build", + build); + } + + static Deployment parse(Object value) { + if (value == null) return null; + var data = Json.object(value); + return new Deployment( + ReleaseVersion.parse(text(data, "version")), + text(data, "sha"), + text(data, "backend"), + text(data, "frontend"), + text(data, "directory"), + text(data, "build")); + } + } + + static String text(Map data, String key) { + return Objects.toString(data.get(key), ""); + } + + static ReleaseState parse(String value) { + var data = Json.object(Json.parse(value)); + if (data.isEmpty()) return new ReleaseState(null, null, null); + Pipeline.check("1".equals(text(data, "schema")), "Unsupported deployment state schema"); + Deployment current = Deployment.parse(data.get("current")); + Deployment previous = Deployment.parse(data.get("previous")); + ReleaseVersion highest = ReleaseVersion.parse(text(data, "highest")); + Pipeline.check( + current != null + && highest.compareTo(current.version()) >= 0 + && (previous == null || highest.compareTo(previous.version()) >= 0), + "Inconsistent successful release state"); + return new ReleaseState(current, previous, highest); + } + + String json() { + var data = new LinkedHashMap(); + data.put("schema", "1"); + data.put("current", current == null ? null : current.json()); + data.put("previous", previous == null ? null : previous.json()); + data.put("highest", highest == null ? null : highest.toString()); + return Json.write(data) + "\n"; + } + + ReleaseState success(Deployment selected) { + ReleaseVersion watermark = + highest == null || selected.version().compareTo(highest) > 0 ? selected.version() : highest; + // A redeploy of the same release must not discard the distinct rollback target. + Deployment fallback = current != null && current.version().equals(selected.version()) ? previous : current; + return new ReleaseState(selected, fallback, watermark); + } +} diff --git a/ci/ReleaseTests.java b/ci/ReleaseTests.java new file mode 100644 index 0000000..bc0e907 --- /dev/null +++ b/ci/ReleaseTests.java @@ -0,0 +1,549 @@ +import java.nio.file.Files; +import java.nio.file.Path; +import java.time.Duration; +import java.util.ArrayList; +import java.util.HashMap; +import java.util.List; +import java.util.Map; + +/** Release state-machine and real POSIX remote-script tests, without server writes. */ +class ReleaseTests { + static final String SHA = "a".repeat(40), OLD = "b".repeat(40); + static int count; + + static void test(String name, CiTests.Test test) throws Exception { + test.run(); + count++; + System.out.println("PASS release: " + name); + } + + static Map environment() { + var env = new HashMap<>(System.getenv()); + env.putAll(Map.of( + "CI_REF", + "refs/heads/main", + "CI_EVENT", + "workflow_dispatch", + "CI_RUN_ID", + "123", + "CI_COMMIT", + SHA, + "RELEASE_MODE", + "new", + "RELEASE_INCREMENT", + "patch", + "RELEASE_VERSION", + "", + "GITEA_SERVER_URL", + "http://nas:3000", + "CI_REPOSITORY", + "owner/agiledancers", + "RELEASE_TOKEN", + "test")); + env.putAll(Map.of( + "DEPLOY_HOST", + "nas", + "DEPLOY_USER", + "deploy", + "DEPLOY_DIRECTORY", + "/tmp/production", + "DEPLOY_SSH_KEY", + "test-private-key", + "DEPLOY_KNOWN_HOSTS", + "test-pinned-host")); + return env; + } + + static ReleaseState.Deployment deployment(String version, String sha, String directory) { + return new ReleaseState.Deployment( + ReleaseVersion.parse(version), + sha, + "nas:3000/owner/agiledancers-backend@sha256:" + "1".repeat(64), + "nas:3000/owner/agiledancers-frontend@sha256:" + "2".repeat(64), + directory, + "10"); + } + + static ReleaseState prior() { + return new ReleaseState( + deployment("v1.0.1", OLD, "previous-current"), + deployment("v1.0.0", OLD, "previous-old"), + ReleaseVersion.parse("v1.0.1")); + } + + static void run() throws Exception { + count = 0; + test("first release and all increments reset the correct components", () -> { + CiTests.equal("v1.0.0", ReleaseVersion.next(List.of(), "major").toString()); + var versions = List.of(ReleaseVersion.parse("v1.2.3"), ReleaseVersion.parse("v1.1.9")); + CiTests.equal("v1.2.4", ReleaseVersion.next(versions, "patch").toString()); + CiTests.equal("v1.3.0", ReleaseVersion.next(versions, "minor").toString()); + CiTests.equal("v2.0.0", ReleaseVersion.next(versions, "major").toString()); + CiTests.equal( + "v1.0.100000000000000000000", + ReleaseVersion.next(List.of(ReleaseVersion.parse("v1.0.99999999999999999999")), "patch") + .toString()); + for (String invalid : List.of("1.0.0", "v01.0.0", "v1.0.0-rc1", "v1.0.0+build", "v1.0", "v-1.0.0")) + CiTests.failure(() -> ReleaseVersion.parse(invalid)); + }); + test("branch, nonmanual, malformed input and conflicting new-release inputs are rejected", () -> { + var env = environment(); + Release.validate(env, "deploy"); + env.put("CI_REF", "refs/heads/feature/S6"); + CiTests.failure(() -> Release.validate(env, "deploy")); + env.put("CI_REF", "refs/heads/main"); + env.put("CI_EVENT", "push"); + CiTests.failure(() -> Release.validate(env, "rollback")); + env.put("CI_EVENT", "workflow_dispatch"); + env.put("RELEASE_VERSION", "v1.0.0"); + CiTests.failure(() -> Release.validate(env, "deploy")); + env.put("RELEASE_MODE", "retry"); + Release.validate(env, "deploy"); + env.put("RELEASE_VERSION", "bad"); + CiTests.failure(() -> Release.validate(env, "deploy")); + }); + test("successful release stages images before tag, checks main again and commits state", () -> { + try (var f = new Fixture()) { + f.release.run(); + CiTests.equal("v1.0.0", f.remote.state.current().version().toString()); + CiTests.equal(false, f.remote.locked); + CiTests.equal(false, f.remote.pending); + CiTests.equal(true, f.events.indexOf("prepare") < f.events.indexOf("tag")); + CiTests.equal(true, f.events.indexOf("tag") < f.events.lastIndexOf("head")); + CiTests.equal(true, f.events.lastIndexOf("head") < f.events.indexOf("apply:attempt")); + } + }); + test("unverified commit never creates a tag or replaces production", () -> { + try (var f = new Fixture()) { + f.api.verified = false; + CiTests.failure(f.release::run); + CiTests.equal(false, f.events.contains("tag")); + CiTests.equal(false, f.events.contains("apply:attempt")); + CiTests.equal(false, f.remote.locked); + } + }); + test("missing images prevent tag allocation and replacement", () -> { + try (var f = new Fixture()) { + f.remote.failAction = "prepare"; + CiTests.failure(f.release::run); + CiTests.equal(false, f.events.contains("tag")); + CiTests.equal(false, f.remote.pending); + } + }); + test("a tagged commit cannot receive another new version", () -> { + try (var f = new Fixture()) { + f.api.tags.add(new ReleaseApi.Tag(ReleaseVersion.parse("v1.0.0"), SHA, OLD)); + CiTests.failure(f.release::run); + CiTests.equal(false, f.events.contains("prepare")); + } + }); + test("stale dispatch is rejected before pulling or tagging", () -> { + try (var f = new Fixture()) { + f.api.head = OLD; + CiTests.failure(f.release::run); + CiTests.equal(false, f.events.contains("prepare")); + } + }); + test("main advancing after tag creation preserves tag without changing production", () -> { + try (var f = new Fixture()) { + f.api.advanceAfterTag = true; + CiTests.failure(f.release::run); + CiTests.equal(1, f.api.tags.size()); + CiTests.equal(false, f.remote.pending); + CiTests.equal(false, f.events.contains("apply:attempt")); + } + }); + test("unavailable recovery images block replacement", () -> { + try (var f = new Fixture()) { + f.withPrior(); + f.remote.failAction = "ensure"; + CiTests.failure(f.release::run); + CiTests.equal(false, f.events.contains("tag")); + CiTests.equal(false, f.remote.pending); + } + }); + test("failed candidate restores prior version and still fails the workflow", () -> { + try (var f = new Fixture()) { + f.withPrior(); + f.remote.failedDirectories.add("attempt"); + CiTests.failure(f.release::run); + CiTests.equal( + List.of("apply:attempt", "apply:previous-current"), + f.events.stream().filter(e -> e.startsWith("apply:")).toList()); + CiTests.equal(prior(), f.remote.state); + CiTests.equal(false, f.remote.pending); + CiTests.equal(false, f.remote.locked); + CiTests.equal(2, f.api.tags.size()); + } + }); + test("failed first release retains journal and lock for manual recovery", () -> { + try (var f = new Fixture()) { + f.remote.failedDirectories.add("attempt"); + CiTests.failure(f.release::run); + CiTests.equal(true, f.remote.pending); + CiTests.equal(true, f.remote.locked); + CiTests.equal(null, f.remote.state.current()); + } + }); + test("failed automatic recovery retains journal and lock without retry loop", () -> { + try (var f = new Fixture()) { + f.withPrior(); + f.remote.failedDirectories.addAll(List.of("attempt", "previous-current")); + CiTests.failure(f.release::run); + CiTests.equal( + 2L, + f.events.stream().filter(e -> e.startsWith("apply:")).count()); + CiTests.equal(true, f.remote.pending); + CiTests.equal(true, f.remote.locked); + } + }); + test("ambiguous state commit never launches speculative rollback", () -> { + try (var f = new Fixture()) { + f.withPrior(); + f.remote.failAction = "commit"; + CiTests.failure(f.release::run); + CiTests.equal( + 1L, + f.events.stream().filter(e -> e.startsWith("apply:")).count()); + CiTests.equal(true, f.remote.pending); + CiTests.equal(true, f.remote.locked); + } + }); + test("retry keeps existing version and SHA after main advances", () -> { + try (var f = new Fixture("retry")) { + f.api.head = OLD; + f.release.run(); + CiTests.equal(SHA, f.remote.state.current().sha()); + CiTests.equal("v1.0.0", f.remote.state.current().version().toString()); + CiTests.equal(false, f.events.contains("tag")); + CiTests.equal(true, f.events.contains("annotated")); + } + }); + test("retry cannot cross the successful-version high watermark after rollback", () -> { + try (var f = new Fixture("retry")) { + f.remote.state = prior().success(prior().previous()); + CiTests.failure(f.release::run); + CiTests.equal(false, f.events.contains("apply:attempt")); + } + }); + test("manual rollback preserves version baseline and creates no tag", () -> { + try (var f = new Fixture("rollback")) { + f.remote.state = prior(); + f.release.run(); + CiTests.equal("v1.0.0", f.remote.state.current().version().toString()); + CiTests.equal("v1.0.1", f.remote.state.previous().version().toString()); + CiTests.equal("v1.0.1", f.remote.state.highest().toString()); + CiTests.equal(false, f.events.contains("tag")); + } + }); + test("redeploying current version preserves a distinct rollback target", () -> { + var state = prior(); + CiTests.equal(state.previous(), state.success(state.current()).previous()); + CiTests.equal(state, ReleaseState.parse(state.json())); + }); + test("NAS lock contention cannot allocate tags or replace production", () -> { + try (var f = new Fixture()) { + f.remote.locked = true; + CiTests.failure(f.release::run); + CiTests.equal(false, f.events.contains("tag")); + CiTests.equal(true, f.remote.locked); + } + }); + apiTests(); + shellTests(); + ReleaseShellTests.run(); + System.out.println(count + " release behavior tests passed"); + } + + static class FakeApi extends ReleaseApi { + final List events; + final List tags = new ArrayList<>(); + String head = SHA; + boolean verified = true, advanceAfterTag; + + FakeApi(Map env, List events) { + super(env); + this.events = events; + } + + @Override + String head() { + events.add("head"); + return head; + } + + @Override + List tags() { + return List.copyOf(tags); + } + + @Override + String verified(String sha) { + Pipeline.check(verified, "Unverified test commit"); + events.add("verified"); + return "10"; + } + + @Override + void annotated(Tag tag) { + events.add("annotated"); + } + + @Override + void create(ReleaseVersion version, String sha, String attempt) { + events.add("tag"); + tags.add(new Tag(version, sha, OLD)); + if (advanceAfterTag) head = OLD; + } + } + + static class FakeRemote extends ReleaseRemote { + final List events; + final List failedDirectories = new ArrayList<>(); + ReleaseState state = new ReleaseState(null, null, null); + boolean locked, pending; + String failAction = ""; + + FakeRemote(Path root, Map env, List events) throws Exception { + super(root, env, new Commands(Duration.ofMinutes(1)), "attempt"); + this.events = events; + } + + @Override + String call(String action, String input, String... args) { + events.add(action.equals("apply") ? action + ":" + args[0] : action); + Pipeline.check(!action.equals(failAction), "Injected remote failure"); + switch (action) { + case "lock" -> { + Pipeline.check(!locked && !pending, "Locked"); + locked = true; + } + case "unlock" -> { + Pipeline.check(!pending, "Pending"); + locked = false; + } + case "state" -> { + return state.current() == null ? "{}" : state.json(); + } + case "pending" -> pending = true; + case "finish" -> pending = false; + case "commit" -> state = ReleaseState.parse(input); + case "prepare" -> { + return "S6_BACKEND=" + deployment("v1.0.0", SHA, "attempt").backend() + "\nS6_FRONTEND=" + + deployment("v1.0.0", SHA, "attempt").frontend(); + } + case "apply" -> Pipeline.check(!failedDirectories.contains(args[0]), "Injected unhealthy release"); + default -> {} + } + return ""; + } + } + + static class Fixture implements AutoCloseable { + final Path root = Files.createTempDirectory("release-test-"); + final Map env = environment(); + final List events = new ArrayList<>(); + final FakeApi api; + final FakeRemote remote; + final Release release; + + Fixture() throws Exception { + this("new"); + } + + Fixture(String mode) throws Exception { + Files.createDirectories(root.resolve("deploy")); + Files.writeString(root.resolve("deploy/remote.sh"), ""); + Files.writeString(root.resolve("deploy/compose.yaml"), "services: {}\n"); + env.put("RELEASE_MODE", mode); + if (mode.equals("retry")) env.put("RELEASE_VERSION", "v1.0.0"); + api = new FakeApi(env, events); + if (mode.equals("retry")) api.tags.add(new ReleaseApi.Tag(ReleaseVersion.parse("v1.0.0"), SHA, OLD)); + remote = new FakeRemote(root, env, events); + release = new Release(root, env, api, remote, mode.equals("rollback") ? "rollback" : "deploy", "attempt"); + } + + void withPrior() { + remote.state = prior(); + api.tags.add(new ReleaseApi.Tag(ReleaseVersion.parse("v1.0.1"), OLD, OLD)); + } + + @Override + public void close() throws Exception { + remote.close(); + Pipeline.deleteTree(root); + } + } + + static void apiTests() throws Exception { + test("NAS registry alias keeps the same repository and rejects unrelated registries", () -> { + try (var f = new Fixture()) { + f.env.put("DEPLOY_REGISTRY", "127.0.0.1:3000"); + CiTests.equal("127.0.0.1:3000/owner/agiledancers-", f.release.prefix()); + f.env.put("DEPLOY_REGISTRY", "other-registry:3000"); + CiTests.failure(f.release::prefix); + } + }); + test("repeating remote tag pages fail closed instead of allocating from incomplete history", () -> { + var api = new ReleaseApi(environment()) { + @Override + Object request(String method, String path, Object body) { + return List.of(Map.of("name", "v1.0.0", "id", OLD, "commit", Map.of("sha", SHA))); + } + }; + CiTests.failure(api::tags); + }); + test("annotated release tag must identify the recorded version and exact SHA", () -> { + var tag = new ReleaseApi.Tag(ReleaseVersion.parse("v1.0.0"), SHA, OLD); + var api = new ReleaseApi(environment()) { + @Override + Object request(String method, String path, Object body) { + return Map.of("tag", "v1.0.0", "object", Map.of("sha", SHA)); + } + }; + api.annotated(tag); + var wrong = new ReleaseApi(environment()) { + @Override + Object request(String method, String path, Object body) { + return Map.of("tag", "v1.0.0", "object", Map.of("sha", OLD)); + } + }; + CiTests.failure(() -> wrong.annotated(tag)); + }); + test("Gitea API eligibility validates SHA, event, branch, workflow path and successful conclusion", () -> { + var valid = Map.of( + "id", + 1, + "head_sha", + SHA, + "event", + "push", + "head_branch", + "main", + "path", + "build.yaml@refs/heads/main", + "status", + "completed", + "conclusion", + "success"); + for (String field : List.of("head_sha", "event", "head_branch", "path", "status", "conclusion")) { + var invalid = new HashMap<>(valid); + invalid.put(field, "wrong"); + var api = new ReleaseApi(environment()) { + @Override + Object request(String method, String path, Object body) { + return Map.of("workflow_runs", List.of(invalid)); + } + }; + CiTests.failure(() -> api.verified(SHA)); + } + var api = new ReleaseApi(environment()) { + @Override + Object request(String method, String path, Object body) { + return Map.of("workflow_runs", List.of(valid)); + } + }; + CiTests.equal("1", api.verified(SHA)); + }); + test("remote tag pagination includes failed-release baseline and rejects ambiguous tag formats", () -> { + var api = new ReleaseApi(environment()) { + @Override + Object request(String method, String path, Object body) { + return path.endsWith("page=1") + ? List.of(Map.of("name", "v1.2.3", "id", OLD, "commit", Map.of("sha", SHA))) + : List.of(); + } + }; + CiTests.equal( + "v1.2.4", + ReleaseVersion.next( + api.tags().stream() + .map(ReleaseApi.Tag::version) + .toList(), + "patch") + .toString()); + var invalid = new ReleaseApi(environment()) { + @Override + Object request(String method, String path, Object body) { + return List.of(Map.of("name", "1.0", "id", OLD, "commit", Map.of("sha", SHA))); + } + }; + CiTests.failure(invalid::tags); + }); + } + + static void shellTests() throws Exception { + test("real remote shell atomically excludes a second owner and retains unresolved journal", () -> { + Path directory = Files.createTempDirectory("release-shell-test-"); + try { + String script = Files.readString(Path.of("deploy/remote.sh")); + var commands = new Commands(Duration.ofSeconds(30)); + var env = environment(); + var first = commands.execute( + List.of("sh", "-c", script, "--", "lock", directory.toString(), "one"), + directory, + env, + null, + false); + first.checked(); + CiTests.equal( + 1, + commands.execute( + List.of("sh", "-c", script, "--", "lock", directory.toString(), "two"), + directory, + env, + null, + false) + .code()); + commands.execute( + List.of("sh", "-c", script, "--", "pending", directory.toString(), "one"), + directory, + env, + "{\"candidate\":\"test\"}", + false) + .checked(); + CiTests.equal( + 1, + commands.execute( + List.of("sh", "-c", script, "--", "unlock", directory.toString(), "one"), + directory, + env, + null, + false) + .code()); + CiTests.equal(true, Files.exists(directory.resolve("pending.json"))); + commands.execute( + List.of("sh", "-c", script, "--", "finish", directory.toString(), "one"), + directory, + env, + null, + false) + .checked(); + commands.execute( + List.of("sh", "-c", script, "--", "unlock", directory.toString(), "one"), + directory, + env, + null, + false) + .checked(); + CiTests.equal(false, Files.exists(directory.resolve("deploy.lock"))); + CiTests.equal(true, Files.exists(directory.resolve("history/one.json"))); + } finally { + Pipeline.deleteTree(directory); + } + }); + test("SSH shell quoting preserves metacharacters as data", () -> { + String value = "quote' dollar$ backtick` newline\n"; + var commands = new Commands(Duration.ofSeconds(5)); + CiTests.equal( + value, + commands.execute( + List.of("sh", "-c", "printf '%s' " + ReleaseRemote.quote(value)), + Path.of("."), + environment(), + null, + false) + .output()); + }); + } +} diff --git a/ci/ReleaseVersion.java b/ci/ReleaseVersion.java new file mode 100644 index 0000000..9cc9e11 --- /dev/null +++ b/ci/ReleaseVersion.java @@ -0,0 +1,39 @@ +import java.math.BigInteger; +import java.util.List; +import java.util.regex.Pattern; + +/** Strict stable release versions; arbitrary-size integers avoid overflow/reuse. */ +record ReleaseVersion(BigInteger major, BigInteger minor, BigInteger patch) implements Comparable { + private static final Pattern FORMAT = Pattern.compile("v(0|[1-9][0-9]*)\\.(0|[1-9][0-9]*)\\.(0|[1-9][0-9]*)"); + + static ReleaseVersion parse(String value) { + var match = FORMAT.matcher(value); + Pipeline.check(match.matches(), "Expected release version vMAJOR.MINOR.PATCH"); + return new ReleaseVersion( + new BigInteger(match.group(1)), new BigInteger(match.group(2)), new BigInteger(match.group(3))); + } + + static ReleaseVersion next(List versions, String increment) { + Pipeline.check(List.of("patch", "minor", "major").contains(increment), "Choose patch, minor or major"); + var highest = versions.stream().max(ReleaseVersion::compareTo); + if (highest.isEmpty()) return parse("v1.0.0"); + var current = highest.get(); + return switch (increment) { + case "major" -> new ReleaseVersion(current.major.add(BigInteger.ONE), BigInteger.ZERO, BigInteger.ZERO); + case "minor" -> new ReleaseVersion(current.major, current.minor.add(BigInteger.ONE), BigInteger.ZERO); + default -> new ReleaseVersion(current.major, current.minor, current.patch.add(BigInteger.ONE)); + }; + } + + @Override + public int compareTo(ReleaseVersion other) { + int value = major.compareTo(other.major); + if (value == 0) value = minor.compareTo(other.minor); + return value == 0 ? patch.compareTo(other.patch) : value; + } + + @Override + public String toString() { + return "v" + major + "." + minor + "." + patch; + } +} diff --git a/deploy/compose.yaml b/deploy/compose.yaml new file mode 100644 index 0000000..6a58bd6 --- /dev/null +++ b/deploy/compose.yaml @@ -0,0 +1,34 @@ +# Images are resolved to registry digests before replacement. Never use :main here. +services: + backend: + image: ${BACKEND_IMAGE:?Required verified backend digest} + platform: linux/amd64 + restart: unless-stopped + stop_grace_period: 20s + labels: + nl.agilecc.release: ${RELEASE_VERSION:?Required release version} + nl.agilecc.revision: ${RELEASE_SHA:?Required full commit SHA} + logging: + driver: json-file + options: + max-size: '10m' + max-file: '3' + frontend: + image: ${FRONTEND_IMAGE:?Required verified frontend digest} + platform: linux/amd64 + restart: unless-stopped + stop_grace_period: 20s + depends_on: + - backend + environment: + BACKEND_UPSTREAM: backend:8080 + ports: + - '${BIND_IP:?Required private NAS IP}:${FRONTEND_PORT:?Required port}:8080' + labels: + nl.agilecc.release: ${RELEASE_VERSION:?Required release version} + nl.agilecc.revision: ${RELEASE_SHA:?Required full commit SHA} + logging: + driver: json-file + options: + max-size: '10m' + max-file: '3' diff --git a/deploy/enable-docker.sh b/deploy/enable-docker.sh new file mode 100644 index 0000000..4ef58b9 --- /dev/null +++ b/deploy/enable-docker.sh @@ -0,0 +1,28 @@ +#!/bin/sh +# One-time DSM administrator action, run with sudo; does not restart any service. +set -eu +test "$(id -u)" = 0 || { echo 'Run this setup with sudo' >&2; exit 1; } +account=agiledancers-deploy +docker=/usr/local/bin/docker +target=/etc/sudoers.d/agiledancers-deploy +rule="$account ALL=(root) NOPASSWD: $docker" +test -x "$docker" && id "$account" >/dev/null +test -d /etc/sudoers.d || { echo 'Missing sudoers.d; no changes made' >&2; exit 1; } +if [ -e "$target" ]; then + test "$(cat "$target")" = "$rule" || { echo 'Existing sudoers entry differs; no changes made' >&2; exit 1; } +else + umask 077 + temporary=$(mktemp /etc/sudoers.d/agiledancers-deploy.XXXXXX) + trap 'rm -f "$temporary"' EXIT + printf '%s\n' "$rule" > "$temporary" + chmod 440 "$temporary" + if command -v visudo >/dev/null 2>&1; then visudo -cf "$temporary"; fi + # A dot in the temporary name keeps sudo from loading the uncommitted file. + mv "$temporary" "$target" + if ! sudo -n -u "$account" sudo -n "$docker" info --format '{{.OSType}}/{{.Architecture}}'; then + rm "$target" + echo 'Docker permission check failed; new sudoers entry removed' >&2 + exit 1 + fi +fi +echo 'Dedicated Docker sudo permission is configured; no services were restarted.' diff --git a/deploy/production.env.example b/deploy/production.env.example new file mode 100644 index 0000000..abc8002 --- /dev/null +++ b/deploy/production.env.example @@ -0,0 +1,9 @@ +# Copy to /production.env on the NAS, mode 600. +# This is shell syntax: do not add spaces around '=' or put secrets here. +BIND_IP=192.168.2.28 +FRONTEND_PORT=8088 +DOCKER_BIN=/usr/local/bin/docker +# 1 uses sudo -n for Docker only; configure the documented sudoers entry first. +DOCKER_SUDO=1 +# The pull-only registry login is provisioned separately using docker login. +DOCKER_CONFIG=/volume1/docker/agiledancers-production/docker-auth diff --git a/deploy/provision.sh b/deploy/provision.sh new file mode 100644 index 0000000..75ed8c1 --- /dev/null +++ b/deploy/provision.sh @@ -0,0 +1,38 @@ +#!/bin/sh +# Account-level provisioning. Docker sudo must already be enabled separately. +set -eu +umask 077 +root=$1 +registry_user=$2 +registry=$3 +bind=$4 +port=$5 +docker=/usr/local/bin/docker +test -x "$docker" +sudo -n "$docker" compose version +sudo -n "$docker" info --format '{{.OSType}}/{{.Architecture}}' +test ! -L "$root" +mkdir -p "$root" "$root/docker-auth" +chmod 700 "$root" "$root/docker-auth" +if [ ! -f "$root/production.env" ]; then + # First provisioning must not take a port already occupied by another service. + command -v netstat >/dev/null + if netstat -lnt | awk -v port=":$port" '$4 ~ (port "$") { found=1 } END { exit !found }'; then + echo 'Frontend port is already occupied; production configuration unchanged' >&2 + exit 1 + fi + printf 'BIND_IP=%s\nFRONTEND_PORT=%s\nDOCKER_BIN=%s\nDOCKER_SUDO=1\nDOCKER_CONFIG=%s/docker-auth\n' \ + "$bind" "$port" "$docker" "$root" > "$root/production.env" + chmod 600 "$root/production.env" +else + # Preserve existing config; refuse to silently rebind a configured deployment. + . "$root/production.env" + test "$BIND_IP" = "$bind" && test "$FRONTEND_PORT" = "$port" \ + && test "$DOCKER_CONFIG" = "$root/docker-auth" || { + echo 'Existing production.env differs; review it before provisioning' >&2 + exit 1 + } +fi +# Credential enters only via stdin. No password, environment or auth config is logged. +sudo -n "$docker" --config "$root/docker-auth" login "$registry" --username "$registry_user" --password-stdin +echo 'Production directory and dedicated registry login prepared; no containers were changed.' diff --git a/deploy/remote.sh b/deploy/remote.sh new file mode 100644 index 0000000..3f28fa5 --- /dev/null +++ b/deploy/remote.sh @@ -0,0 +1,186 @@ +#!/bin/sh +# Invoked over a pinned SSH connection by ReleaseRemote. No secrets in arguments. +# mkdir is the NAS-wide lock; a disconnected runner leaves it for explicit recovery. +set -eu +umask 077 +action=$1 +root=$2 +owner=$3 +shift 3 +die() { echo "S6: $*" >&2; exit 1; } +case "$root" in /*) ;; *) die 'Deployment directory must be absolute';; esac +case "$owner" in ''|*[!a-zA-Z0-9-]*) die 'Invalid lock owner';; esac +test -d "$root" || die 'Provision the deployment directory first' +cd "$root" +lock="$root/deploy.lock" +release="$root/releases/$owner" + +if [ "$action" = lock ]; then + test ! -e pending.json || die 'Unresolved deployment: follow manual recovery runbook' + mkdir "$lock" 2>/dev/null || die 'Deployment locked: another run or manual recovery owns the NAS' + printf '%s\n' "$owner" > "$lock/owner" + # Check again after acquiring, covering races with another lock holder. + if [ -e pending.json ]; then + rm "$lock/owner" + rmdir "$lock" + die 'Unresolved deployment: follow manual recovery runbook' + fi + exit 0 +fi +test -f "$lock/owner" && [ "$(cat "$lock/owner")" = "$owner" ] || die 'Deployment lock ownership mismatch' + +case "$action" in + unlock) + test ! -e pending.json || die 'Pending deployment retained; manual recovery required' + rm "$lock/owner" + rmdir "$lock" + exit 0;; + state) + if [ -f state.json ]; then cat state.json; else printf '{}\n'; fi + exit 0;; + pending) + test ! -e pending.json || die 'Deployment already pending' + cat > pending.json.tmp + mv pending.json.tmp pending.json + exit 0;; + commit) + test -f pending.json || die 'No pending deployment' + cat > state.json.tmp + mv state.json.tmp state.json + exit 0;; + finish) + test -f pending.json || die 'No pending deployment' + mkdir -p history + mv pending.json "history/$owner.json" + exit 0;; + event) + mkdir -p history + cat > "history/$owner-outcome.json.tmp" + mv "history/$owner-outcome.json.tmp" "history/$owner-outcome.json" + exit 0;; +esac + +test -f production.env || die 'Missing production.env; follow provisioning runbook' +# This account-owned file is trusted shell configuration, never a workflow input. +. "$root/production.env" +: "${BIND_IP:?}" "${FRONTEND_PORT:?}" "${DOCKER_BIN:?}" "${DOCKER_CONFIG:?}" +export DOCKER_CONFIG +docker_cmd() { + if [ "${DOCKER_SUDO:-1}" = 1 ]; then + sudo -n "$DOCKER_BIN" --config "$DOCKER_CONFIG" "$@" + else + "$DOCKER_BIN" --config "$DOCKER_CONFIG" "$@" + fi +} +compose() { + docker_cmd compose --project-name agiledancers-production --project-directory "$root" \ + --env-file "$release/release.env" -f "$release/compose.yaml" "$@" +} +identity() { + image=$1 + sha=$2 + test "$(docker_cmd image inspect --format '{{index .Config.Labels "org.opencontainers.image.revision"}}' "$image")" = "$sha" \ + || die 'Image revision does not match selected SHA' + test "$(docker_cmd image inspect --format '{{.Os}}/{{.Architecture}}' "$image")" = linux/amd64 \ + || die 'Image platform must be linux/amd64' +} +use_release() { + case "$1" in ''|*[!a-zA-Z0-9-]*) die 'Invalid release directory';; esac + release="$root/releases/$1" + test -f "$release/compose.yaml" && test -f "$release/release.env" || die 'Missing retained release configuration' +} +health() { + backend=$(compose ps -q backend) + frontend=$(compose ps -q frontend) + test -n "$backend" && test -n "$frontend" || return 1 + # Compare image IDs, revision labels and the container release labels. + for role in backend frontend; do + id=$(compose ps -q "$role") + if [ "$role" = backend ]; then expected=$BACKEND_IMAGE; else expected=$FRONTEND_IMAGE; fi + test "$(docker_cmd inspect --format '{{.State.Running}}' "$id")" = true || return 1 + test "$(docker_cmd inspect --format '{{.Image}}' "$id")" = "$(docker_cmd image inspect --format '{{.Id}}' "$expected")" || return 1 + test "$(docker_cmd inspect --format '{{index .Config.Labels "nl.agilecc.revision"}}' "$id")" = "$RELEASE_SHA" || return 1 + test "$(docker_cmd inspect --format '{{index .Config.Labels "nl.agilecc.release"}}' "$id")" = "$RELEASE_VERSION" || return 1 + done + docker_cmd exec "$frontend" wget -q -T 5 -O /dev/null http://backend:8080/q/health/ready || return 1 + curl --noproxy '*' --fail --silent --show-error --max-time 5 "http://$BIND_IP:$FRONTEND_PORT/" -o /dev/null || return 1 + curl --noproxy '*' --fail --silent --show-error --max-time 5 "http://$BIND_IP:$FRONTEND_PORT/api/dance-schools" -o /dev/null +} + +case "$action" in + preflight) + command -v timeout >/dev/null || die 'timeout is required for bounded remote operations' + command -v curl >/dev/null || die 'curl is required' + test -x "$DOCKER_BIN" || die 'Docker executable missing' + docker_cmd compose version >/dev/null + test "$(docker_cmd info --format '{{.OSType}}/{{.Architecture}}')" = linux/x86_64 \ + || test "$(docker_cmd info --format '{{.OSType}}/{{.Architecture}}')" = linux/amd64 \ + || die 'Synology Docker platform is not linux/amd64' + test -d "$DOCKER_CONFIG" && test -f "$DOCKER_CONFIG/config.json" || die 'Provision registry pull login first' + # Explicit private IPv4 binding; prevent accidental 0.0.0.0 publication. + case "$BIND_IP" in + 10.*|192.168.*|172.16.*|172.17.*|172.18.*|172.19.*|172.2[0-9].*|172.30.*|172.31.*|127.*) ;; + *) die 'Bind frontend to a private IPv4 address';; + esac + case "$FRONTEND_PORT" in ''|*[!0-9]*) die 'Invalid frontend port';; esac + test "$FRONTEND_PORT" -ge 1024 && test "$FRONTEND_PORT" -le 65535 || die 'Use port 1024..65535' + ;; + stage) + mkdir -p "$release" + test ! -e "$release/compose.yaml" || die 'Attempt configuration already exists' + cat > "$release/compose.yaml" + ;; + prepare) + version=$1; sha=$2; prefix=$3 + backend=''; frontend='' + for role in backend frontend; do + repository="$prefix$role" + image="$repository:$sha" + docker_cmd pull --platform linux/amd64 "$image" >&2 + identity "$image" "$sha" + digest=$(docker_cmd image inspect --format '{{range .RepoDigests}}{{println .}}{{end}}' "$image" \ + | awk -v prefix="$repository@sha256:" 'index($0,prefix)==1 { print; exit }') + test -n "$digest" || die 'Missing registry digest' + if [ "$role" = backend ]; then backend=$digest; else frontend=$digest; fi + done + printf 'RELEASE_VERSION=%s\nRELEASE_SHA=%s\nBACKEND_IMAGE=%s\nFRONTEND_IMAGE=%s\nBIND_IP=%s\nFRONTEND_PORT=%s\n' \ + "$version" "$sha" "$backend" "$frontend" "$BIND_IP" "$FRONTEND_PORT" > "$release/release.env" + compose config --quiet + printf 'S6_BACKEND=%s\nS6_FRONTEND=%s\n' "$backend" "$frontend" + ;; + ensure) + use_release "$1" + . "$release/release.env" + test "$RELEASE_VERSION" = "$2" && test "$RELEASE_SHA" = "$3" \ + && test "$BACKEND_IMAGE" = "$4" && test "$FRONTEND_IMAGE" = "$5" \ + || die 'Retained configuration differs from recorded successful release' + for image in "$BACKEND_IMAGE" "$FRONTEND_IMAGE"; do + docker_cmd pull --platform linux/amd64 "$image" >&2 + identity "$image" "$RELEASE_SHA" + done + compose config --quiet + ;; + apply) + mkdir "$lock/operation" 2>/dev/null || die 'A remote replacement is still active; manual recovery required' + trap 'rmdir "$lock/operation"' EXIT + use_release "$1" + . "$release/release.env" + health_seconds=${2:-280} + case "$health_seconds" in ''|*[!0-9]*) die 'Invalid readiness window';; esac + test "$health_seconds" -ge 1 && test "$health_seconds" -le 280 || die 'Invalid readiness window' + end=$(( $(date +%s) + health_seconds )) + # Always recreate the frontend too: nginx must resolve the new backend IP. + compose up -d --no-build --pull never --force-recreate --timeout 20 + while ! health; do + test "$(date +%s)" -lt "$end" || die 'Readiness checks timed out' + sleep 1 + done + echo 'S6_HEALTHY' + ;; + diagnose) + use_release "$1" + compose ps + compose logs --no-color --tail 200 + ;; + *) die 'Unknown remote deployment operation';; +esac diff --git a/docs/gitea-ci.md b/docs/gitea-ci.md index 9ef61ca..78649cc 100644 --- a/docs/gitea-ci.md +++ b/docs/gitea-ci.md @@ -2,7 +2,9 @@ De workflow staat in `.gitea/workflows/build.yaml`. Elke branchpush bouwt en test backend en frontend, bouwt twee images en draait Playwright tegen die containers. -Alleen `main` mag daarna images publiceren. Er vindt geen deployment plaats. +Alleen `main` mag daarna images publiceren. Deze buildworkflow voert geen deployment uit. +S6 voegt aparte handmatige workflows toe voor productie en rollback; zie +[productiedeployment](production-deployment.md). Tagpushes en aparte pull-requestevents starten geen extra workflow. Een auteur kan een branchbuild bewust overslaan met een door Gitea ondersteunde markering, zoals `[skip ci]`; dit is expliciet toegestaan. diff --git a/docs/production-deployment.md b/docs/production-deployment.md new file mode 100644 index 0000000..6e8187a --- /dev/null +++ b/docs/production-deployment.md @@ -0,0 +1,281 @@ +# Productie uitrollen op de Synology (S6) + +De buildworkflow bouwt, test en publiceert automatisch. Productie verandert +alleen via **Actions → Deploy production → Run workflow** of **Rollback +production**. Start beide vanaf `main`; de workflows verschijnen na de merge. + +**Tot S16 is geïmplementeerd kan iedere deployment, retry, rollback of automatisch +herstel alle applicatiegegevens wissen.** Informeer de launching partner vooraf. +Een onderbroken berekening moet opnieuw worden gestart. Een herstelde +applicatieversie herstelt geen verloren gegevens. + +## Eenmalige DSM-inrichting + +Gebruik je bestaande beheeraccount voor deze stappen: + +1. Maak onder **Configuratiescherm → Gebruiker en groep → Gebruiker → Maken** + het account `agiledancers-deploy` met een uniek sterk wachtwoord. Voeg het toe + aan **administrators**: DSM vereist deze groep voor SSH. Het is daardoor een + beheerder, geen beperkt bestandsaccount. +2. Bij **Gedeelde mapmachtigingen**: `docker` op **Lezen/Schrijven**, `homes` + ongewijzigd, overige mappen op **Geen toegang**. `git` is niet nodig: deployment + gebruikt de Gitea-API en registry. Blokkeer `homes` niet, omdat dit ook de eigen + thuismap kan blokkeren. +3. De toepassingen onder **Toegangsmachtigingen toewijzen** zijn niet nodig en + mogen op **Weigeren**. Gebruik je eigen beheeraccount voor DSM. Deployment + gebruikt SSH-shellopdrachten, geen SFTP, rsync of bestandsdeling. +4. Schakel onder **Gebruiker en groep → Geavanceerd** de + **gebruikersbasismapservice** in als deze uitstaat. +5. Schakel **Terminal en SNMP → Terminal → SSH-service inschakelen** in. De + maintainer heeft SSH-poort **22** bevestigd. Beperk toegang tot mini-pc/LAN/VPN; + configureer geen internet-portforwarding. +6. Controleer dat Container Manager/Docker draait, dat Docker Compose **v2**, + `curl`, `timeout` met `-k` en `sh` beschikbaar zijn. Op de NAS is geen Java, + Python of Git nodig. Controleer het Docker-pad, doorgaans `/usr/local/bin/docker`. + +Map- en toepassingsrechten vormen geen beveiligde isolatie voor een account met +beheerder-/Dockerrechten. De runner voert ook vertrouwde repositorycode uit. +Geef alleen vertrouwde beheerders schrijftoegang tot workflows en credentials. + +Bronnen: [SSH op DSM](https://kb.synology.com/en-au/DSM/help/DSM/AdminCenter/system_terminal), +[sleutelaanmelding en basismappen](https://kb.synology.com/en-us/DSM/tutorial/How_to_log_in_to_DSM_with_key_pairs_as_admin_or_root_permission_via_SSH_on_computers), +[maprechten](https://kb.synology.com/nl-nl/DSM/help/DSM/AdminCenter/file_share_privilege?version=7). + +## SSH-sleutel en hostkey + +Maak op de mini-pc een aparte sleutel buiten de repository. De lege passphrase +maakt niet-interactieve uitvoering mogelijk; behandel de private sleutel als secret. + +```sh +ssh-keygen -t ed25519 -f "$HOME/.config/agiledancers-runner/s6-deploy-key" -N '' -C agiledancers-production +ssh-copy-id -i "$HOME/.config/agiledancers-runner/s6-deploy-key.pub" -p 22 agiledancers-deploy@192.168.2.28 +``` + +Genereer niet opnieuw als de sleutel al bestaat. De implementatiesessie heeft +deze dedicated sleutel op de mini-pc aangemaakt. De gebruiker installeert de +publieke sleutel via een eigen interactieve terminal; deel geen wachtwoorden. + +Controleer de serverfingerprint via een reeds vertrouwde beheerverbinding, tegen +`/etc/ssh/ssh_host_ed25519_key.pub`. Een keyscan alleen bewijst de serveridentiteit +niet. Bewaar de geverifieerde known_hosts-regel; bij een andere poort gebruikt +die `[host]:poort`. De helper vereist `StrictHostKeyChecking=yes`, gebruikt alleen +de dedicated sleutel en negeert SSH-agent en lokale SSH-configuratie. + +Op de NAS: eigen `~/.ssh` mode `700`, `authorized_keys` mode `600`, juiste eigenaar, +thuismap niet schrijfbaar voor andere gebruikers. De public-keyregel kan worden +beperkt met de werkelijke vaste mini-pc-IP en zonder forwarding/TTY: + +```text +from="",no-agent-forwarding,no-port-forwarding,no-X11-forwarding,no-pty ssh-ed25519 agiledancers-production +``` + +Als de sleutel na `ssh-copy-id` nog wordt geweigerd door ruime DSM-basismaprechten, +voer dan eenmaal met wachtwoordauthenticatie uit als dit account: + +```sh +chmod go-w "$HOME" +chmod 700 "$HOME/.ssh" +chmod 600 "$HOME/.ssh/authorized_keys" +``` + +## Productiemap en Dockerrechten + +Controleer het volume: `/volume1/docker/agiledancers-production` is een voorbeeld, +geen vastgestelde locatie. Maak de map als deploymentaccount, mode `700`, met +dat account als eigenaar. Houd deze stack gescheiden van Gitea en andere projecten. + +Kopieer `deploy/production.env.example` naar `/production.env`, mode +`600`. Stel de private NAS-IP, vrije frontendpoort (voorbeeld `8088`), absolute +Docker-locatie en `DOCKER_CONFIG` in. Gebruik een eigen `docker-auth`-submap. +Alleen de frontend bindt aan de private IPv4; de backend krijgt geen hostpoort. +De Docker-projectnaam is vast `agiledancers-production`. + +Als Docker sudo vereist, laat een beheerder met `visudo` een afzonderlijke +sudoers-regel installeren en valideren, met het gecontroleerde Docker-pad: + +```text +agiledancers-deploy ALL=(root) NOPASSWD: /usr/local/bin/docker +``` + +Dit staat alleen dat executable toe, maar Dockerbeheer geeft in de praktijk +rootmogelijkheden. Gebruik geen `NOPASSWD: ALL`, wereldschrijfbare Docker-socket +of wachtwoord in de workflow. De helper gebruikt `sudo -n`. Als Docker al direct +toegankelijk is voor het account, stel `DOCKER_SUDO=0` in. + +Provision apart een **pull-only** Gitea-packagecredential. Log als deploymentaccount +op de NAS in, via de interactieve wachtwoordprompt: + +```sh +mkdir -p /volume1/docker/agiledancers-production/docker-auth +chmod 700 /volume1/docker/agiledancers-production/docker-auth +sudo -n /usr/local/bin/docker --config /volume1/docker/agiledancers-production/docker-auth login 127.0.0.1:3000 --username +sudo -n /usr/local/bin/docker compose version +sudo -n /usr/local/bin/docker info --format '{{.OSType}}/{{.Architecture}}' +``` + +Als DSM geen `visudo` bevat, is `deploy/enable-docker.sh` het eenmalige alternatief. +Kopieer het naar de eigen NAS-thuismap en voer het daar met sudo uit. Het maakt +uitsluitend de bovengenoemde regel, controleert die door als deploymentaccount +Docker te starten en verwijdert een nieuwe regel als die controle faalt. Een +afwijkende bestaande regel wordt niet overschreven. De implementatiesessie heeft +deze stap op de NAS uitgevoerd; Docker rapporteert `linux/x86_64`. + +Pas paden aan voor de echte configuratie. De bevestigde NAS-Dockerdaemon staat +HTTP-registryverkeer alleen op loopback toe. Gebruik daarom vanaf die NAS +`127.0.0.1:3000`: dezelfde Gitea-registry en gepubliceerde SHA-images, zonder een +daemonherstart die Gitea en Kimai zou onderbreken. De mini-pc blijft naar +`192.168.2.28:3000` publiceren. Deploymentrecords bevatten de op de NAS bruikbare +loopback-digestreferenties. Authenticatie-/netwerkfouten zijn deploymentfouten. + +## Gitea-configuratie + +Stel onder repository **Settings → Actions** in: + +| Soort | Naam | Inhoud | +| --- | --- | --- | +| Variabele | `DEPLOY_HOST` | Private NAS-IP/hostnaam, passend bij known_hosts | +| Variabele | `DEPLOY_USER` | `agiledancers-deploy` | +| Variabele | `DEPLOY_SSH_PORT` | `22` | +| Variabele | `DEPLOY_DIRECTORY` | Absolute productiemap | +| Variabele | `DEPLOY_REGISTRY` | `127.0.0.1:3000`, de NAS-lokale alias van dezelfde registry | +| Secret | `DEPLOY_SSH_KEY` | Volledige private sleutel, inclusief begin/eindregels | +| Secret | `DEPLOY_KNOWN_HOSTS` | Onafhankelijk geverifieerde NAS-hostkey-regel | +| Secret | `RELEASE_TOKEN` | Apart Gitea-token voor Actions lezen en Git-tags maken | + +Het releaseaccount heeft repositorylees-/schrijfrechten en Actions-metadatatoegang +nodig (`write:repository` op deze Gitea). Het NAS-registrycredential is apart en +heeft alleen de noodzakelijke package-leestoegang. Hergebruik de publicatie-PAT +niet voor deployment. Bewaar secrets buiten Git en logs. + +De setuphelper kan deze configuratie en de productiemap zonder containerstart +inrichten. Bewaar `admin-token`, `s6-release-token`, `s6-pull-token` en de private +`s6-deploy-key` als owner-only bestanden (mode `600`) in +`~/.config/agiledancers-runner/`. Gebruik de eerder vertrouwde NAS-hostkey in +`~/.ssh/known_hosts` en voer uit: + +```sh +java ci/Ci.java configure-deployment +``` + +Defaults zijn de bevestigde NAS/account/SSH-poort, repository +`wanjakrah/agiledancers`, productiemap `/volume1/docker/agiledancers-production`, +NAS-IP-binding `192.168.2.28` en frontendpoort `8088`. De helper controleert een +geslaagde main-build, weigert een bezette poort bij eerste inrichting, bewaart +bestaande productieconfiguratie en controleert de aparte registry-login voordat +Gitea-secrets/variabelen worden gezet. Er wordt geen release/tag gemaakt en geen +container vervangen. Variabelen met de namen uit bovenstaande tabel en +`DEPLOY_BIND_IP`, `DEPLOY_FRONTEND_PORT`, `PULL_USER` en de credential-`*_FILE`- +paden kunnen de defaults expliciet vervangen. + +Bescherm release-tags `v*` in Gitea en beperk tagcreatie tot de releasebeheerder. +De helper maakt alleen nieuwe tags; hij verplaatst of verwijdert ze nooit. Laat +registry-cleanup de images van huidig/vorig niet verwijderen. Gebruik geen +algemene Docker-image/volume-prune als onderhoud van deze stack. + +## Nieuwe release, retry en rollback + +1. Merge naar `main` en wacht op een geslaagde **Build, test and publish** voor + die SHA, inclusief imagepublicatie. +2. Open **Deploy production → Run workflow**, branch `main`, mode `new`, keuze + `patch`, `minor` of `major`. Laat het versieveld leeg. De eerste release is + `v1.0.0`; daarna telt de hoogste bestaande release-tag, inclusief mislukte + deployments. Minor/major zetten de lagere componenten op nul. +3. De workflow toont versie/SHA. De immutable annotated Git-tag wordt na + imagevoorbereiding, vóór vervanging gemaakt en blijft bij falen bestaan. + Een tag bewijst geen succesvolle productiedeployment. +4. Gebruik mode `retry` met een bestaande tag om dezelfde SHA/versie opnieuw uit + te rollen. De incrementkeuze wordt genegeerd: Gitea toont alle velden, ook + wanneer ze voor de gekozen mode niet relevant zijn. Een retry mag afwijken + van huidig main, maar geen nieuwere succesvolle release verdringen. +5. **Rollback production → Run workflow → main** herstelt de vorige succesvolle + release zonder versie/tag. Na succes wisselen huidig/vorig om; nogmaals + rollback kan dus terugschakelen. De hoogste succesvolle versie blijft bewaard. + +De NAS-lock omvat alle modes en versieallocatie. Een gelijktijdige run faalt +zichtbaar. Gitea 1.25 negeert workflow-`concurrency`; de serverlock is leidend. +Main wordt direct vóór vervanging opnieuw gecontroleerd. Zodra vervanging start, +wordt de gepinde versie afgemaakt of hersteld. Branchbuilds rollen niets uit. + +## Checks en operationele state + +De helper haalt beide SHA-images terug, controleert revisie/platform en legt +registry-digests vast. Compose gebruikt die digests, nooit de bewegende main-tag. +De recovery-images worden eveneens vooraf gecontroleerd. Binnen vijf minuten +moeten backend-readiness, frontendpagina, GET `/api/dance-schools` via de +gepubliceerde frontendpoort en containeridentiteit slagen. Deze checks schrijven +geen gebruikersdata. De frontend herstart ook om de nieuwe backend-IP te resolven. + +De productiemap bevat: + +- `state.json`: atomisch geschreven huidig/vorig en hoogste succesvolle versie. + Redeployment van dezelfde versie behoudt de afzonderlijke rollbackversie. +- `releases//`: behouden Compose- en digest-/versieconfiguratie. +- `pending.json`: kandidaat en vorige state, vóór iedere vervanging geschreven. +- `deploy.lock/owner`: unieke run/poging; `operation/` markeert een actieve + vervanging op de NAS. +- `history/`: pogingjournalen en uitkomsten; niet opruimen tijdens een release. + +Een ongezonde kandidaat herstelt automatisch de vorige succesvolle release en +blijft een mislukte workflow. Bij de eerste release zonder voorganger, mislukt +herstel, runner/SSH-onderbreking of onduidelijke statecommit blijven journal en +lock staan. Nieuwe runs weigeren dan wijzigingen. Remote opdrachten hebben ook +op de NAS een timeout; herstel overlapt geen nog actieve vervanging. Geen +eindeloze herstelpogingen. + +Download artifact `deployment--` of `rollback--` voor +log, uitkomst, versie/SHA/digests en beschikbare foutdiagnostiek. Retentie is +veertien dagen; serverinstellingen mogen dit niet inkorten. Log geen environment, +Docker-authconfig of volledige container-inspect met eventuele secrets. + +## Handmatig herstel + +Gebruik een vertrouwde beheerverbinding. Verwijder nooit alleen de lock om een +nieuwe release door te drukken. + +1. Annuleer de betrokken Gitea-run. Controleer op de NAS dat geen SSH-deployment, + timeout, Compose-vervanging of herstel van deze poging meer actief is. Onderzoek + `deploy.lock/operation`; wacht op de timeout of beëindig alleen de geverifieerde + vastgelopen deploymentprocessen. +2. Bewaar state, pending journal, history en containerlogs. Vergelijk de kandidaat, + de `before`-state en werkelijke images. De statecommit kan al zijn uitgevoerd + terwijl de runner geen antwoord ontving. +3. Behoud een aantoonbaar gezonde kandidaat met overeenkomende state. Anders: + herstel `before.current` uit het journal met de behouden Compose-/env-bestanden, + dezelfde projectnaam en digest-images. Start geen tweede stack. +4. Voer dezelfde checks uit. Vanaf de checkout op de mini-pc kun je het echte + controlescript uitvoeren met bestaande lockowner en gekozen release-directory: + + ```sh + ssh -i -p @ \ + 'timeout -k 5 300 sh -s -- apply ' < deploy/remote.sh + ``` + + Gebruik de eerder gepinde hostkey en gecontroleerde waarden uit het journal. + Verwijder een achtergebleven lege `operation/` alleen als stap 1 heeft bewezen + dat de oude operatie gestopt is. +5. Bij herstel van de oude release: herstel ook de volledige `before`-state + atomisch, inclusief high-watermark en rollbackdoel. Bij een mislukte eerste + release: stop de defecte stack en behoud de lege succesvolle state. Verwijder + geen volumes of release-tags. +6. Noteer het herstel in history, archiveer het opgeloste journal en verwijder + daarna pas de lockowner en lege lockdirectory. Start vervolgens een retry. + Laat de blokkade staan als gezondheid of state onduidelijk is. + +## Na S16 en lokale verificatie + +S16 blokkeert S6 niet. Zodra S16 is ingevoerd, moeten database/volumes buiten de +vervangbare containers blijven en saved data zowel deployment als rollback +overleven. De gewone procedure vereist dan databasecompatibiliteit met de vorige +applicatieversie. Plan incompatibele migraties apart: imageherstel maakt geen +migratie ongedaan. Backup/restore blijven S17-werk. + +```sh +java ci/Ci.java test +java ci/Ci.java release-test +sh -n deploy/remote.sh +mvn -N -Dformat.mode=check exec:exec@format +``` + +De buildworkflow voert beide Java-testsuites uit vóór publicatie. Release-tests +wijzigen geen productie, registry of tags. Leg echte NAS-/Gitea-bewijzen vast in +`docs/s6-validation.md`; configuratie en lokale tests alleen voltooien S6 niet. diff --git a/docs/s6-validation.md b/docs/s6-validation.md new file mode 100644 index 0000000..4c88474 --- /dev/null +++ b/docs/s6-validation.md @@ -0,0 +1,107 @@ +# S6 implementation and operational evidence + +Status: IN_PROGRESS. Repository implementation and local validation do not yet +establish an operational Synology deployment. No production release tags have +been created by this implementation session. + +## Confirmed environment + +- Work is on `feature/S6_deployment`. +- Read-only authenticated inspection of the installed Gitea confirmed run fields + `path=build.yaml@refs/heads/main`, `status=completed`, `conclusion=success`, + `event=push`, `head_branch=main` and `head_sha`. +- No remote Git tags existed at the time of that read-only inspection. +- The maintainer created the dedicated DSM account `agiledancers-deploy` and + confirmed SSH port 22. A separate Ed25519 deployment key was generated outside + Git on the mini-pc and installed on the NAS. + +Subsequent live checks confirmed key-based access after removing group/other +write permissions from the account home. The maintainer installed the dedicated +Docker sudo rule using `deploy/enable-docker.sh`; a subsequent noninteractive +check reports `linux/x86_64`. NAS model is **DS224+**, Compose is +**v2.20.1-6047-g6817716**, `timeout -k` and `curl` are available, and port **8088** +was free. The daemon allows HTTP registry access only through `127.0.0.0/8`. +Deployment therefore uses the NAS-local `127.0.0.1:3000` alias of the same Gitea +registry, without restarting Gitea/Kimai or changing the shared Docker daemon. + +## NAS setup and real image preparation + +`java ci/Ci.java configure-deployment` succeeded using separate owner-only local +pull/release tokens. It provisioned `/volume1/docker/agiledancers-production`, +the private IP/port configuration and registry pull login, verified the real NAS +preflight, and stored the three deployment secrets and five variables in Gitea. +No secret values are recorded here. + +A real preparation probe exercised the production coordinator/API/SSH adapter +and NAS script, stopping explicitly at the tag-creation boundary. It retrieved +both images for verified main commit +`b1544357bf6ea5e1607e1de1fa4872b9510690a8` (Gitea API build run ID **35**), verified +revision/platform labels, selected registry digests and passed Compose v2.20 +configuration validation. It released the deployment lock without creating a +Git tag, pending journal, successful release state or application containers. +The displayed candidate `v1.0.0` was not allocated as a release. + +| Component | Retrieved digest (NAS registry alias `127.0.0.1:3000/wanjakrah/`) | +| --- | --- | +| Backend | `sha256:d46af705c527a8eaa506407206cfce927edbc4b774951cf13da8702b5cb8c56d` | +| Frontend | `sha256:d6c09ad839b929591878195c7403cbf0bbf24ac6e8d335ef90d72b7982c4ba31` | + +Local probe evidence: `ci/artifacts/deployment/setup.log`, `nas-image-probe.log` +and `nas-image-probe.json` (ignored by Git). Retained configuration on the NAS: +`releases/image-probe-bbb2a980-e624-4331-adb6-b8fffd51fa5d/`. This is preparation +evidence, not a successful production release or Gitea dispatch run. + +## Local checks + +- `java ci/Ci.java test`: **17 passed**, including publication gates and process + deadlines. +- `java ci/Ci.java release-test`: **32 passed**, including all version increments, + input/main/manual gates, verified-build provenance, digest preparation failures, + duplicate/stale release rejection, retry, version high-watermark after rollback, + automatic recovery, failed recovery/first release, ambiguous state commits, + lock contention, annotated-tag identity and registry aliases. +- Seven of those release tests execute the actual POSIX NAS script against + disposable command adapters, covering image/platform checks, published endpoint + requests, retained-configuration mismatch, bounded readiness failure, overlapping + replacement fencing and atomic state/journal handling. These are simulated + failure-path tests, separate from the real NAS preparation probe above. +- `sh -n` passed separately for `deploy/remote.sh`, `deploy/provision.sh` and + `deploy/enable-docker.sh`. +- `mvn -N -Dformat.mode=check exec:exec@format`: passed with no differences. +- All workflow YAML parsed; assertions confirmed automatic builds, manual-only + release/rollback triggers and the patch/minor/major form choices. +- `docker compose -f deploy/compose.yaml config --quiet` passed with representative + digest references; real NAS Compose validation passed during preparation too. +- `git diff --check`: passed. + +Backend/frontend application code is unchanged. A full Gitea feature/main build +and dispatched production/recovery evidence remain to be collected after review. + +## Required operational acceptance + +- [x] Record NAS model/platform, Docker/Compose versions, private endpoint and + production directory (no secret values). +- [x] Verify pinned SSH hostkey, key-only automation and separate registry pull + credential. Release-token read access is verified; actual tag-writing + acceptance awaits the first release. +- [ ] Verify release-tag creation/protection with the dedicated token. +- [ ] Merge workflows through the protected-branch process and identify the + successful main build/publication run and SHA. +- [ ] Demonstrate release form, first release tag and real Synology access; + record workflow URL, version, SHA and both deployed digests. +- [ ] Demonstrate rejected unverified/missing-image release and branch exclusion + without modifying production. +- [ ] Demonstrate duplicate/stale/concurrent release protection, retained tags + after failure and exact-SHA retry without another version. +- [ ] Demonstrate failed candidate recovery while the workflow reports failure, + with downloadable diagnostics. +- [ ] Demonstrate manual rollback without a new version and retained image pairs. +- [ ] Exercise first-release/recovery failure and interruption/manual recovery + using isolated acceptance resources where appropriate. +- [ ] Confirm fourteen-day artifact retention. +- [ ] Confirm operator communication of the accepted pre-S16 data-loss policy. +- [ ] Once S16 applies, verify data preservation and migration compatibility + during deployment and rollback (conditional on S16 implementation). + +Identify isolated acceptance resources explicitly. Simulated tests are not real +production evidence. diff --git a/specs/refined/S6_deployment-gitea-action-to-production.md b/specs/in_progress/S6_deployment-gitea-action-to-production.md similarity index 95% rename from specs/refined/S6_deployment-gitea-action-to-production.md rename to specs/in_progress/S6_deployment-gitea-action-to-production.md index 1bf322d..b6779b6 100644 --- a/specs/refined/S6_deployment-gitea-action-to-production.md +++ b/specs/in_progress/S6_deployment-gitea-action-to-production.md @@ -1,6 +1,6 @@ # S6 — Manual versioned production deployment through Gitea -Status: REFINED +Status: IN_PROGRESS Agreed after the grill-me interview; shared understanding confirmed. @@ -175,3 +175,10 @@ This specification records the confirmed design. Implementing workflows, release automation, Compose configuration, Synology provisioning and collecting actual deployment/recovery evidence are subsequent work. Configuration alone does not complete S6. + +Implementation is in progress on `feature/S6_deployment`: workflows, Compose, +release/retry/rollback helpers, failure-path tests and provisioning/runbook are +provided. The dedicated NAS account, registry login and Gitea deployment secrets +are provisioned. Real image preparation passed without a release or container +replacement. See `docs/s6-validation.md`; workflow execution and production +deployment/recovery evidence remain required after the protected main merge. From 60496de8bb7e5923af12c2997a7c8f3c8956a357 Mon Sep 17 00:00:00 2001 From: Wanja Krah Date: Tue, 3 Nov 2026 14:48:59 +0100 Subject: [PATCH 3/3] Record successful S6 feature build validation [skip ci] --- docs/s6-validation.md | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/docs/s6-validation.md b/docs/s6-validation.md index 4c88474..ca98216 100644 --- a/docs/s6-validation.md +++ b/docs/s6-validation.md @@ -74,8 +74,23 @@ evidence, not a successful production release or Gitea dispatch run. digest references; real NAS Compose validation passed during preparation too. - `git diff --check`: passed. -Backend/frontend application code is unchanged. A full Gitea feature/main build -and dispatched production/recovery evidence remain to be collected after review. +Backend/frontend application code is unchanged. + +## Full Gitea feature build + +[Feature build #36](http://192.168.2.28:3000/wanjakrah/agiledancers/actions/runs/36) +(API run ID **37**) completed successfully for implementation commit +`525e9f9335e4343c28f91664c149d24da9284904` on `feature/S6_deployment`. This exercises +the full existing verification path, including Maven verification, both helper +test suites, frontend checks/build, both Docker image builds and container E2E. +Image publication is skipped for the feature branch. + +The subsequent evidence-only commit intentionally uses the accepted `[skip ci]` +marker; implementation code is unchanged from this successful run. + +Main build/publication and dispatched production/recovery evidence remain to be +collected after review and the protected merge. Review PR: +[S6 #12](http://192.168.2.28:3000/wanjakrah/agiledancers/pulls/12). ## Required operational acceptance