S16 - some transactional hardening

specs/implemented/S16_store-state-in-persistent-storage

@@ -0,0 +1,155 @@
+# S16 — Store state in persistent storage
+
+Status: IMPLEMENTED
+
+Agreed after the grill-me interview; shared understanding confirmed.
+
+## User story
+
+As a dance school maintainer, I want application data to be stored persistently,
+so that confirmed changes and saved scheduling results survive server restarts
+and application crashes.
+
+## Storage decision
+
+Use PostgreSQL with relational tables for schools, dancers, coaches, groups,
+shows and their relationships. Use Flyway to version and apply database schema
+changes. Use JPA with Hibernate ORM for application persistence; keep the mappings
+inside the repository layer so the existing domain model remains framework independent.
+
+The existing domain has shared relationships and operations on individual school
+components. Related changes must be atomic, and multiple maintainers must be able
+to work without silently overwriting each other's changes. These requirements
+favor relational storage with transactions and referential constraints.
+
+A document database was considered: a whole-school document would align with
+JSON import/export. However, the agreed requirements provide no concrete benefit
+for that approach over relational storage. The application owns a fixed domain
+model; school-defined arbitrary fields and structures are not required.
+
+JSON remains the import/export format, rather than the primary storage model.
+
+## Scope and deployment
+
+- Support multiple schools and concurrent maintainers.
+- Target one production environment with one active backend instance.
+- A separate database service and its operational management are acceptable.
+- Multiple active backend instances, distribution across servers or regions, and
+  automatic resumption of calculations are outside S16.
+- Keep the implementation within existing project architecture and coding
+  standards; this specification grants no standards waiver.
+
+## Durability and consistency
+
+- Confirm a mutation as successful only after its database transaction commits
+  durably. All confirmed domain changes and saved scheduling results survive an
+  application restart or crash.
+- Persist schools and all their maintained domain data, relationships and saved
+  planning state, rather than only a subset of the existing in-memory state.
+- Related mutations either succeed completely or leave the previous state intact.
+  This includes deleting a dancer and cleaning up memberships, deleting a group
+  and cleaning up show references, deleting a school and its children, and
+  replacing a school through import.
+- Validate imports before publishing the replacement. Preserve the current
+  import/export behavior and identity semantics.
+- Enforce valid relationships through database constraints alongside application
+  validation.
+
+## Concurrent changes and scheduling
+
+- Use one persistent revision per school for optimistic concurrency control.
+- Mutations must identify the school revision on which they are based. Check
+  that revision and commit the change and new revision atomically.
+- Reject stale mutations with a clear conflict message; do not silently overwrite
+  newer changes. The user can reload current data and decide again.
+- Changes to different components of the same school may therefore conflict.
+  Finer-grained conflict detection is outside S16.
+- Retain the scheduling input revision check: a result computed from an outdated
+  input revision must not replace current planning state.
+- After restart, calculations interrupted by the restart must be recognizably
+  interrupted, rather than remain indefinitely marked as running. They do not
+  resume automatically. Previously saved results remain available.
+
+## Persistent scheduling status (transferred from S12)
+
+The maintainer explicitly moved durable scheduling status and backend-restart
+recovery from S12 to S16 when completing S12. S12 supplies background execution,
+status snapshots, SSE and notifications within the lifetime of the backend.
+
+- Persist the latest calculation per show in PostgreSQL, including its run
+  identifier, school/show identity, start and finish timestamps, state, latest
+  phase, whether a valid solution was found, outcome explanation and final
+  attendance/spread/variety summary when available. A run history is not required.
+- Persist acceptance of a run before acknowledging its start. After restart or
+  crash, an accepted calculation must remain identifiable even if it never
+  produced a result.
+- Commit successful assignment replacement and its completed run status
+  atomically. Publish success through the status API and SSE only after the
+  transaction commits. A failed commit must not leave a successful status with
+  missing assignments, or replaced assignments with an outdated running status.
+- On startup, mark calculations left Running by the previous backend instance
+  as Interrupted before serving their recovered status. Explain that no new
+  result was applied, retain previously saved assignments and allow a new run.
+  Do not resume calculations automatically or retain their old run-slot guards.
+- Recover the latest status and outcome through S12's existing status endpoint
+  and initial/reconnection SSE snapshots. Preserve stable run identifiers so
+  reconnecting clients can deduplicate notifications; reopening the application
+  must not replay old completion snackbars.
+- Verify recovery after both graceful restart and abrupt process termination,
+  including a calculation running at termination and a completed saved result.
+
+## Database failures and schema changes
+
+- Refuse mutations while the database is unavailable and provide a clear failure
+  message. Do not acknowledge successful storage or fall back to volatile writes.
+- Connection and Flyway migration failures at startup must be clearly visible;
+  the application must not present itself as ready for normal use when required
+  database initialization has failed.
+- Maintain the database schema through versioned Flyway migrations.
+
+## Transition from in-memory storage
+
+Start with an empty database. Existing data is transferred through the current
+JSON export/import path; no separate in-memory migration mechanism is required.
+Document that existing data must be exported before stopping the old application
+for the transition, because its current state exists only in memory.
+
+## Deferred work
+
+Database backup and restore are outside S16 and will be specified separately in
+`specs/S17_database-backup-and-restore`. That specification is intentionally empty
+for now. S16 does not promise recovery after loss of the database or its volume.
+
+## Acceptance criteria
+
+- [x] PostgreSQL stores the complete maintained domain state and saved scheduling
+      results, with Flyway managing the relational schema.
+- [x] Confirmed changes remain available after both a normal restart and an
+      application crash, using the same persistent database.
+- [x] Failed related mutations and failed school replacement imports leave no
+      partial changes or dangling references.
+- [x] Two edits based on the same school revision cannot silently overwrite one
+      another: the stale edit receives a clear conflict response, and the user
+      can reload and retry.
+- [x] Scheduling results based on outdated input revisions are rejected.
+- [x] The latest accepted scheduling run, its stable identifier, timestamps,
+      progress, outcome and objective summary survive normal restart and crash
+      and are returned through the status API and SSE snapshots.
+- [x] Successful assignment replacement and completed run status commit
+      atomically; success is published only after durable commit.
+- [x] Calculations left Running become Interrupted before recovered status is
+      served; previous assignments remain, no automatic resumption occurs and a
+      new run can start. Reopening does not replay old completion snackbars.
+- [x] Database unavailability prevents successful mutation acknowledgements;
+      startup connection and migration failures are clearly reported.
+- [x] An empty database can be initialized through Flyway and populated through
+      existing JSON import, preserving import/export behavior and identities.
+- [x] Setup and transition instructions cover the database service, persistent
+      storage, connection configuration, migrations and export-before-transition.
+
+## Implementation handoff
+
+PostgreSQL/JPA/Hibernate storage, Flyway migrations, school revision checks, durable scheduling status and
+restart recovery are implemented and verified. See [S16 validation](../S16-validation.md)
+and [setup and transition instructions](../../docs/persistent-storage.md).
+Implementation is ready for review; final acceptance is pending.