tegwick
fd8537af7a
Update app-data-backup-restore-handoff and vergabe-teilnahme with backup and restore evidence references. Mark workplan finished and archive.
183 lines
7.2 KiB
Markdown
183 lines
7.2 KiB
Markdown
# vergabe-teilnahme — operator runbook
|
|
|
|
Production deployment of the Django tender-management app, shipped
|
|
under `RAILIANCE-WP-0002`.
|
|
|
|
## Identity
|
|
|
|
| | |
|
|
|---|---|
|
|
| Public URL | https://vergabe-teilnahme.whywhynot.de |
|
|
| Namespace | `vergabe-teilnahme` |
|
|
| Helm release | `vergabe-teilnahme` |
|
|
| Chart | `charts/vergabe-teilnahme/` |
|
|
| Values | `helm/vergabe-teilnahme-values.yaml` (plain — no SOPS) |
|
|
| Ingress | `manifests/vergabe-teilnahme-ingress.yaml` |
|
|
| Image | `forgejo.coulomb.social/coulomb/vergabe-teilnahme:<tag>` |
|
|
| Database | `vergabe_db` on shared cnpg `apps-pg` (see `railiance-platform/docs/apps-pg.md`) |
|
|
| TLS | `vergabe-teilnahme-tls`, issued by cert-manager `letsencrypt-prod` |
|
|
|
|
## Secrets
|
|
|
|
Two K8s Secrets in the `vergabe-teilnahme` namespace:
|
|
|
|
| Secret | Type | Source of truth | Used for |
|
|
|--------|------|-----------------|----------|
|
|
| `vergabe-app-credentials` | `kubernetes.io/basic-auth` | mirror of `databases/vergabe-app-credentials` (cnpg-owned) | raw DB role credential |
|
|
| `vergabe-teilnahme-env` | `Opaque` | created by operator | `SECRET_KEY` + URL-encoded `DATABASE_URL` (envFrom on the Deployment) |
|
|
|
|
**No SOPS encryption** for this app — all sensitive material lives in
|
|
K8s Secrets, not in committed values files.
|
|
|
|
### Rotating the DB password
|
|
|
|
1. Have `railiance-platform` rotate the cnpg-managed Secret
|
|
(`databases/vergabe-app-credentials`).
|
|
2. Mirror the new password into `vergabe-teilnahme/vergabe-app-credentials`.
|
|
3. Rebuild `DATABASE_URL` in `vergabe-teilnahme-env`, **URL-encoding
|
|
the password** (the base64 character set breaks the URL parser
|
|
otherwise - see `RAILIANCE-WP-0004 I01`):
|
|
```bash
|
|
make vergabe-db-url-secret
|
|
kubectl rollout restart deploy/vergabe-teilnahme -n vergabe-teilnahme
|
|
```
|
|
|
|
### Rotating `SECRET_KEY`
|
|
|
|
Django `SECRET_KEY` rotation invalidates active sessions but is
|
|
otherwise zero-downtime:
|
|
|
|
```bash
|
|
NEW=$(openssl rand -base64 50 | tr -d '\n' | tr '/+=' 'abc')
|
|
kubectl patch secret vergabe-teilnahme-env -n vergabe-teilnahme \
|
|
--type=merge -p "{\"stringData\":{\"SECRET_KEY\":\"$NEW\"}}"
|
|
kubectl rollout restart deploy/vergabe-teilnahme -n vergabe-teilnahme
|
|
```
|
|
|
|
## Day-to-day commands
|
|
|
|
```bash
|
|
make vergabe-status # pods, svc, ingress, certificate
|
|
make vergabe-logs # tail app logs
|
|
make vergabe-dry-run # helm template render (audit values)
|
|
make vergabe-deploy # helm upgrade --install (idempotent)
|
|
make vergabe-migrate # manage.py migrate against live deploy
|
|
make vergabe-seed # seed_dev — DEV ONLY, creates max.muster/testpass123 (do not run in prod)
|
|
make vergabe-superuser # interactive createsuperuser
|
|
```
|
|
|
|
## Promoting a new image tag
|
|
|
|
1. Build + push via Forgejo Actions (`.forgejo/workflows/image.yaml` on push to
|
|
`main`) or locally:
|
|
`docker login forgejo.coulomb.social` then build/push
|
|
`forgejo.coulomb.social/coulomb/vergabe-teilnahme:<sha>`.
|
|
2. Set `VERGABE_IMAGE_TAG=<sha> make vergabe-deploy` (values default repository
|
|
is `forgejo.coulomb.social/coulomb/vergabe-teilnahme`).
|
|
3. Helm rolls a new ReplicaSet with zero-downtime (`maxSurge: 1, maxUnavailable: 0`).
|
|
4. Verify via `make vergabe-status` and an HTTPS probe.
|
|
5. If migrations are needed, run `make vergabe-migrate` after the
|
|
rollout completes.
|
|
|
|
## Rollback
|
|
|
|
```bash
|
|
helm history vergabe-teilnahme -n vergabe-teilnahme
|
|
helm rollback vergabe-teilnahme <REVISION> -n vergabe-teilnahme
|
|
```
|
|
|
|
Rollback does **not** unwind DB migrations. For any rollback that
|
|
crosses a migration boundary, plan a `manage.py migrate <app> <name>`
|
|
reverse step explicitly.
|
|
|
|
## Troubleshooting
|
|
|
|
### Pod stuck `Running` 0/1, kube-probe failing
|
|
|
|
Most likely the probe's `Host` header doesn't match
|
|
`ALLOWED_HOSTS`. The chart sets `probes.hostHeader:
|
|
vergabe-teilnahme.whywhynot.de` precisely to avoid this — if you
|
|
change `ALLOWED_HOSTS` in values, also update `probes.hostHeader`.
|
|
Symptom in `kubectl logs`: kube-probe requests returning HTTP 400.
|
|
See `docs/django-on-railiance.md` for the reusable pattern.
|
|
|
|
### `dj-database-url` error: "The database name 'XYZ...' is longer than 63 characters"
|
|
|
|
The `DATABASE_URL` password isn't URL-encoded. See the rotation
|
|
recipe above. Tracked in `RAILIANCE-WP-0004 I01`.
|
|
|
|
### Cert-manager: cert stuck in `False`
|
|
|
|
Check the Order/Challenge resources:
|
|
```bash
|
|
kubectl get order,challenge -n vergabe-teilnahme
|
|
kubectl describe challenge -n vergabe-teilnahme
|
|
```
|
|
Common causes: DNS not yet propagated to all resolvers, Let's
|
|
Encrypt rate-limited, or the ingress controller isn't forwarding
|
|
`/.well-known/acme-challenge/` requests.
|
|
|
|
### `make vergabe-status` shows certificate `False`
|
|
|
|
The chart leaves cert lifecycle to cert-manager. If the cert renews
|
|
fail, cert-manager keeps serving the old cert until it expires.
|
|
Investigate with `kubectl describe certificate vergabe-teilnahme-tls
|
|
-n vergabe-teilnahme`.
|
|
|
|
## Data durability and restore readiness
|
|
|
|
`vergabe_db` lives on the shared `apps-pg` CNPG cluster owned by
|
|
`railiance-platform`. S5 owns the app release runbook and post-restore app
|
|
checks; platform owns the database backup and restore mechanism.
|
|
|
|
**Phase 1 gate (2026-07-12): satisfied.** Platform logical backup lane and an
|
|
isolated restore drill are evidenced under `RAILIANCE-WP-0013`:
|
|
|
|
| Artifact | Reference |
|
|
| --- | --- |
|
|
| Backup uploads | `docs/evidence/apps-pg-backup-lane-20260711.json` |
|
|
| Restore drill | `docs/evidence/apps-pg-restore-drill-20260711T230725Z.json` |
|
|
|
|
Operator commands:
|
|
|
|
```bash
|
|
make apps-pg-backup # encrypt + upload vergabe_db (+ sibling DBs)
|
|
make apps-pg-restore-drill # isolated restore + row-count gate
|
|
make cnpg-backup-status # barman ScheduledBackup posture (still degraded)
|
|
```
|
|
|
|
CNPG `ScheduledBackup` remains platform follow-up; Phase 1 logical dumps satisfy
|
|
the interim gate in `docs/app-data-backup-restore-handoff.md`. Before treating
|
|
data as fully production-critical beyond Phase 1, complete post-restore migration
|
|
smoke, HTTPS health check, and a representative workflow verification.
|
|
|
|
Break-glass manual dump (not the durable contract):
|
|
|
|
```bash
|
|
kubectl exec -n databases apps-pg-1 -- pg_dump -U postgres -Fc vergabe_db > vergabe_db-$(date +%F).dump
|
|
```
|
|
|
|
## Deferred for v1
|
|
|
|
- Multi-replica HA (`replicaCount: 1`).
|
|
- Media-upload PVC (`persistence.media.enabled: false` — Django
|
|
`MEDIA_ROOT` is in-pod ephemeral).
|
|
- 3-stage canary (the Staged Promotion Lifecycle workstream is still
|
|
0/7).
|
|
- SSO / Keycloak integration (Django built-in auth only).
|
|
- Celery + Redis workers.
|
|
|
|
## Cross-references
|
|
|
|
- Workplan: `workplans/railiance-apps-WP-0002-vergabe-teilnahme-on-railiance01.md`
|
|
- Improvements backlog: `workplans/railiance-apps-WP-0004-app-deployment-improvements.md`
|
|
- Shared DB cluster: `railiance-platform/docs/apps-pg.md`
|
|
- Container registry: `/home/worsch/railiance-forge/docs/gitea-container-registry.md`
|
|
- Python package registry: `/home/worsch/railiance-forge/docs/gitea-package-registry.md`
|
|
- S5 app onboarding checklist: `docs/s5-app-onboarding-checklist.md`
|
|
- App data backup handoff: `docs/app-data-backup-restore-handoff.md`
|
|
- Manifest dry-run prerequisites: `docs/manifest-server-dry-run.md`
|
|
- Django deployment recipe: `docs/django-on-railiance.md`
|
|
- Operator setup: `docs/operator-setup.md`
|
|
- Operator recipes: `docs/operator-recipes.md`
|
|
- App source: https://forgejo.coulomb.social/coulomb/vergabe-teilnahme
|