mi
217 lines
8.2 KiB
Markdown
217 lines
8.2 KiB
Markdown
# HAN Chat production-like deployment runbook
|
||
|
||
This is the executable checklist for the single-VM contour. PostgreSQL and S3
|
||
are managed external services. Never use `docker compose down -v`, an Alembic
|
||
downgrade, or a mutable image tag during deployment.
|
||
|
||
## Gate 0 — decisions and ownership
|
||
|
||
- [ ] Release SHA/digests, maintenance window, on-call and rollback owner recorded.
|
||
- [ ] RPO/RTO accepted; initial targets are PG RPO <=15 minutes and RTO <=4 hours.
|
||
- [ ] Remote OTLP backend selected, or debug-only acceptance limitation accepted.
|
||
- [ ] Mock OTP, Safety stub and bitrix-sync stub risks explicitly accepted.
|
||
|
||
## Gate 1 — VPC, DNS and security groups
|
||
|
||
- [ ] Managed PostgreSQL has only a private endpoint and accepts traffic from VM SG.
|
||
- [ ] Internet can reach only VM TCP 80/443; SSH is restricted to VPN/ops CIDR.
|
||
- [ ] Ports 6379, 4317/4318, 8000, 8080 and 9000 are denied externally.
|
||
- [ ] DNS `A` for `PUBLIC_HOST` points at the VM and outbound HTTPS is available.
|
||
|
||
## Gate 2 — VM hardening
|
||
|
||
On a fresh Ubuntu 24.04 VM, run:
|
||
|
||
```sh
|
||
sudo deployment/scripts/setup-vm.sh
|
||
```
|
||
|
||
Before setting `HARDEN_SSH=true`, verify key-based access in a separate SSH
|
||
session. The script header documents its parameters and safe defaults.
|
||
|
||
- [ ] Ubuntu 24.04, NTP, unattended security updates and disk alerts are active.
|
||
- [ ] Key-only deploy account works in a second session; root/password SSH is off.
|
||
- [ ] UFW/cloud SG and `DOCKER-USER` policy survive reboot.
|
||
- [ ] Docker Engine and Compose support `include` and long-form `env_file`.
|
||
|
||
## Gate 3 — managed PostgreSQL
|
||
|
||
- [ ] Daily backup, PITR, deletion protection, encryption and alerts are enabled.
|
||
- [ ] Provider CA is installed at `PG_CA_HOST_PATH`; all DSNs use `verify-full`.
|
||
- [ ] Schemas `han_app`, `bitrix_local`, `bitrix_sync`, `message_safety`, `keycloak`
|
||
have separate migration/runtime roles with tested negative grants.
|
||
- [ ] Migration tested against an empty DB and a clone of the previous release.
|
||
|
||
## Gate 4 — Selectel S3
|
||
|
||
- [ ] Quarantine, attachments and documents buckets are private and encrypted.
|
||
- [ ] API credentials are prefix-scoped; Safety credentials are quarantine read-only.
|
||
- [ ] Browser CORS permits exact HTTPS origin and PUT headers only.
|
||
- [ ] Quarantine lifecycle exceeds Safety poll/recovery; data retention is approved.
|
||
|
||
## Gate 5 — immutable release
|
||
|
||
- [ ] Checkout is detached at the approved SHA and working tree is clean.
|
||
- [ ] Service images are immutable and scanned; no unresolved critical/high issue.
|
||
- [ ] Root `docker-compose.yml` is the only deployment entry point.
|
||
|
||
## Gate 6 — environment and secrets
|
||
|
||
```sh
|
||
umask 077
|
||
cp .env.example .env
|
||
chmod 600 .env
|
||
# Replace placeholders using a protected editor/secret manager.
|
||
./scripts/validate-env .env
|
||
docker compose --env-file .env config --quiet
|
||
```
|
||
|
||
- [ ] Token pairs match, PG verifies TLS, public URLs are HTTPS.
|
||
- [ ] Mock OTP risk is accepted and all secrets are unique >=128-bit values.
|
||
- [ ] `FRONTEND_DEV_PROXY_ENABLED=false` and Safety/nginx timeout budgets match.
|
||
|
||
## Gate 7 — images and static frontend
|
||
|
||
```sh
|
||
docker compose --env-file .env pull
|
||
docker compose --env-file .env build --pull frontend-static nginx redis
|
||
docker compose --env-file .env run --rm frontend-static
|
||
```
|
||
|
||
- [ ] Frontend export was tested/scanned and copied by `frontend-static` into its named volume.
|
||
- [ ] Build artifacts contain no secrets or unintended source maps.
|
||
- [ ] At least 30% VM disk remains free.
|
||
|
||
## Gate 8 — topology
|
||
|
||
```sh
|
||
docker compose --env-file .env config --services
|
||
python3 -m unittest discover -s tests -v
|
||
```
|
||
|
||
- [ ] Exactly nginx publishes `80:80` and `443:443`; no PostgreSQL service exists.
|
||
- [ ] Redis AOF/RDB/ACL and OTEL persistent queue volumes are present.
|
||
- [ ] `backend` and `observability` are internal networks.
|
||
|
||
## Gate 9 — ACME/TLS bootstrap
|
||
|
||
Set `NGINX_TLS_ENABLED=false` only for this bootstrap command:
|
||
|
||
```sh
|
||
NGINX_TLS_ENABLED=false docker compose --env-file .env up -d nginx
|
||
docker compose --profile certbot run --rm certbot certonly \
|
||
--webroot -w /var/www/certbot -d "$PUBLIC_HOST" \
|
||
--cert-name "$PUBLIC_HOST" --email "$ACME_EMAIL" \
|
||
--agree-tos --no-eff-email --non-interactive
|
||
docker compose --env-file .env up -d --force-recreate nginx
|
||
docker compose exec -T nginx nginx -t -c /tmp/nginx.conf
|
||
```
|
||
|
||
First rehearse with Certbot `--staging`. Install a twice-daily systemd timer for
|
||
`deployment/scripts/ssl-renew.sh`; test `certbot renew --dry-run`. Enable HSTS
|
||
only after chain, hostname, redirect and TLS 1.2/1.3 checks pass.
|
||
|
||
## Gate 10 — migrations and seed
|
||
|
||
Create a provider PITR marker, then:
|
||
|
||
```sh
|
||
PITR_MARKER_CONFIRMED=true deployment/scripts/migrate.sh
|
||
deployment/scripts/seed.sh
|
||
```
|
||
|
||
- [ ] Expected Alembic revisions are active and runtime users did not perform DDL.
|
||
- [ ] Seed succeeds twice and mandatory settings contain no secret.
|
||
- [ ] Schema remains backward-compatible with the previous images.
|
||
|
||
## Gate 11 — Keycloak
|
||
|
||
```sh
|
||
docker compose up -d keycloak
|
||
docker compose ps keycloak
|
||
```
|
||
|
||
- [ ] Discovery/JWKS issuer is the exact public `/auth` HTTPS URL.
|
||
- [ ] Frontend client is public PKCE S256; implicit/password/social flows are off.
|
||
- [ ] Wrong/replayed OTP and limits fail safely; settings bridge is fail-closed.
|
||
- [ ] Bootstrap admin was removed/rotated and named admin MFA is enabled.
|
||
|
||
## Gate 12 — ordered startup and readiness
|
||
|
||
```sh
|
||
docker compose up -d redis
|
||
docker compose up -d keycloak otel-collector
|
||
docker compose up -d message-safety
|
||
docker compose up -d api-backend
|
||
docker compose up -d bitrix-local-app bitrix-sync
|
||
docker compose up -d nginx
|
||
docker compose ps
|
||
```
|
||
|
||
- [ ] No restart loop/OOM; critical readiness is green.
|
||
- [ ] Only documented Bitrix not-installed/sync-stub degradation remains.
|
||
- [ ] External `/internal/*` is 404 and OTEL accepts telemetry.
|
||
|
||
## Gate 13 — Bitrix24
|
||
|
||
- [ ] Install, handler and placement URLs use the exact public HTTPS paths.
|
||
- [ ] Connector `han_mobile_app` is active on Open Line 8; events are bound once.
|
||
- [ ] OAuth is encrypted; callback/application/service tokens never enter logs.
|
||
- [ ] Outbound and operator reply paths are idempotent; internal status is private.
|
||
|
||
## Gate 14 — smoke and E2E
|
||
|
||
```sh
|
||
deployment/scripts/smoke.sh
|
||
```
|
||
|
||
- [ ] Guest, OTP/PKCE/bootstrap/session, refresh and logout paths pass.
|
||
- [ ] Safety allow/deny/pending/timeout and one concurrent slow poll pass.
|
||
- [ ] File quarantine/promote/delete, owner-only download and audit pass.
|
||
- [ ] WS reconnect plus REST reconciliation, ownership 404, idempotency and 429 pass.
|
||
- [ ] Logs contain no PII, message body, token or presigned query.
|
||
|
||
## Gate 15 — observability
|
||
|
||
- [ ] Known request ID links nginx, API and downstream trace; UX ID is not a label.
|
||
- [ ] Three signals reach the selected backend; SLO queries and alerts are tested.
|
||
- [ ] Remote outage fills/drains the bounded persistent queue without business outage.
|
||
- [ ] Secret/PII canary is absent. Collector restart/drop/refused metrics are checked.
|
||
|
||
For local acceptance only, start the redacted debug collector with:
|
||
`docker compose --profile observability-local up -d otel-collector-local`.
|
||
|
||
## Gate 16 — open traffic
|
||
|
||
- [ ] Gates 0–15 are signed; fresh backup/PITR evidence and previous images exist.
|
||
- [ ] HSTS is enabled, release digests/schema/realm versions are recorded.
|
||
- [ ] No active page; on-call and product owner accept stub limitations.
|
||
- [ ] Observe 5xx/auth/delivery/DB/Redis/OOM/OTEL queue/Bitrix for 60 minutes.
|
||
|
||
## Backup and restore
|
||
|
||
Provider backup/PITR is authoritative. A supplemental verified logical dump:
|
||
|
||
```sh
|
||
PG_BACKUP_DSN='postgresql://...?...sslmode=verify-full&sslrootcert=...' \
|
||
deployment/scripts/backup.sh /opt/han-chat/backups
|
||
```
|
||
|
||
Quarterly, restore PG and S3 into an isolated VPC, deploy the same image digests,
|
||
do not route production DNS/Bitrix callbacks, run smoke, and record measured RPO/RTO.
|
||
Redis may be restored empty; its AOF/RDB is not a business backup.
|
||
|
||
## Rollback
|
||
|
||
Only roll back to images compatible with the current schema:
|
||
|
||
```sh
|
||
SCHEMA_BACKWARD_COMPATIBLE_CONFIRMED=true \
|
||
deployment/scripts/rollback.sh /secure/path/previous-release.env
|
||
ENV_FILE=/secure/path/previous-release.env deployment/scripts/smoke.sh
|
||
```
|
||
|
||
Never run Alembic downgrade. After a backward-incompatible migration choose a
|
||
forward fix or coordinated PITR/S3/Bitrix reconciliation under maintenance.
|
||
Always verify outbox/inbox/recovery so an ambiguous message is not sent twice.
|