han-app/codebase/backend/deployment/RUNBOOK.md

HAN Chat production-like deployment runbook

This is the executable checklist for the single-VM contour. PostgreSQL and S3 are managed external services. Never use docker compose down -v, an Alembic downgrade, or a mutable image tag during deployment.

Gate 0 — decisions and ownership

• Release SHA/digests, maintenance window, on-call and rollback owner recorded.
• RPO/RTO accepted; initial targets are PG RPO <=15 minutes and RTO <=4 hours.
• Remote OTLP backend selected, or debug-only acceptance limitation accepted.
• Mock OTP, Safety stub and bitrix-sync stub risks explicitly accepted.

Gate 1 — VPC, DNS and security groups

• Managed PostgreSQL has only a private endpoint and accepts traffic from VM SG.
• Internet can reach only VM TCP 80/443; SSH is restricted to VPN/ops CIDR.
• Ports 6379, 4317/4318, 8000, 8080 and 9000 are denied externally.
• DNS A for PUBLIC_HOST points at the VM and outbound HTTPS is available.

Gate 2 — VM hardening

On a fresh Ubuntu 24.04 VM, run:

sudo deployment/scripts/setup-vm.sh

Before setting HARDEN_SSH=true, verify key-based access in a separate SSH session. The script header documents its parameters and safe defaults.

• Ubuntu 24.04, NTP, unattended security updates and disk alerts are active.
• Key-only deploy account works in a second session; root/password SSH is off.
• UFW/cloud SG and DOCKER-USER policy survive reboot.
• Docker Engine and Compose support include and long-form env_file.

Gate 3 — managed PostgreSQL

• Daily backup, PITR, deletion protection, encryption and alerts are enabled.
• Provider CA is installed at PG_CA_HOST_PATH; all DSNs use verify-full.
• Schemas han_app, bitrix_local, bitrix_sync, message_safety, keycloak have separate migration/runtime roles with tested negative grants.
• Migration tested against an empty DB and a clone of the previous release.

Gate 4 — Selectel S3

• Quarantine, attachments and documents buckets are private and encrypted.
• API credentials are prefix-scoped; Safety credentials are quarantine read-only.
• Browser CORS permits exact HTTPS origin and PUT headers only.
• Quarantine lifecycle exceeds Safety poll/recovery; data retention is approved.

Gate 5 — immutable release

• Checkout is detached at the approved SHA and working tree is clean.
• Service images are immutable and scanned; no unresolved critical/high issue.
• Root docker-compose.yml is the only deployment entry point.

Gate 6 — environment and secrets

umask 077
cp .env.example .env
chmod 600 .env
# Replace placeholders using a protected editor/secret manager.
./scripts/validate-env .env
docker compose --env-file .env config --quiet

• Token pairs match, PG verifies TLS, public URLs are HTTPS.
• Mock OTP risk is accepted and all secrets are unique >=128-bit values.
• NOTIFICATIONS_TOKEN_PRODUCER_TEST is unique and supplied only through secret/env; the producer_test source seed stores only its hash.
• FRONTEND_DEV_PROXY_ENABLED=false and Safety/nginx timeout budgets match.

Gate 7 — images and static frontend

docker compose --env-file .env pull
docker compose --env-file .env build --pull frontend-static nginx redis
docker compose --env-file .env run --rm frontend-static

• Frontend export was tested/scanned and copied by frontend-static into its named volume.
• Build artifacts contain no secrets or unintended source maps.
• At least 30% VM disk remains free.

Gate 8 — topology

docker compose --env-file .env config --services
python3 -m unittest discover -s tests -v

• Exactly nginx publishes 80:80 and 443:443; no PostgreSQL service exists.
• Redis AOF/RDB/ACL and OTEL persistent queue volumes are present.
• backend and observability are internal networks.

Gate 9 — ACME/TLS bootstrap

Set NGINX_TLS_ENABLED=false only for this bootstrap command:

NGINX_TLS_ENABLED=false docker compose --env-file .env up -d nginx
docker compose --profile certbot run --rm certbot certonly \
  --webroot -w /var/www/certbot -d "$PUBLIC_HOST" \
  --cert-name "$PUBLIC_HOST" --email "$ACME_EMAIL" \
  --agree-tos --no-eff-email --non-interactive
docker compose --env-file .env up -d --force-recreate nginx
docker compose exec -T nginx nginx -t -c /tmp/nginx.conf

First rehearse with Certbot --staging. Install a twice-daily systemd timer for deployment/scripts/ssl-renew.sh; test certbot renew --dry-run. Enable HSTS only after chain, hostname, redirect and TLS 1.2/1.3 checks pass.

Gate 10 — migrations and seed

Create a provider PITR marker, then:

PITR_MARKER_CONFIRMED=true deployment/scripts/migrate.sh
deployment/scripts/seed.sh

• Expected Alembic revisions are active and runtime users did not perform DDL.
• Seed succeeds twice and mandatory settings contain no secret.
• Schema remains backward-compatible with the previous images.

Gate 11 — Keycloak

docker compose up -d keycloak
docker compose ps keycloak

• Discovery/JWKS issuer is the exact public /auth HTTPS URL.
• Frontend client is public PKCE S256; implicit/password/social flows are off.
• Wrong/replayed OTP and limits fail safely; settings bridge is fail-closed.
• Bootstrap admin was removed/rotated and named admin MFA is enabled.

Gate 12 — ordered startup and readiness

docker compose up -d redis
docker compose up -d keycloak otel-collector
docker compose up -d message-safety
docker compose up -d api-backend
docker compose up -d delivery-worker safety-recovery-worker cleanup-worker \
  notification-expire-worker notification-draft-cleanup-worker
docker compose up -d bitrix-local-app bitrix-sync
docker compose up -d nginx
docker compose up -d --wait api-backend keycloak sms-service bitrix-local-app
docker compose exec -T nginx nginx -t -c /tmp/nginx.conf
docker compose kill -s HUP nginx
docker compose ps

Nginx resolves Docker upstream names when its configuration is loaded. After recreating api-backend, keycloak, sms-service, or bitrix-local-app, wait for readiness, validate the active /tmp/nginx.conf, and signal the master process with HUP as shown above. Do not use bare nginx -t or nginx -s reload: they target the default config/PID under read-only /var/run, not the running Nginx instance.

• No restart loop/OOM; critical readiness is green.
• notification-expire-worker runs daily closure with an advisory lock; notification-draft-cleanup-worker removes expired drafts/S3 objects. Both entrypoints exist in the installed image.
• Only documented Bitrix not-installed/sync-stub degradation remains.
• External /internal/* is 404 and OTEL accepts telemetry.

Gate 13 — Bitrix24

• Install, handler and placement URLs use the exact public HTTPS paths.
• Connector han_mobile_app is active on Open Line 8; events are bound once.
• OAuth is encrypted; callback/application/service tokens never enter logs.
• Outbound and operator reply paths are idempotent; internal status is private.

Gate 14 — smoke and E2E

deployment/scripts/smoke.sh

• Guest, OTP/PKCE/bootstrap/session, refresh and logout paths pass.
• Safety allow/deny/pending/timeout and one concurrent slow poll pass.
• File quarantine/promote/delete, owner-only download and audit pass.
• WS reconnect plus REST reconciliation, ownership 404, idempotency and 429 pass.
• Closed-network producer_test Create/Cancel smoke passes; identical Create returns 200, changed payload returns 409, and the external internal route returns 404.
• Expire advisory locking and first download of any linked document are verified; hiding is one-time and an existing date_expired is preserved.
• Logs contain no PII, message body, token or presigned query.

Gate 15 — observability

• Known request ID links nginx, API and downstream trace; UX ID is not a label.
• Three signals reach the selected backend; SLO queries and alerts are tested.
• Remote outage fills/drains the bounded persistent queue without business outage.
• Secret/PII canary is absent. Collector restart/drop/refused metrics are checked.

For local acceptance only, start the redacted debug collector with: docker compose --profile observability-local up -d otel-collector-local.

Gate 16 — open traffic

• Gates 0–15 are signed; fresh backup/PITR evidence and previous images exist.
• HSTS is enabled, release digests/schema/realm versions are recorded.
• No active page; on-call and product owner accept stub limitations.
• Observe 5xx/auth/delivery/DB/Redis/OOM/OTEL queue/Bitrix for 60 minutes.

Backup and restore

Provider backup/PITR is authoritative. A supplemental verified logical dump:

PG_BACKUP_DSN='postgresql://...?...sslmode=verify-full&sslrootcert=...' \
  deployment/scripts/backup.sh /opt/han-chat/backups

Quarterly, restore PG and S3 into an isolated VPC, deploy the same image digests, do not route production DNS/Bitrix callbacks, run smoke, and record measured RPO/RTO. Redis may be restored empty; its AOF/RDB is not a business backup.

Rollback

Only roll back to images compatible with the current schema:

SCHEMA_BACKWARD_COMPATIBLE_CONFIRMED=true \
  deployment/scripts/rollback.sh /secure/path/previous-release.env
ENV_FILE=/secure/path/previous-release.env deployment/scripts/smoke.sh

Real SMS rollout addendum

This runbook remains mock-only until module-11 artifacts exist. An SMS release requires schema/role sms, versioned migrations and an active approved auth_otp seed, sms-service/worker, the exact callback route, paired service tokens, Direct TOKEN_1, approved sender/template, separate callback credentials, a reconfirmed callback source IP, and a static worker egress IP.

Order: App DB OTP seed → SMS schema/migrations/seed → mock Direct tests → production SMS deployment while Keycloak remains in mock mode → Keycloak expand migration/SPI → controlled provider smoke plus callback/redaction evidence → real mode. Roll back by restoring mock mode without deleting the journal/schema; stop new real orders and drain or record in-flight/uncertain rows. Downgrade only with proven schema compatibility.

Never run Alembic downgrade. After a backward-incompatible migration choose a forward fix or coordinated PITR/S3/Bitrix reconciliation under maintenance. Always verify outbox/inbox/recovery so an ambiguous message is not sent twice.