han-app/VM1_app/codebase/backend/deployment/RUNBOOK.md

# HAN Chat production-like deployment runbook

This is the executable checklist for the single-VM contour. PostgreSQL and S3
are managed external services. Never use `docker compose down -v`, an Alembic
downgrade, or a mutable image tag during deployment.

## VM2 Processing is a separate host

Do not run this backend/VM1 setup script on VM2. VM2 has its own bootstrap:
`codebase/services/deployment/scripts/setup-vm.sh`, and its authoritative
operator checklist is `codebase/services/deployment/RUNBOOK.ru.md`.

The VM2 ownership boundary is intentionally different from the legacy VM1
script: `deploy` is **not** a member of the `docker` group. Root owns
`/opt/han-chat/services`, Compose, units, helpers, `.env`, allow-lists and
secret mappings. Deploy may write only to `/var/lib/han-deploy/incoming` and
may invoke exact systemd/safety-mode commands installed in sudoers.
The separate `admin` account is break-glass only: it has its own Ed25519 key
and a separate local sudo password. Root, deploy and admin keys must differ.

Initial VM2 bootstrap commands:

```sh
# Local operator workstation: upload only the reviewed setup script.
scp codebase/services/deployment/scripts/setup-vm.sh \
  root@<VM2_PUBLIC_IP>:/root/setup-vm2.sh

# VM2 root: install host packages/roles/firewalls; this does not start Compose.
chmod 0700 /root/setup-vm2.sh
DEPLOY_AUTHORIZED_KEY_FILE=/root/bootstrap/deploy.pub \
ADMIN_AUTHORIZED_KEY_FILE=/root/bootstrap/admin.pub \
OPS_CIDRS='<OPS_PUBLIC_IP>/32' \
VM1_PRIVATE_CIDRS='<VM1_PRIVATE_IP>/32' \
/root/setup-vm2.sh
```

Generate and upload the two public keys before this command; never copy the
root key into either account. Set the admin sudo password with `passwd admin`.
Keep the root session open and verify both key-based logins plus `sudo -v` as
admin in separate sessions. Only then rerun as VM2 root with
`HARDEN_SSH=true SKIP_APT_UPGRADE=true` to disable direct root SSH.

Release transfer is performed as deploy, while activation and installation
remain root operations:

```sh
# deploy: receive and inspect only.
cd /var/lib/han-deploy/incoming
sha256sum vm2-services-<RELEASE>.tar.gz
tar -tzf vm2-services-<RELEASE>.tar.gz

# root: verify the operator-provided digest, activate root-owned files,
# then rerun setup-vm.sh so it installs fixed helpers and systemd units.
printf '%s  %s\n' '<EXPECTED_SHA256>' \
  /var/lib/han-deploy/incoming/vm2-services-<RELEASE>.tar.gz | sha256sum --check -
ARCHIVE=/var/lib/han-deploy/incoming/vm2-services-<RELEASE>.tar.gz
if tar -tzf "$ARCHIVE" | grep -Eq '(^/|(^|/)\.\.(/|$)|^services/\.env$)'; then exit 1; fi
if tar -tzf "$ARCHIVE" | grep -Ev '^services(/|$)' | grep -q .; then exit 1; fi
if tar -tvzf "$ARCHIVE" | awk '$1 ~ /^[lh]/ {found=1} END {exit !found}'; then exit 1; fi
STAGING="$(mktemp -d /opt/han-chat/.vm2-release.XXXXXX)"
tar -xzf "$ARCHIVE" \
  -C "$STAGING" --no-same-owner --no-same-permissions
test -f "$STAGING/services/docker-compose.yml"
rsync -a --delete --exclude=.env --chown=root:root --chmod=D755,F644 \
  "$STAGING/services/" /opt/han-chat/services/
rm -rf -- "$STAGING"
OPS_CIDRS='<OPS_PUBLIC_IP>/32' \
VM1_PRIVATE_CIDRS='<VM1_PRIVATE_IP>/32' \
DEPLOY_AUTHORIZED_KEY_FILE=/root/bootstrap/deploy.pub \
ADMIN_AUTHORIZED_KEY_FILE=/root/bootstrap/admin.pub \
HARDEN_SSH=true SKIP_APT_UPGRADE=true \
/root/setup-vm2.sh
```

After root configures `.env`, Selectel encrypted credentials, loader mapping,
TLS and CIDR allow-lists, root synchronizes secrets, runs preflight/migrations
and performs the first start. Subsequent routine operations available to
deploy are limited to:

```sh
sudo systemctl restart han-secrets-vm2.service
sudo systemctl restart han-processing.service
sudo systemctl --no-pager status han-processing.service
sudo journalctl --no-pager -u han-processing.service
```

Exact archive activation, file installation, credential creation, migration
and first-start commands are documented in the VM2 Russian runbook referenced
above. They must not be replaced with direct Docker access for deploy.

## Gate 0 — decisions and ownership

- [ ] Release SHA/digests, maintenance window, on-call and rollback owner recorded.
- [ ] RPO/RTO accepted; initial targets are PG RPO <=15 minutes and RTO <=4 hours.
- [ ] Remote OTLP backend selected, or debug-only acceptance limitation accepted.
- [ ] Mock OTP, Safety stub and bitrix-sync stub risks explicitly accepted.

## Gate 1 — VPC, DNS and security groups

- [ ] Managed PostgreSQL has only a private endpoint and accepts traffic from VM SG.
- [ ] Internet can reach only VM TCP 80/443; SSH is restricted to VPN/ops CIDR.
- [ ] Ports 6379, 4317/4318, 8000, 8080 and 9000 are denied externally.
- [ ] DNS `A` for `PUBLIC_HOST` points at the VM and outbound HTTPS is available.

## Gate 2 — VM hardening

On a fresh Ubuntu 24.04 VM, run:

```sh
sudo deployment/scripts/setup-vm.sh
```

The script disables password SSH and X11 forwarding by default, then locks the
local `root` and `deploy` passwords after checking authorized keys. Before
setting `HARDEN_SSH=true`, which also disables root login and TCP forwarding,
verify key-based deploy access in a separate SSH session.

- [ ] Ubuntu 24.04, NTP, unattended security updates and disk alerts are active.
- [ ] Key-only deploy account works in a second session; root/password SSH is off.
- [ ] UFW/cloud SG and `DOCKER-USER` policy survive reboot.
- [ ] Docker Engine and Compose support `include` and long-form `env_file`.

## Gate 3 — managed PostgreSQL

- [ ] Daily backup, PITR, deletion protection, encryption and alerts are enabled.
- [ ] Provider CA is installed at `PG_CA_HOST_PATH`; all DSNs use `verify-full`.
- [ ] Schemas `han_app`, `bitrix_local`, `bitrix_sync`, `message_safety`, `keycloak`
      have separate migration/runtime roles with tested negative grants.
- [ ] Migration tested against an empty DB and a clone of the previous release.

## Gate 4 — Selectel S3

- [ ] Quarantine, attachments and documents buckets are private and encrypted.
- [ ] API credentials are prefix-scoped; Safety credentials are quarantine read-only.
- [ ] Browser CORS permits exact HTTPS origin and PUT headers only.
- [ ] Quarantine lifecycle exceeds Safety poll/recovery; data retention is approved.

## Gate 5 — immutable release

- [ ] Checkout is detached at the approved SHA and working tree is clean.
- [ ] Service images are immutable and scanned; no unresolved critical/high issue.
- [ ] Root `docker-compose.yml` is the only deployment entry point.

## Gate 6 — environment and secrets

```sh
cp .env.example .env
# Replace non-secret configuration placeholders only.
./scripts/validate-env .env
sudo systemctl restart han-secrets@production.service
sudo ./scripts/validate-env .env \
  --runtime-manifest /run/han-chat/secrets/manifest
sudo deployment/secrets/han-compose config --quiet
```

- [ ] `SECRETS_SOURCE=file|selectel`; `.env` contains no secret keys or credential-bearing DSNs.
- [ ] `deployment/secrets/han-secrets` sets `HAN_SECRETS_ACTIVE=1`, does not log values,
      and optionally exposes a paths-only `HAN_RUNTIME_SECRET_MANIFEST`.
- [ ] Runtime token pairs match, PG verifies TLS, public URLs are HTTPS.
- [ ] Mock OTP risk is accepted and runtime secrets are unique >=128-bit values.
- [ ] `NOTIFICATIONS_TOKEN_PRODUCER_TEST` is unique and supplied only through secret/env; the `producer_test` source seed stores only its hash.
- [ ] `FRONTEND_DEV_PROXY_ENABLED=false` and Safety/nginx timeout budgets match.

## Gate 7 — images and static frontend

```sh
docker compose --env-file .env pull
docker compose --env-file .env build --pull frontend-static nginx redis
docker compose --env-file .env run --rm frontend-static
```

- [ ] Frontend export was tested/scanned and copied by `frontend-static` into its named volume.
- [ ] Build artifacts contain no secrets or unintended source maps.
- [ ] At least 30% VM disk remains free.

## Gate 8 — topology

```sh
docker compose --env-file .env config --services
python3 -m unittest discover -s tests -v
```

- [ ] Exactly nginx publishes `80:80` and `443:443`; no PostgreSQL service exists.
- [ ] Redis AOF/RDB/ACL and OTEL persistent queue volumes are present.
- [ ] `backend` and `observability` are internal networks.

## Gate 9 — ACME/TLS bootstrap

Set `NGINX_TLS_ENABLED=false` only for this bootstrap command:

```sh
NGINX_TLS_ENABLED=false docker compose --env-file .env up -d nginx
docker compose --profile certbot run --rm certbot certonly \
  --webroot -w /var/www/certbot -d "$PUBLIC_HOST" \
  --cert-name "$PUBLIC_HOST" --email "$ACME_EMAIL" \
  --agree-tos --no-eff-email --non-interactive
docker compose --env-file .env up -d --force-recreate nginx
docker compose exec -T nginx nginx -t -c /tmp/nginx.conf
```

First rehearse with Certbot `--staging`. Install a twice-daily systemd timer for
`deployment/scripts/ssl-renew.sh`; test `certbot renew --dry-run`. Enable HSTS
only after chain, hostname, redirect and TLS 1.2/1.3 checks pass.

## Gate 10 — migrations and seed

Create a provider PITR marker, then:

```sh
PITR_MARKER_CONFIRMED=true deployment/scripts/migrate.sh
deployment/scripts/seed.sh
```

- [ ] Expected Alembic revisions are active and runtime users did not perform DDL.
- [ ] Seed succeeds twice and mandatory settings contain no secret.
- [ ] Schema remains backward-compatible with the previous images.

## Gate 11 — Keycloak

```sh
docker compose up -d keycloak
docker compose ps keycloak
```

- [ ] Discovery/JWKS issuer is the exact public `/auth` HTTPS URL.
- [ ] Frontend client is public PKCE S256; implicit/password/social flows are off.
- [ ] Wrong/replayed OTP and limits fail safely; settings bridge is fail-closed.
- [ ] Bootstrap admin was removed/rotated and named admin MFA is enabled.

## Gate 12 — ordered startup and readiness

```sh
docker compose up -d redis
docker compose up -d keycloak otel-collector
docker compose up -d message-safety
docker compose up -d api-backend
docker compose up -d delivery-worker safety-recovery-worker cleanup-worker \
  notification-expire-worker notification-draft-cleanup-worker
docker compose up -d bitrix-local-app bitrix-sync
docker compose up -d nginx
docker compose up -d --wait api-backend keycloak sms-service bitrix-local-app
docker compose exec -T nginx nginx -t -c /tmp/nginx.conf
docker compose kill -s HUP nginx
docker compose ps
```

Nginx resolves Docker upstream names when its configuration is loaded. After
recreating `api-backend`, `keycloak`, `sms-service`, or `bitrix-local-app`,
wait for readiness, validate the active `/tmp/nginx.conf`, and signal the
master process with `HUP` as shown above. Do not use bare `nginx -t` or
`nginx -s reload`: they target the default config/PID under read-only
`/var/run`, not the running Nginx instance.

- [ ] No restart loop/OOM; critical readiness is green.
- [ ] `notification-expire-worker` runs daily closure with an advisory lock; `notification-draft-cleanup-worker` removes expired drafts/S3 objects. Both entrypoints exist in the installed image.
- [ ] Only documented Bitrix not-installed/sync-stub degradation remains.
- [ ] External `/internal/*` is 404 and OTEL accepts telemetry.

## Gate 13 — Bitrix24

- [ ] Install, handler and placement URLs use the exact public HTTPS paths.
- [ ] Connector `han_mobile_app` is active on Open Line 8; events are bound once.
- [ ] OAuth is encrypted; callback/application/service tokens never enter logs.
- [ ] Outbound and operator reply paths are idempotent; internal status is private.

## Gate 14 — smoke and E2E

```sh
deployment/scripts/smoke.sh
```

- [ ] Guest, OTP/PKCE/bootstrap/session, refresh and logout paths pass.
- [ ] Safety allow/deny/pending/timeout and one concurrent slow poll pass.
- [ ] File quarantine/promote/delete, owner-only download and audit pass.
- [ ] WS reconnect plus REST reconciliation, ownership 404, idempotency and 429 pass.
- [ ] Closed-network `producer_test` Create/Cancel smoke passes; identical Create returns `200`, changed payload returns `409`, and the external internal route returns `404`.
- [ ] Expire advisory locking and first download of any linked document are verified; hiding is one-time and an existing `date_expired` is preserved.
- [ ] Logs contain no PII, message body, token or presigned query.

## Gate 15 — observability

- [ ] Known request ID links nginx, API and downstream trace; UX ID is not a label.
- [ ] Three signals reach the selected backend; SLO queries and alerts are tested.
- [ ] Remote outage fills/drains the bounded persistent queue without business outage.
- [ ] Secret/PII canary is absent. Collector restart/drop/refused metrics are checked.

For local acceptance only, start the redacted debug collector with:
`docker compose --profile observability-local up -d otel-collector-local`.

## Gate 16 — open traffic

- [ ] Gates 0–15 are signed; fresh backup/PITR evidence and previous images exist.
- [ ] HSTS is enabled, release digests/schema/realm versions are recorded.
- [ ] No active page; on-call and product owner accept stub limitations.
- [ ] Observe 5xx/auth/delivery/DB/Redis/OOM/OTEL queue/Bitrix for 60 minutes.

## Backup and restore

Provider backup/PITR is authoritative. A supplemental verified logical dump:

```sh
deployment/scripts/backup.sh /opt/han-chat/backups
```

Quarterly, restore PG and S3 into an isolated VPC, deploy the same image digests,
do not route production DNS/Bitrix callbacks, run smoke, and record measured RPO/RTO.
Redis may be restored empty; its AOF/RDB is not a business backup.

## Rollback

Only roll back to images compatible with the current schema:

```sh
SCHEMA_BACKWARD_COMPATIBLE_CONFIRMED=true \
  deployment/scripts/rollback.sh <PREVIOUS_IMMUTABLE_RELEASE>
deployment/scripts/smoke.sh
```

Rollback reuses the current runtime secret set and non-secret config. Do not
create or restore an environment snapshot.

## Real SMS rollout addendum

This runbook remains mock-only until module-11 artifacts exist. An SMS release requires schema/role `sms`, versioned migrations and an active approved `auth_otp` seed, `sms-service`/worker, the exact callback route, paired service tokens, Direct `TOKEN_1`, approved sender/template, separate callback credentials, a reconfirmed callback source IP, and a static worker egress IP.

Order: App DB OTP seed → SMS schema/migrations/seed → mock Direct tests → production SMS deployment while Keycloak remains in mock mode → Keycloak expand migration/SPI → controlled provider smoke plus callback/redaction evidence → real mode. Roll back by restoring mock mode without deleting the journal/schema; stop new real orders and drain or record in-flight/`uncertain` rows. Downgrade only with proven schema compatibility.

Never run Alembic downgrade. After a backward-incompatible migration choose a
forward fix or coordinated PITR/S3/Bitrix reconciliation under maintenance.
Always verify outbox/inbox/recovery so an ambiguous message is not sent twice.