han-app/VM2_services/codebase/services/deployment/RUNBOOK.md

VM2 Processing deployment runbook

This directory is the independent VM2 foundation in the VM2_services repository. It does not deploy VM1 or VM1_app/codebase/backend. All commands below are operator commands; repository creation does not execute them.

Repository root: HAN_chat_specification/VM2_services. Compose and deployment artifacts live under VM2_services/codebase/services/. Local operator commands assume the current directory is VM2_services unless stated otherwise.

The full step-by-step procedure with gates and copy-paste commands is in RUNBOOK.ru.md.

Production blockers before first start

1. Replace every .env placeholder with reviewed non-secret values. Keep BITRIX_SYNC_ENABLED=false until migrations, grants, portal fields, robot contracts and cutover are signed off. Build and push application images to the registry first (blocker 2).
2. Fill every *_IMAGE variable with a reviewed registry digest. Root Compose rejects missing image references; mutable tags are not production evidence.
3. Install production files as root:root; deploy must not be in docker and must not be able to write Compose, units, helpers, allow-lists or secret mappings.
4. Populate separate reviewed active CIDR files from the two .template files. Their committed active versions are intentionally deny all.
5. Provision public ACME material under host /etc/letsencrypt and the managed PostgreSQL CA under /etc/han/ca. Provision an internal-CA certificate whose SAN matches the private VM2 name. Permit host port 8443 only from VM1 SG and, when needed, approved private/VPN ops CIDRs.
6. Create a dedicated VM2 Selectel IAM principal. It may read only names in deployment/secrets/config.example.json. Never reuse the VM1 principal.
7. REDIS_SAFETY_ACL is the complete ACL file, not merely a password. It must expose unauthenticated PING only for health and a password-protected safety user limited to required han:safety:* keys/commands. The password in MESSAGE_SAFETY_REDIS_URL must match. Start from redis/redis-safety.acl.template, replace both placeholders, and add the exporter user limited to PING/INFO. Its ACL password must exactly match the separate REDIS_EXPORTER_PASSWORD secret. That secret is a JSON password map, {"redis://redis-safety:6379":"<THE_SAME_PASSWORD>"}, not a raw password string. Never commit either password.
8. Provision distinct runtime and migration DB credentials. MESSAGE_SAFETY_CONFIG_ADMIN_DATABASE_URL may migrate/activate policy while MESSAGE_SAFETY_DATABASE_URL cannot; BITRIX_SYNC_MIGRATION_DATABASE_URL owns DDL while BITRIX_SYNC_DATABASE_URL is the least-privilege runtime role. Migration credentials are mounted only into the ops profile jobs.
9. The setup script leaves UFW egress open for bootstrap. Before production, constrain egress through Selectel SG/NAT/proxy to the approved PostgreSQL, S3, Secrets Manager, Bitrix24, DNS/NTP and SigNoz destinations. Host KESL receives only update-source egress approved by deployment/kesl/RUNBOOK.KESL.ru.md. Registry/package access exists only during controlled maintenance windows.

Who runs what

• Operator workstation: builds the release archive and transfers it to VM2. Local examples use PowerShell from VM2_services.
• root on VM2: host bootstrap, verified release activation, root-owned files, .env, secret mapping, credentials, TLS/allow-lists, migrations and first start.
• deploy on VM2: accepts releases only in /var/lib/han-deploy/incoming, checks status/logs and runs installed fixed systemd operations through exact sudo rules. deploy must not run docker, edit /opt/han-chat/services or join the docker group.
• admin on VM2: personal break-glass role with a separate SSH key and local sudo password. Not used for routine deploy; not in docker/lxd; every login and sudo call is an incident operation.

Prerequisites (before §1)

This runbook covers operations on an already provisioned VM2. Prepare outside Compose first:

1. Selectel infrastructure — VPC/subnet, SG (public 80/443/22; private 8443 only; default-deny egress after bootstrap), sizing (4 vCPU / 8 GB RAM / 80 GB SSD — see module-10-deployment-vm2.md), public and private VM2 IPs, DNS A record for PROCESSING_PUBLIC_HOST.
2. Managed PostgreSQL — schemas/roles for message_safety and bitrix_sync, separate migration/runtime DSNs; see arch-10-deployment.md §6.
3. Images — build and push han-message-safety, han-bitrix-sync; record immutable digests for every *_IMAGE in .env.example (nginx, redis, otel-collector, Redis exporter and nginx exporter). clamd/freshclam are absent from Compose; KESL 12.4 standalone and its broker run on the host.
4. Selectel Secrets Manager — populate all remote names from deployment/secrets/config.example.json (DSNs, tokens, S3 read-only keys, REDIS_SAFETY_ACL, REDIS_EXPORTER_PASSWORD, internal TLS PEM for 8443). Dedicated VM2 IAM principal with read-only access to those names only.
5. S3 quarantine bucket and SigNoz OTLP endpoint — non-secret values in .env. Current self-hosted SigNoz accepts private plaintext OTLP without authentication; do not provision a fake auth secret or mandatory non-empty header.
6. Internal TLS — internal-CA certificate with SAN = VM2 private DNS; PEM stored in Secrets Manager, not in the release tree.

Section order: §1–§5 → Gates 1–9 → §7 (post-acceptance). Run systemctl enable and systemctl start han-processing.service only after Gate 5 succeeds.

1. Bootstrap a fresh VM2

From VM2_services on the operator workstation, copy the setup script:

scp -i C:\Users\MI\.ssh\hansel `
  .\codebase\services\deployment\scripts\setup-vm.sh `
  root@<VM2_PUBLIC_IP>:/root/setup-vm2.sh

Bootstrap as root with separate Ed25519 deploy/admin keys, VM1_PRIVATE_CIDRS, optional OPS_CIDRS, admin sudo password, deploy/admin login verification, then rerun with HARDEN_SSH=true. Full commands: RUNBOOK.ru.md §1.

2. Release transfer under deploy

From VM2_services:

$Release = "<VERSION_OR_GIT_SHA>"
tar --exclude=services/.env `
  --exclude='services/**/__pycache__' `
  --exclude='services/**/.pytest_cache' `
  --exclude='services/**/.ruff_cache' `
  -czf "vm2-services-$Release.tar.gz" -C .\codebase services
Get-FileHash "vm2-services-$Release.tar.gz" -Algorithm SHA256
scp -i C:\Users\MI\.ssh\hansel "vm2-services-$Release.tar.gz" `
  deploy@<VM2_PUBLIC_IP>:/var/lib/han-deploy/incoming/

deploy verifies SHA-256 and archive listing only; it must not unpack into production. Details: RUNBOOK.ru.md §2–§3.

3. Activation and root-owned install

root verifies the archive, extracts to staging, rsyncs into /opt/han-chat/services, reruns setup-vm.sh to install helpers/units. Details: RUNBOOK.ru.md §3.

4. Non-secret config and Selectel

Copy .env.example → .env, install loader config as /etc/han/secrets/vm2-<APP_ENV>.selectel.json, encrypt Selectel service-user password with systemd-creds, edit nginx allow-lists. Details: RUNBOOK.ru.md §4.

The local Collector receives logs directly over OTLP (no filelog), scrapes only itself plus the Redis/nginx exporters, and collects host metrics through read-only /hostfs. Exporters and nginx stub_status use internal networks and expose only; they have no host-published ports.

5. PostgreSQL CA and initial public TLS

Install managed PostgreSQL CA under /etc/han/ca, issue Let's Encrypt cert for PROCESSING_PUBLIC_HOST, stage public cert/key for nginx. Details: RUNBOOK.ru.md §5.

6. Gates 1–9: preflight, migrations and first start

Execute gates in order 1 → 2 → 3 → 4 → 5 → 6 → 7 → 8 → 9. Do not enable han-processing.service until Gate 5 completes successfully.

Before Message Safety starts in Gate 5, complete the operator KESL runbook, then enable/start han-kesl-scan-broker.socket and verify KESL/socket status plus root:han-message-safety 0660 on /run/han-kesl/scan.sock. Only then start Message Safety. The broker is a custom integration: exact output/exit semantics of kesl-control --scan-file --action Inform, cleanup and throughput must pass on the target VM2.

Gate	Purpose
1	Secrets materialized via han-secrets-vm2.service
2	Static preflight and resolved Compose (@sha256: images)
3	Migrations, initial Message Safety config create/activate
4	nginx -t with TLS and upstream placeholders
5	Ordered compose up; then systemctl enable/start han-processing
6	Host ports, public/private certificate chains, certbot renewal
7	Public routing smoke (308/404/426/405)
8	Private Safety/sync API from VM1/ops only
9	Fake-token canary — no secrets in logs/traces

After Gate 5, deploy may run:

sudo systemctl restart han-secrets-vm2.service
sudo systemctl restart han-processing.service
sudo systemctl --no-pager status han-processing.service
sudo journalctl --no-pager -u han-processing.service

Full gate commands: RUNBOOK.ru.md §6.

Do not open webhook traffic while bitrix-sync is disabled. A disabled or failed receiver must return retryable 503/closed routing, never successful 2xx ignored.

7. After Gate 9 — post-acceptance

Gate 9 completes VM2 technical acceptance but does not authorize Message Safety cutover on VM1 or Bitrix sync enablement.

1. Verify autostart: docker, han-chat-vm2-docker-firewall, han-secrets-vm2, han-processing, certbot.timer.
2. Reboot-gate: systemctl reboot, then repeat autostart checks and Gates 6–8 briefly.
3. Record release evidence: han-vm2-compose config --images, ps, certbot timer, unit journals — without secret values.
4. Configure operational monitoring (unhealthy/restart/OOM, TLS expiry, KESL version/database date, hourly update, broker/socket status, OTEL queue, disk/RAM, MOCK mode, private Safety API).
5. Proceed to controlled Message Safety cutover on VM1 — see module-10-deployment-vm2.md §13.
6. Keep BITRIX_SYNC_ENABLED=false and BITRIX_SYNC_MODE=disabled; Bitrix public allow-list remains deny all; until module-07-bitrix-sync.md cutover gates are signed off.

Details: RUNBOOK.ru.md §7.

Failure policy

• Safety dependency failure is fail-closed: VM1 must not send/promote content.
• A stale/unavailable KESL database or broker error disables file capability only; scan errors remain retryable and eventually return 503, never allow.
• Redis loss may remove acceleration but PostgreSQL remains authoritative.
• OTEL outage queues within the bounded volume and must not change verdicts.
• Rollback does not downgrade schemas, delete durable tasks/mappings, or run docker compose down -v.

Emergency MOCK

Only these five sudo commands are allowed:

han-message-safety-mode standard
han-message-safety-mode mock --text-free true --file-free true
han-message-safety-mode mock --text-free true --file-free false
han-message-safety-mode mock --text-free false --file-free true
han-message-safety-mode mock --text-free false --file-free false

The helper atomically writes only /etc/han-chat/message-safety-mode.env, recreates only the Safety API, checks health, and restores the previous mode on failure. MOCK has no timeout: keep a high-severity alert active until explicit standard, then verify normal text/link/file capabilities and an EICAR canary.

Host KESL and broker

KESL 12.4 standalone and the root-owned fail-closed broker are not Compose images. The Message Safety worker receives only /run/han-kesl/scan.sock, not kesl-control, the Docker socket or host-root access. Runtime records scanner_engine=kesl; signatures_version is the hash of KESL version plus database date. Installation, hourly database updates, socket permissions, clean/EICAR/error/stale gates and rollback follow deployment/kesl/RUNBOOK.KESL.ru.md.