han-app/VM2_services/codebase/services/deployment/RUNBOOK.md

VM2 Processing deployment runbook

This directory is the independent VM2 foundation in the VM2_services repository. It does not deploy VM1 or VM1_app/codebase/backend. All commands below are operator commands; repository creation does not execute them.

Repository root: HAN_chat_specification/VM2_services. Compose and deployment artifacts live under VM2_services/codebase/services/. Local operator commands assume the current directory is VM2_services unless stated otherwise.

The full step-by-step procedure with gates and copy-paste commands is in RUNBOOK.ru.md.

Production blockers before first start

1. Replace every .env placeholder with reviewed non-secret values. Keep BITRIX_SYNC_ENABLED=false until migrations, grants, portal fields, robot contracts and cutover are signed off. Build and push application images to the registry first (blocker 2).
2. Fill every *_IMAGE variable with a reviewed registry digest. Root Compose rejects missing image references; mutable tags are not production evidence.
3. Install production files as root:root; deploy must not be in docker and must not be able to write Compose, units, helpers, allow-lists or secret mappings.
4. Populate separate reviewed active CIDR files from the two .template files. Their committed active versions are intentionally deny all.
5. Provision public ACME material under host /etc/letsencrypt and the managed PostgreSQL CA under /etc/han/ca. Provision an internal-CA certificate whose SAN matches the private VM2 name. Permit host port 8443 only from VM1 SG and, when needed, approved private/VPN ops CIDRs.
6. Create a dedicated VM2 Selectel IAM principal. It may read only names in deployment/secrets/config.example.json. Never reuse the VM1 principal.
7. REDIS_SAFETY_ACL is the complete ACL file, not merely a password. It must expose unauthenticated PING only for health and a password-protected safety user limited to required han:safety:* keys/commands. The password in MESSAGE_SAFETY_REDIS_URL must match. Start from redis/redis-safety.acl.template, replace REPLACE_WITH_LONG_RANDOM_PASSWORD, and never commit the password.
8. Provision distinct runtime and migration DB credentials. MESSAGE_SAFETY_CONFIG_ADMIN_DATABASE_URL may migrate/activate policy while MESSAGE_SAFETY_DATABASE_URL cannot; BITRIX_SYNC_MIGRATION_DATABASE_URL owns DDL while BITRIX_SYNC_DATABASE_URL is the least-privilege runtime role. Migration credentials are mounted only into the ops profile jobs.
9. The setup script leaves UFW egress open for bootstrap. Before production, constrain egress through Selectel SG/NAT/proxy to the approved PostgreSQL, S3, Secrets Manager, Bitrix24, DNS/NTP, SigNoz and ClamAV destinations. Registry/package access exists only during controlled maintenance windows.

Who runs what

• Operator workstation: builds the release archive and transfers it to VM2. Local examples use PowerShell from VM2_services.
• root on VM2: host bootstrap, verified release activation, root-owned files, .env, secret mapping, credentials, TLS/allow-lists, migrations and first start.
• deploy on VM2: accepts releases only in /var/lib/han-deploy/incoming, checks status/logs and runs installed fixed systemd operations through exact sudo rules. deploy must not run docker, edit /opt/han-chat/services or join the docker group.
• admin on VM2: personal break-glass role with a separate SSH key and local sudo password. Not used for routine deploy; not in docker/lxd; every login and sudo call is an incident operation.

Prerequisites (before §1)

This runbook covers operations on an already provisioned VM2. Prepare outside Compose first:

1. Selectel infrastructure — VPC/subnet, SG (public 80/443/22; private 8443 only; default-deny egress after bootstrap), sizing (4 vCPU / 8 GB RAM / 80 GB SSD — see module-10-deployment-vm2.md), public and private VM2 IPs, DNS A record for PROCESSING_PUBLIC_HOST.
2. Managed PostgreSQL — schemas/roles for message_safety and bitrix_sync, separate migration/runtime DSNs; see arch-10-deployment.md §6.
3. Images — build and push han-message-safety, han-bitrix-sync; record immutable digests for every *_IMAGE in .env.example (nginx, redis, clamav, otel-collector).
4. Selectel Secrets Manager — populate all remote names from deployment/secrets/config.example.json (DSNs, tokens, S3 read-only keys, REDIS_SAFETY_ACL, internal TLS PEM for 8443). Dedicated VM2 IAM principal with read-only access to those names only.
5. S3 quarantine bucket and SigNoz OTLP endpoint — non-secret values in .env.
6. Internal TLS — internal-CA certificate with SAN = VM2 private DNS; PEM stored in Secrets Manager, not in the release tree.

Section order: §1–§5 → Gates 1–9 → §7 (post-acceptance). Run systemctl enable and systemctl start han-processing.service only after Gate 5 succeeds.

1. Bootstrap a fresh VM2

From VM2_services on the operator workstation, copy the setup script:

scp -i C:\Users\MI\.ssh\hansel `
  .\codebase\services\deployment\scripts\setup-vm.sh `
  root@<VM2_PUBLIC_IP>:/root/setup-vm2.sh

Bootstrap as root with separate Ed25519 deploy/admin keys, VM1_PRIVATE_CIDRS, optional OPS_CIDRS, admin sudo password, deploy/admin login verification, then rerun with HARDEN_SSH=true. Full commands: RUNBOOK.ru.md §1.

2. Release transfer under deploy

From VM2_services:

$Release = "<VERSION_OR_GIT_SHA>"
tar --exclude=services/.env `
  --exclude='services/**/__pycache__' `
  --exclude='services/**/.pytest_cache' `
  --exclude='services/**/.ruff_cache' `
  -czf "vm2-services-$Release.tar.gz" -C .\codebase services
Get-FileHash "vm2-services-$Release.tar.gz" -Algorithm SHA256
scp -i C:\Users\MI\.ssh\hansel "vm2-services-$Release.tar.gz" `
  deploy@<VM2_PUBLIC_IP>:/var/lib/han-deploy/incoming/

deploy verifies SHA-256 and archive listing only; it must not unpack into production. Details: RUNBOOK.ru.md §2–§3.

3. Activation and root-owned install

root verifies the archive, extracts to staging, rsyncs into /opt/han-chat/services, reruns setup-vm.sh to install helpers/units. Details: RUNBOOK.ru.md §3.

4. Non-secret config and Selectel

Copy .env.example → .env, install loader config as /etc/han/secrets/vm2-<APP_ENV>.selectel.json, encrypt Selectel service-user password with systemd-creds, edit nginx allow-lists. Details: RUNBOOK.ru.md §4.

5. PostgreSQL CA and initial public TLS

Install managed PostgreSQL CA under /etc/han/ca, issue Let's Encrypt cert for PROCESSING_PUBLIC_HOST, stage public cert/key for nginx. Details: RUNBOOK.ru.md §5.

6. Gates 1–9: preflight, migrations and first start

Execute gates in order 1 → 2 → 3 → 4 → 5 → 6 → 7 → 8 → 9. Do not enable han-processing.service until Gate 5 completes successfully.

Gate	Purpose
1	Secrets materialized via han-secrets-vm2.service
2	Static preflight and resolved Compose (@sha256: images)
3	Migrations, initial Message Safety config create/activate
4	nginx -t with TLS and upstream placeholders
5	Ordered compose up; then systemctl enable/start han-processing
6	Host ports, public/private certificate chains, certbot renewal
7	Public routing smoke (308/404/426/405)
8	Private Safety/sync API from VM1/ops only
9	Fake-token canary — no secrets in logs/traces

After Gate 5, deploy may run:

sudo systemctl restart han-secrets-vm2.service
sudo systemctl restart han-processing.service
sudo systemctl --no-pager status han-processing.service
sudo journalctl --no-pager -u han-processing.service

Full gate commands: RUNBOOK.ru.md §6.

Do not open webhook traffic while bitrix-sync is disabled. A disabled or failed receiver must return retryable 503/closed routing, never successful 2xx ignored.

7. After Gate 9 — post-acceptance

Gate 9 completes VM2 technical acceptance but does not authorize Message Safety cutover on VM1 or Bitrix sync enablement.

1. Verify autostart: docker, han-chat-vm2-docker-firewall, han-secrets-vm2, han-processing, certbot.timer.
2. Reboot-gate: systemctl reboot, then repeat autostart checks and Gates 6–8 briefly.
3. Record release evidence: han-vm2-compose config --images, ps, certbot timer, unit journals — without secret values.
4. Configure operational monitoring (unhealthy/restart/OOM, TLS expiry, ClamAV signature age, OTEL queue, disk/RAM, MOCK mode, private Safety API).
5. Proceed to controlled Message Safety cutover on VM1 — see module-10-deployment-vm2.md §13.
6. Keep BITRIX_SYNC_ENABLED=false and BITRIX_SYNC_MODE=disabled; Bitrix public allow-list remains deny all; until module-07-bitrix-sync.md cutover gates are signed off.

Details: RUNBOOK.ru.md §7.

Failure policy

• Safety dependency failure is fail-closed: VM1 must not send/promote content.
• Stale/unavailable ClamAV signatures disable file capability only; they never convert a scan error to allow.
• Redis loss may remove acceleration but PostgreSQL remains authoritative.
• OTEL outage queues within the bounded volume and must not change verdicts.
• Rollback does not downgrade schemas, delete durable tasks/mappings, or run docker compose down -v.

Emergency MOCK

Only these five sudo commands are allowed:

han-message-safety-mode standard
han-message-safety-mode mock --text-free true --file-free true
han-message-safety-mode mock --text-free true --file-free false
han-message-safety-mode mock --text-free false --file-free true
han-message-safety-mode mock --text-free false --file-free false

The helper atomically writes only /etc/han-chat/message-safety-mode.env, recreates only the Safety API, checks health, and restores the previous mode on failure. MOCK has no timeout: keep a high-severity alert active until explicit standard, then verify normal text/link/file capabilities and an EICAR canary.

Known image exceptions

ClamAV images may require UID/path adjustments after validating the exact digest. Do not weaken read_only, capabilities or mounts globally: document the smallest writable signature/runtime paths and compensate with network and resource limits. freshclam alone receives signature-CDN egress; clamd receives none.