VM2 Processing deployment runbook

This directory is the independent VM2 foundation. It does not deploy VM1 or codebase/backend. All commands below are operator commands; repository creation does not execute them.

Production blockers before first start

Replace every .env placeholder with reviewed non-secret values. Keep BITRIX_SYNC_ENABLED=false until migrations, grants, portal fields, robot contracts and cutover are signed off.
Fill every *_IMAGE variable with a reviewed registry digest. Root Compose rejects missing image references; mutable tags are not production evidence.
Install production files as root:root; deploy must not be in docker and must not be able to write Compose, units, helpers, allow-lists or secret mappings.
Populate separate reviewed active CIDR files from the two .template files. Their committed active versions are intentionally deny all.
Provision public ACME material under host /etc/letsencrypt and the managed PostgreSQL CA under /etc/han/ca. Provision an internal-CA certificate whose SAN matches the private VM2 name. Permit host port 8443 only from VM1 SG and, when needed, approved private/VPN ops CIDRs.
Create a dedicated VM2 Selectel IAM principal. It may read only names in deployment/secrets/config.example.json. Never reuse the VM1 principal.
REDIS_SAFETY_ACL is the complete ACL file, not merely a password. It must expose unauthenticated PING only for health and a password-protected safety user limited to required han:safety:* keys/commands. The password in MESSAGE_SAFETY_REDIS_URL must match. Start from redis/redis-safety.acl.template, replace REPLACE_WITH_LONG_RANDOM_PASSWORD, and never commit the password.
Provision distinct runtime and migration DB credentials. MESSAGE_SAFETY_CONFIG_ADMIN_DATABASE_URL may migrate/activate policy while MESSAGE_SAFETY_DATABASE_URL cannot; BITRIX_SYNC_MIGRATION_DATABASE_URL owns DDL while BITRIX_SYNC_DATABASE_URL is the least-privilege runtime role. Migration credentials are mounted only into the ops profile jobs.
The setup script leaves UFW egress open for bootstrap. Before production, constrain egress through Selectel SG/NAT/proxy to the approved PostgreSQL, S3, Secrets Manager, Bitrix24, DNS/NTP, SigNoz and ClamAV destinations. Registry/package access exists only during controlled maintenance windows.

Install

Bootstrap a fresh Ubuntu 24.04 VM as root with deployment/scripts/setup-vm.sh, supplying VM1_PRIVATE_CIDRS, optional private/VPN OPS_CIDRS, and separate Ed25519 public-key files for deploy and break-glass admin. SSH is publicly reachable but key-only and protected by fail2ban; the CIDR variables apply only to private port 8443. The script installs host packages/firewalls and roles but never starts Compose. Set a separate admin sudo password; verify deploy login, admin login and admin sudo in independent sessions before rerunning with HARDEN_SSH=true. Root/deploy/admin key reuse is rejected.
Checkout an immutable release under /opt/han-chat/services.
Copy .env.example to root-owned mode 0600 .env.
Install secrets_loader.py and han-secrets under /usr/local/lib/han-secrets-vm2/, root-owned and non-writable.
Install han-compose as /usr/local/sbin/han-vm2-compose.
Install han-secrets-vm2.service and han-processing.service under /etc/systemd/system/.
Install han-message-safety-mode as root-owned 0755 and the sudoers template as /etc/sudoers.d/deploy-message-safety-mode mode 0440; validate with visudo -cf. Create the dedicated host group han-message-safety with GID 10001. Before the first Compose validation, create /etc/han-chat/message-safety-mode.env as root:han-message-safety 0640 with all three flags false (or invoke the helper's standard transition after the fixed launcher is installed).
Install loader config using the exact APP_ENV suffix. With the committed example (APP_ENV=production-like) the path is /etc/han/secrets/vm2-production-like.selectel.json mode 0600. For controlled no-provider recovery use an explicit file config pointing to a root-only 0700 directory containing exactly one file per configured key. Selectel failure never falls back automatically.

Preflight and startup

Run deployment/preflight.sh first. Then, through the approved root units:

synchronize secrets; any missing/oversized/invalid secret blocks startup;
validate resolved Compose without storing its output;
run the two ops migration jobs and create/activate the reviewed initial Message Safety config before starting either runtime;
validate nginx config and both certificate chains;
start Redis/Collector, ClamAV, application API/workers, then nginx;
verify that only nginx publishes 80, 443, and private-bound 8443;
verify all non-exact public paths return 404, HTTP webhook paths return 426 without redirect/query reflection, wrong methods fail, and wrong source CIDRs are rejected before upstream;
verify private Safety check/task/status and sync status only from approved callers; verify public /internal/* is 404;
canary telemetry with a fake token marker and prove query, form body, Authorization, DSN, S3 key and object key are absent from logs/traces.

Do not open webhook traffic while bitrix-sync is disabled. A disabled or failed receiver must return retryable 503/closed routing, never successful 2xx ignored.

Failure policy

Safety dependency failure is fail-closed: VM1 must not send/promote content.
Stale/unavailable ClamAV signatures disable file capability only; they never convert a scan error to allow.
Redis loss may remove acceleration but PostgreSQL remains authoritative.
OTEL outage queues within the bounded volume and must not change verdicts.
Rollback does not downgrade schemas, delete durable tasks/mappings, or run docker compose down -v.

Emergency MOCK

Only these five sudo commands are allowed:

han-message-safety-mode standard
han-message-safety-mode mock --text-free true --file-free true
han-message-safety-mode mock --text-free true --file-free false
han-message-safety-mode mock --text-free false --file-free true
han-message-safety-mode mock --text-free false --file-free false

The helper atomically writes only /etc/han-chat/message-safety-mode.env, recreates only the Safety API, checks health, and restores the previous mode on failure. MOCK has no timeout: keep a high-severity alert active until explicit standard, then verify normal text/link/file capabilities and an EICAR canary.

Known image exceptions

ClamAV images may require UID/path adjustments after validating the exact digest. Do not weaken read_only, capabilities or mounts globally: document the smallest writable signature/runtime paths and compensate with network and resource limits. freshclam alone receives signature-CDN egress; clamd receives none.

6.9 KiB Raw Blame History