# VM2 Processing deployment runbook

This directory is the independent VM2 foundation. It does not deploy VM1 or
`codebase/backend`. All commands below are operator commands; repository
creation does not execute them.

## Production blockers before first start

1. Replace every `.env` placeholder with reviewed non-secret values. Keep
   `BITRIX_SYNC_ENABLED=false` until migrations, grants, portal fields, robot
   contracts and cutover are signed off.
2. Fill every `*_IMAGE` variable with a reviewed registry digest. Root Compose
   rejects missing image references; mutable tags are not production evidence.
3. Install production files as `root:root`; `deploy` must not be in `docker`
   and must not be able to write Compose, units, helpers, allow-lists or secret
   mappings.
4. Populate separate reviewed active CIDR files from the two `.template`
   files. Their committed active versions are intentionally `deny all`.
5. Provision public ACME material under host `/etc/letsencrypt` and the managed
   PostgreSQL CA under `/etc/han/ca`. Provision an internal-CA certificate whose
   SAN matches the private VM2 name. Permit host port `8443` only from VM1 SG
   and, when needed, approved private/VPN ops CIDRs.
6. Create a dedicated VM2 Selectel IAM principal. It may read only names in
   `deployment/secrets/config.example.json`. Never reuse the VM1 principal.
7. `REDIS_SAFETY_ACL` is the complete ACL file, not merely a password. It must
   expose unauthenticated `PING` only for health and a password-protected
   `safety` user limited to required `han:safety:*` keys/commands. The password
   in `MESSAGE_SAFETY_REDIS_URL` must match. Start from
   `redis/redis-safety.acl.template`, replace
   `REPLACE_WITH_LONG_RANDOM_PASSWORD`, and never commit the password.
8. Provision distinct runtime and migration DB credentials.
   `MESSAGE_SAFETY_CONFIG_ADMIN_DATABASE_URL` may migrate/activate policy while
   `MESSAGE_SAFETY_DATABASE_URL` cannot; `BITRIX_SYNC_MIGRATION_DATABASE_URL`
   owns DDL while `BITRIX_SYNC_DATABASE_URL` is the least-privilege runtime
   role. Migration credentials are mounted only into the `ops` profile jobs.
9. The setup script leaves UFW egress open for bootstrap. Before production,
   constrain egress through Selectel SG/NAT/proxy to the approved PostgreSQL,
   S3, Secrets Manager, Bitrix24, DNS/NTP, SigNoz and ClamAV destinations.
   Registry/package access exists only during controlled maintenance windows.

## Install

- Bootstrap a fresh Ubuntu 24.04 VM as root with
  `deployment/scripts/setup-vm.sh`, supplying `VM1_PRIVATE_CIDRS`, optional
  private/VPN `OPS_CIDRS`, and separate Ed25519 public-key files for deploy and
  break-glass admin. SSH is publicly reachable but key-only and protected by
  fail2ban; the CIDR variables apply only to private port `8443`. The script
  installs host packages/firewalls and roles but never starts Compose. Set a
  separate admin sudo password; verify deploy login, admin login and admin sudo
  in independent sessions before rerunning with `HARDEN_SSH=true`.
  Root/deploy/admin key reuse is rejected.
- Checkout an immutable release under `/opt/han-chat/services`.
- Copy `.env.example` to root-owned mode `0600` `.env`.
- Install `secrets_loader.py` and `han-secrets` under
  `/usr/local/lib/han-secrets-vm2/`, root-owned and non-writable.
- Install `han-compose` as `/usr/local/sbin/han-vm2-compose`.
- Install `han-secrets-vm2.service` and `han-processing.service` under
  `/etc/systemd/system/`.
- Install `han-message-safety-mode` as root-owned `0755` and the sudoers
  template as `/etc/sudoers.d/deploy-message-safety-mode` mode `0440`; validate
  with `visudo -cf`. Create the dedicated host group `han-message-safety` with
  GID `10001`. Before the first Compose validation, create
  `/etc/han-chat/message-safety-mode.env` as
  `root:han-message-safety 0640` with all three flags `false` (or invoke the
  helper's `standard` transition after the fixed launcher is installed).
- Install loader config using the exact `APP_ENV` suffix. With the committed
  example (`APP_ENV=production-like`) the path is
  `/etc/han/secrets/vm2-production-like.selectel.json` mode `0600`. For
  controlled no-provider recovery use an explicit `file`
  config pointing to a root-only `0700` directory containing exactly one file
  per configured key. Selectel failure never falls back automatically.

## Preflight and startup

Run `deployment/preflight.sh` first. Then, through the approved root units:

1. synchronize secrets; any missing/oversized/invalid secret blocks startup;
2. validate resolved Compose without storing its output;
3. run the two `ops` migration jobs and create/activate the reviewed initial
   Message Safety config before starting either runtime;
4. validate nginx config and both certificate chains;
5. start Redis/Collector, ClamAV, application API/workers, then nginx;
6. verify that only nginx publishes `80`, `443`, and private-bound `8443`;
7. verify all non-exact public paths return `404`, HTTP webhook paths return
   `426` without redirect/query reflection, wrong methods fail, and wrong
   source CIDRs are rejected before upstream;
8. verify private Safety check/task/status and sync status only from approved
   callers; verify public `/internal/*` is `404`;
9. canary telemetry with a fake token marker and prove query, form body,
   Authorization, DSN, S3 key and object key are absent from logs/traces.

Do not open webhook traffic while `bitrix-sync` is disabled. A disabled or
failed receiver must return retryable `503`/closed routing, never successful
`2xx ignored`.

## Failure policy

- Safety dependency failure is fail-closed: VM1 must not send/promote content.
- Stale/unavailable ClamAV signatures disable file capability only; they never
  convert a scan error to allow.
- Redis loss may remove acceleration but PostgreSQL remains authoritative.
- OTEL outage queues within the bounded volume and must not change verdicts.
- Rollback does not downgrade schemas, delete durable tasks/mappings, or run
  `docker compose down -v`.

## Emergency MOCK

Only these five sudo commands are allowed:

```text
han-message-safety-mode standard
han-message-safety-mode mock --text-free true --file-free true
han-message-safety-mode mock --text-free true --file-free false
han-message-safety-mode mock --text-free false --file-free true
han-message-safety-mode mock --text-free false --file-free false
```

The helper atomically writes only
`/etc/han-chat/message-safety-mode.env`, recreates only the Safety API, checks
health, and restores the previous mode on failure. MOCK has no timeout: keep a
high-severity alert active until explicit `standard`, then verify normal
text/link/file capabilities and an EICAR canary.

## Known image exceptions

ClamAV images may require UID/path adjustments after validating the exact
digest. Do not weaken `read_only`, capabilities or mounts globally: document
the smallest writable signature/runtime paths and compensate with network and
resource limits. `freshclam` alone receives signature-CDN egress; `clamd`
receives none.