han-app/VM2_services/codebase/services/deployment/RUNBOOK.md

# VM2 Processing deployment runbook

This directory is the independent VM2 foundation in the `VM2_services`
repository. It does not deploy VM1 or `VM1_app/codebase/backend`. All commands
below are operator commands; repository creation does not execute them.

Repository root: `HAN_chat_specification/VM2_services`. Compose and deployment
artifacts live under `VM2_services/codebase/services/`. Local operator commands
assume the current directory is `VM2_services` unless stated otherwise.

The full step-by-step procedure with gates and copy-paste commands is in
[`RUNBOOK.ru.md`](RUNBOOK.ru.md).

## Production blockers before first start

1. Replace every `.env` placeholder with reviewed non-secret values. Keep
   `BITRIX_SYNC_ENABLED=false` until migrations, grants, portal fields, robot
   contracts and cutover are signed off. Build and push application images to
   the registry first (blocker 2).
2. Fill every `*_IMAGE` variable with a reviewed registry digest. Root Compose
   rejects missing image references; mutable tags are not production evidence.
3. Install production files as `root:root`; `deploy` must not be in `docker`
   and must not be able to write Compose, units, helpers, allow-lists or secret
   mappings.
4. Populate separate reviewed active CIDR files from the two `.template`
   files. Their committed active versions are intentionally `deny all`.
5. Provision public ACME material under host `/etc/letsencrypt` and the managed
   PostgreSQL CA under `/etc/han/ca`. Provision an internal-CA certificate whose
   SAN matches the private VM2 name. Permit host port `8443` only from VM1 SG
   and, when needed, approved private/VPN ops CIDRs.
6. Create a dedicated VM2 Selectel IAM principal. It may read only names in
   `deployment/secrets/config.example.json`. Never reuse the VM1 principal.
7. `REDIS_SAFETY_ACL` is the complete ACL file, not merely a password. It must
   expose unauthenticated `PING` only for health and a password-protected
   `safety` user limited to required `han:safety:*` keys/commands. The password
   in `MESSAGE_SAFETY_REDIS_URL` must match. Start from
   `redis/redis-safety.acl.template`, replace both placeholders, and add the
   `exporter` user limited to `PING`/`INFO`. Its ACL password must exactly match
   the separate `REDIS_EXPORTER_PASSWORD` secret. That secret is a JSON password
   map, `{"redis://redis-safety:6379":"<THE_SAME_PASSWORD>"}`, not a raw password
   string. Never commit either password.
8. Provision distinct runtime and migration DB credentials.
   `MESSAGE_SAFETY_CONFIG_ADMIN_DATABASE_URL` may migrate/activate policy while
   `MESSAGE_SAFETY_DATABASE_URL` cannot; `BITRIX_SYNC_MIGRATION_DATABASE_URL`
   owns DDL while `BITRIX_SYNC_DATABASE_URL` is the least-privilege runtime
   role. Migration credentials are mounted only into the `ops` profile jobs.
9. The setup script leaves UFW egress open for bootstrap. Before production,
   constrain egress through Selectel SG/NAT/proxy to the approved PostgreSQL,
   S3, Secrets Manager, Bitrix24, DNS/NTP and SigNoz destinations. Host KESL
   receives only update-source egress approved by
   [`deployment/kesl/RUNBOOK.KESL.ru.md`](kesl/RUNBOOK.KESL.ru.md).
   Registry/package access exists only during controlled maintenance windows.

## Who runs what

- **Operator workstation:** builds the release archive and transfers it to VM2.
  Local examples use PowerShell from `VM2_services`.
- **`root` on VM2:** host bootstrap, verified release activation, root-owned
  files, `.env`, secret mapping, credentials, TLS/allow-lists, migrations and
  first start.
- **`deploy` on VM2:** accepts releases only in `/var/lib/han-deploy/incoming`,
  checks status/logs and runs installed fixed systemd operations through exact
  sudo rules. `deploy` must not run `docker`, edit `/opt/han-chat/services` or
  join the `docker` group.
- **`admin` on VM2:** personal break-glass role with a separate SSH key and
  local sudo password. Not used for routine deploy; not in `docker`/`lxd`; every
  login and sudo call is an incident operation.

## Prerequisites (before §1)

This runbook covers operations **on an already provisioned VM2**. Prepare outside
Compose first:

1. **Selectel infrastructure** — VPC/subnet, SG (public `80/443/22`; private
   `8443` only; default-deny egress after bootstrap), sizing (4 vCPU / 8 GB RAM /
   80 GB SSD — see
   [`module-10-deployment-vm2.md`](../../../documentation/module-10-deployment-vm2.md)),
   public and private VM2 IPs, DNS A record for `PROCESSING_PUBLIC_HOST`.
2. **Managed PostgreSQL** — schemas/roles for `message_safety` and
   `bitrix_sync`, separate migration/runtime DSNs; see
   [`arch-10-deployment.md`](../../../../architectory/arch-10-deployment.md) §6.
3. **Images** — build and push `han-message-safety`, `han-bitrix-sync`; record
   immutable digests for every `*_IMAGE` in `.env.example` (nginx, redis,
   otel-collector, Redis exporter and nginx exporter). `clamd`/`freshclam` are
   absent from Compose; KESL 12.4 standalone and its broker run on the host.
4. **Selectel Secrets Manager** — populate all remote names from
   `deployment/secrets/config.example.json` (DSNs, tokens, S3 read-only keys,
   `REDIS_SAFETY_ACL`, `REDIS_EXPORTER_PASSWORD`, internal TLS PEM for `8443`).
   Dedicated VM2 IAM principal with read-only access to those names only.
5. **S3 quarantine bucket** and SigNoz OTLP endpoint — non-secret values in
   `.env`. Current self-hosted SigNoz accepts private plaintext OTLP without
   authentication; do not provision a fake auth secret or mandatory non-empty
   header.
6. **Internal TLS** — internal-CA certificate with SAN = VM2 private DNS; PEM
   stored in Secrets Manager, not in the release tree.

Section order: §1–§5 → Gates 1–9 → §7 (post-acceptance). Run
`systemctl enable` and `systemctl start han-processing.service` **only after
Gate 5 succeeds**.

## 1. Bootstrap a fresh VM2

From `VM2_services` on the operator workstation, copy the setup script:

```powershell
scp -i C:\Users\MI\.ssh\hansel `
  .\codebase\services\deployment\scripts\setup-vm.sh `
  root@<VM2_PUBLIC_IP>:/root/setup-vm2.sh
```

Bootstrap as root with separate Ed25519 deploy/admin keys, `VM1_PRIVATE_CIDRS`,
optional `OPS_CIDRS`, admin sudo password, deploy/admin login verification, then
rerun with `HARDEN_SSH=true`. Full commands: [`RUNBOOK.ru.md`](RUNBOOK.ru.md) §1.

## 2. Release transfer under `deploy`

From `VM2_services`:

```powershell
$Release = "<VERSION_OR_GIT_SHA>"
tar --exclude=services/.env `
  --exclude='services/**/__pycache__' `
  --exclude='services/**/.pytest_cache' `
  --exclude='services/**/.ruff_cache' `
  -czf "vm2-services-$Release.tar.gz" -C .\codebase services
Get-FileHash "vm2-services-$Release.tar.gz" -Algorithm SHA256
scp -i C:\Users\MI\.ssh\hansel "vm2-services-$Release.tar.gz" `
  deploy@<VM2_PUBLIC_IP>:/var/lib/han-deploy/incoming/
```

`deploy` verifies SHA-256 and archive listing only; it must not unpack into
production. Details: [`RUNBOOK.ru.md`](RUNBOOK.ru.md) §2–§3.

## 3. Activation and root-owned install

`root` verifies the archive, extracts to staging, rsyncs into
`/opt/han-chat/services`, reruns `setup-vm.sh` to install helpers/units. Details:
[`RUNBOOK.ru.md`](RUNBOOK.ru.md) §3.

## 4. Non-secret config and Selectel

Copy `.env.example` → `.env`, install loader config as
`/etc/han/secrets/vm2-<APP_ENV>.selectel.json`, encrypt Selectel service-user
password with `systemd-creds`, edit nginx allow-lists. Details:
[`RUNBOOK.ru.md`](RUNBOOK.ru.md) §4.

The local Collector receives logs directly over OTLP (no `filelog`), scrapes
only itself plus the Redis/nginx exporters, and collects host metrics through
read-only `/hostfs`. Exporters and nginx `stub_status` use internal networks and
`expose` only; they have no host-published ports.

## 5. PostgreSQL CA and initial public TLS

Install managed PostgreSQL CA under `/etc/han/ca`, issue Let's Encrypt cert for
`PROCESSING_PUBLIC_HOST`, stage public cert/key for nginx. Details:
[`RUNBOOK.ru.md`](RUNBOOK.ru.md) §5.

## 6. Gates 1–9: preflight, migrations and first start

Execute gates **in order** 1 → 2 → 3 → 4 → 5 → 6 → 7 → 8 → 9. Do not enable
`han-processing.service` until Gate 5 completes successfully.

Before Message Safety starts in Gate 5, complete the operator KESL runbook,
then enable/start `han-kesl-scan-broker.socket` and verify KESL/socket status
plus `root:han-message-safety 0660` on `/run/han-kesl/scan.sock`. Only then
start Message Safety. The broker is a custom integration: exact output/exit
semantics of `kesl-control --scan-file --action Inform`, cleanup and throughput
must pass on the target VM2.

| Gate | Purpose |
| --- | --- |
| 1 | Secrets materialized via `han-secrets-vm2.service` |
| 2 | Static preflight and resolved Compose (`@sha256:` images) |
| 3 | Migrations, initial Message Safety config create/activate |
| 4 | nginx `-t` with TLS and upstream placeholders |
| 5 | Ordered `compose up`; then `systemctl enable/start han-processing` |
| 6 | Host ports, public/private certificate chains, certbot renewal |
| 7 | Public routing smoke (308/404/426/405) |
| 8 | Private Safety/sync API from VM1/ops only |
| 9 | Fake-token canary — no secrets in logs/traces |

After Gate 5, `deploy` may run:

```sh
sudo systemctl restart han-secrets-vm2.service
sudo systemctl restart han-processing.service
sudo systemctl --no-pager status han-processing.service
sudo journalctl --no-pager -u han-processing.service
```

Full gate commands: [`RUNBOOK.ru.md`](RUNBOOK.ru.md) §6.

Do not open webhook traffic while `bitrix-sync` is disabled. A disabled or
failed receiver must return retryable `503`/closed routing, never successful
`2xx ignored`.

## 7. After Gate 9 — post-acceptance

Gate 9 completes VM2 technical acceptance but **does not** authorize Message
Safety cutover on VM1 or Bitrix sync enablement.

1. Verify autostart: `docker`, `han-chat-vm2-docker-firewall`, `han-secrets-vm2`,
   `han-processing`, `certbot.timer`.
2. Reboot-gate: `systemctl reboot`, then repeat autostart checks and Gates 6–8
   briefly.
3. Record release evidence: `han-vm2-compose config --images`, `ps`, certbot
   timer, unit journals — without secret values.
4. Configure operational monitoring (unhealthy/restart/OOM, TLS expiry, KESL
   version/database date, hourly update, broker/socket status, OTEL queue,
   disk/RAM, MOCK mode, private Safety API).
5. Proceed to controlled Message Safety cutover on VM1 — see
   [`module-10-deployment-vm2.md`](../../../documentation/module-10-deployment-vm2.md) §13.
6. Keep `BITRIX_SYNC_ENABLED=false` and `BITRIX_SYNC_MODE=disabled`; Bitrix
   public allow-list remains `deny all;` until
   [`module-07-bitrix-sync.md`](../../../documentation/module-07-bitrix-sync.md)
   cutover gates are signed off.

Details: [`RUNBOOK.ru.md`](RUNBOOK.ru.md) §7.

## Failure policy

- Safety dependency failure is fail-closed: VM1 must not send/promote content.
- A stale/unavailable KESL database or broker error disables file capability
  only; scan errors remain retryable and eventually return `503`, never allow.
- Redis loss may remove acceleration but PostgreSQL remains authoritative.
- OTEL outage queues within the bounded volume and must not change verdicts.
- Rollback does not downgrade schemas, delete durable tasks/mappings, or run
  `docker compose down -v`.

## Emergency MOCK

Only these five sudo commands are allowed:

```text
han-message-safety-mode standard
han-message-safety-mode mock --text-free true --file-free true
han-message-safety-mode mock --text-free true --file-free false
han-message-safety-mode mock --text-free false --file-free true
han-message-safety-mode mock --text-free false --file-free false
```

The helper atomically writes only
`/etc/han-chat/message-safety-mode.env`, recreates only the Safety API, checks
health, and restores the previous mode on failure. MOCK has no timeout: keep a
high-severity alert active until explicit `standard`, then verify normal
text/link/file capabilities and an EICAR canary.

## Host KESL and broker

KESL 12.4 standalone and the root-owned fail-closed broker are not Compose
images. The Message Safety worker receives only
`/run/han-kesl/scan.sock`, not `kesl-control`, the Docker socket or host-root
access. Runtime records `scanner_engine=kesl`; `signatures_version` is the hash
of KESL version plus database date. Installation, hourly database updates,
socket permissions, clean/EICAR/error/stale gates and rollback follow
[`deployment/kesl/RUNBOOK.KESL.ru.md`](kesl/RUNBOOK.KESL.ru.md).