Admin/han-app
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Реализация на отдельных двух машинах с протестированным взаимодействием по проверке сообщений

Browse Source
This commit is contained in:
mi
2026-08-19 18:24:00 +03:00
parent bbef7a30c9
commit c7a80e7256
103 changed files with 3457 additions and 3725 deletions
Download Patch File Download Diff File Expand all files Collapse all files
VM2_services/codebase/services/deployment/RUNBOOK.md
+153 -50
View File
@@ -1,14 +1,22 @@
# VM2 Processing deployment runbook
This directory is the independent VM2 foundation. It does not deploy VM1 or
`codebase/backend`. All commands below are operator commands; repository
creation does not execute them.
This directory is the independent VM2 foundation in the `VM2_services`
repository. It does not deploy VM1 or `VM1_app/codebase/backend`. All commands
below are operator commands; repository creation does not execute them.
Repository root: `HAN_chat_specification/VM2_services`. Compose and deployment
artifacts live under `VM2_services/codebase/services/`. Local operator commands
assume the current directory is `VM2_services` unless stated otherwise.
The full step-by-step procedure with gates and copy-paste commands is in
[`RUNBOOK.ru.md`](RUNBOOK.ru.md).
## Production blockers before first start
1. Replace every `.env` placeholder with reviewed non-secret values. Keep
`BITRIX_SYNC_ENABLED=false` until migrations, grants, portal fields, robot
contracts and cutover are signed off.
contracts and cutover are signed off. Build and push application images to
the registry first (blocker 2).
2. Fill every `*_IMAGE` variable with a reviewed registry digest. Root Compose
rejects missing image references; mutable tags are not production evidence.
3. Install production files as `root:root`; `deploy` must not be in `docker`
@@ -38,61 +46,156 @@ creation does not execute them.
S3, Secrets Manager, Bitrix24, DNS/NTP, SigNoz and ClamAV destinations.
Registry/package access exists only during controlled maintenance windows.
## Install
## Who runs what
- Bootstrap a fresh Ubuntu 24.04 VM as root with
`deployment/scripts/setup-vm.sh`, supplying `VM1_PRIVATE_CIDRS`, optional
private/VPN `OPS_CIDRS`, and separate Ed25519 public-key files for deploy and
break-glass admin. SSH is publicly reachable but key-only and protected by
fail2ban; the CIDR variables apply only to private port `8443`. The script
installs host packages/firewalls and roles but never starts Compose. Set a
separate admin sudo password; verify deploy login, admin login and admin sudo
in independent sessions before rerunning with `HARDEN_SSH=true`.
Root/deploy/admin key reuse is rejected.
- Checkout an immutable release under `/opt/han-chat/services`.
- Copy `.env.example` to root-owned mode `0600` `.env`.
- Install `secrets_loader.py` and `han-secrets` under
`/usr/local/lib/han-secrets-vm2/`, root-owned and non-writable.
- Install `han-compose` as `/usr/local/sbin/han-vm2-compose`.
- Install `han-secrets-vm2.service` and `han-processing.service` under
`/etc/systemd/system/`.
- Install `han-message-safety-mode` as root-owned `0755` and the sudoers
template as `/etc/sudoers.d/deploy-message-safety-mode` mode `0440`; validate
with `visudo -cf`. Create the dedicated host group `han-message-safety` with
GID `10001`. Before the first Compose validation, create
`/etc/han-chat/message-safety-mode.env` as
`root:han-message-safety 0640` with all three flags `false` (or invoke the
helper's `standard` transition after the fixed launcher is installed).
- Install loader config using the exact `APP_ENV` suffix. With the committed
example (`APP_ENV=production-like`) the path is
`/etc/han/secrets/vm2-production-like.selectel.json` mode `0600`. For
controlled no-provider recovery use an explicit `file`
config pointing to a root-only `0700` directory containing exactly one file
per configured key. Selectel failure never falls back automatically.
- **Operator workstation:** builds the release archive and transfers it to VM2.
Local examples use PowerShell from `VM2_services`.
- **`root` on VM2:** host bootstrap, verified release activation, root-owned
files, `.env`, secret mapping, credentials, TLS/allow-lists, migrations and
first start.
- **`deploy` on VM2:** accepts releases only in `/var/lib/han-deploy/incoming`,
checks status/logs and runs installed fixed systemd operations through exact
sudo rules. `deploy` must not run `docker`, edit `/opt/han-chat/services` or
join the `docker` group.
- **`admin` on VM2:** personal break-glass role with a separate SSH key and
local sudo password. Not used for routine deploy; not in `docker`/`lxd`; every
login and sudo call is an incident operation.
## Preflight and startup
## Prerequisites (before §1)
Run `deployment/preflight.sh` first. Then, through the approved root units:
This runbook covers operations **on an already provisioned VM2**. Prepare outside
Compose first:
1. synchronize secrets; any missing/oversized/invalid secret blocks startup;
2. validate resolved Compose without storing its output;
3. run the two `ops` migration jobs and create/activate the reviewed initial
Message Safety config before starting either runtime;
4. validate nginx config and both certificate chains;
5. start Redis/Collector, ClamAV, application API/workers, then nginx;
6. verify that only nginx publishes `80`, `443`, and private-bound `8443`;
7. verify all non-exact public paths return `404`, HTTP webhook paths return
`426` without redirect/query reflection, wrong methods fail, and wrong
source CIDRs are rejected before upstream;
8. verify private Safety check/task/status and sync status only from approved
callers; verify public `/internal/*` is `404`;
9. canary telemetry with a fake token marker and prove query, form body,
Authorization, DSN, S3 key and object key are absent from logs/traces.
1. **Selectel infrastructure** — VPC/subnet, SG (public `80/443/22`; private
`8443` only; default-deny egress after bootstrap), sizing (4 vCPU / 8 GB RAM /
80 GB SSD — see
[`module-10-deployment-vm2.md`](../../../documentation/module-10-deployment-vm2.md)),
public and private VM2 IPs, DNS A record for `PROCESSING_PUBLIC_HOST`.
2. **Managed PostgreSQL** — schemas/roles for `message_safety` and
`bitrix_sync`, separate migration/runtime DSNs; see
[`arch-10-deployment.md`](../../../../architectory/arch-10-deployment.md) §6.
3. **Images** — build and push `han-message-safety`, `han-bitrix-sync`; record
immutable digests for every `*_IMAGE` in `.env.example` (nginx, redis, clamav,
otel-collector).
4. **Selectel Secrets Manager** — populate all remote names from
`deployment/secrets/config.example.json` (DSNs, tokens, S3 read-only keys,
`REDIS_SAFETY_ACL`, internal TLS PEM for `8443`). Dedicated VM2 IAM principal
with read-only access to those names only.
5. **S3 quarantine bucket** and SigNoz OTLP endpoint — non-secret values in
`.env`.
6. **Internal TLS** — internal-CA certificate with SAN = VM2 private DNS; PEM
stored in Secrets Manager, not in the release tree.
Section order: §1–§5 → Gates 1–9 → §7 (post-acceptance). Run
`systemctl enable` and `systemctl start han-processing.service` **only after
Gate 5 succeeds**.
## 1. Bootstrap a fresh VM2
From `VM2_services` on the operator workstation, copy the setup script:
```powershell
scp -i C:\Users\MI\.ssh\hansel `
.\codebase\services\deployment\scripts\setup-vm.sh `
root@<VM2_PUBLIC_IP>:/root/setup-vm2.sh
```
Bootstrap as root with separate Ed25519 deploy/admin keys, `VM1_PRIVATE_CIDRS`,
optional `OPS_CIDRS`, admin sudo password, deploy/admin login verification, then
rerun with `HARDEN_SSH=true`. Full commands: [`RUNBOOK.ru.md`](RUNBOOK.ru.md) §1.
## 2. Release transfer under `deploy`
From `VM2_services`:
```powershell
$Release = "<VERSION_OR_GIT_SHA>"
tar --exclude=services/.env `
--exclude='services/**/__pycache__' `
--exclude='services/**/.pytest_cache' `
--exclude='services/**/.ruff_cache' `
-czf "vm2-services-$Release.tar.gz" -C .\codebase services
Get-FileHash "vm2-services-$Release.tar.gz" -Algorithm SHA256
scp -i C:\Users\MI\.ssh\hansel "vm2-services-$Release.tar.gz" `
deploy@<VM2_PUBLIC_IP>:/var/lib/han-deploy/incoming/
```
`deploy` verifies SHA-256 and archive listing only; it must not unpack into
production. Details: [`RUNBOOK.ru.md`](RUNBOOK.ru.md) §2–§3.
## 3. Activation and root-owned install
`root` verifies the archive, extracts to staging, rsyncs into
`/opt/han-chat/services`, reruns `setup-vm.sh` to install helpers/units. Details:
[`RUNBOOK.ru.md`](RUNBOOK.ru.md) §3.
## 4. Non-secret config and Selectel
Copy `.env.example` → `.env`, install loader config as
`/etc/han/secrets/vm2-<APP_ENV>.selectel.json`, encrypt Selectel service-user
password with `systemd-creds`, edit nginx allow-lists. Details:
[`RUNBOOK.ru.md`](RUNBOOK.ru.md) §4.
## 5. PostgreSQL CA and initial public TLS
Install managed PostgreSQL CA under `/etc/han/ca`, issue Let's Encrypt cert for
`PROCESSING_PUBLIC_HOST`, stage public cert/key for nginx. Details:
[`RUNBOOK.ru.md`](RUNBOOK.ru.md) §5.
## 6. Gates 1–9: preflight, migrations and first start
Execute gates **in order** 1 → 2 → 3 → 4 → 5 → 6 → 7 → 8 → 9. Do not enable
`han-processing.service` until Gate 5 completes successfully.
| Gate | Purpose |
| --- | --- |
| 1 | Secrets materialized via `han-secrets-vm2.service` |
| 2 | Static preflight and resolved Compose (`@sha256:` images) |
| 3 | Migrations, initial Message Safety config create/activate |
| 4 | nginx `-t` with TLS and upstream placeholders |
| 5 | Ordered `compose up`; then `systemctl enable/start han-processing` |
| 6 | Host ports, public/private certificate chains, certbot renewal |
| 7 | Public routing smoke (308/404/426/405) |
| 8 | Private Safety/sync API from VM1/ops only |
| 9 | Fake-token canary — no secrets in logs/traces |
After Gate 5, `deploy` may run:
```sh
sudo systemctl restart han-secrets-vm2.service
sudo systemctl restart han-processing.service
sudo systemctl --no-pager status han-processing.service
sudo journalctl --no-pager -u han-processing.service
```
Full gate commands: [`RUNBOOK.ru.md`](RUNBOOK.ru.md) §6.
Do not open webhook traffic while `bitrix-sync` is disabled. A disabled or
failed receiver must return retryable `503`/closed routing, never successful
`2xx ignored`.
## 7. After Gate 9 — post-acceptance
Gate 9 completes VM2 technical acceptance but **does not** authorize Message
Safety cutover on VM1 or Bitrix sync enablement.
1. Verify autostart: `docker`, `han-chat-vm2-docker-firewall`, `han-secrets-vm2`,
`han-processing`, `certbot.timer`.
2. Reboot-gate: `systemctl reboot`, then repeat autostart checks and Gates 6–8
briefly.
3. Record release evidence: `han-vm2-compose config --images`, `ps`, certbot
timer, unit journals — without secret values.
4. Configure operational monitoring (unhealthy/restart/OOM, TLS expiry, ClamAV
signature age, OTEL queue, disk/RAM, MOCK mode, private Safety API).
5. Proceed to controlled Message Safety cutover on VM1 — see
[`module-10-deployment-vm2.md`](../../../documentation/module-10-deployment-vm2.md) §13.
6. Keep `BITRIX_SYNC_ENABLED=false` and `BITRIX_SYNC_MODE=disabled`; Bitrix
public allow-list remains `deny all;` until
[`module-07-bitrix-sync.md`](../../../documentation/module-07-bitrix-sync.md)
cutover gates are signed off.
Details: [`RUNBOOK.ru.md`](RUNBOOK.ru.md) §7.
## Failure policy
- Safety dependency failure is fail-closed: VM1 must not send/promote content.