Добавлен сбор телеметрии на ВМ2

2026-08-26 15:12:51 +03:00
parent 728b9826a3
commit f989097484
32 changed files with 2029 additions and 134 deletions
@@ -34,8 +34,11 @@ The full step-by-step procedure with gates and copy-paste commands is in
   expose unauthenticated `PING` only for health and a password-protected
   `safety` user limited to required `han:safety:*` keys/commands. The password
   in `MESSAGE_SAFETY_REDIS_URL` must match. Start from
-   `redis/redis-safety.acl.template`, replace
-   `REPLACE_WITH_LONG_RANDOM_PASSWORD`, and never commit the password.
+   `redis/redis-safety.acl.template`, replace both placeholders, and add the
+   `exporter` user limited to `PING`/`INFO`. Its ACL password must exactly match
+   the separate `REDIS_EXPORTER_PASSWORD` secret. That secret is a JSON password
+   map, `{"redis://redis-safety:6379":"<THE_SAME_PASSWORD>"}`, not a raw password
+   string. Never commit either password.
 8. Provision distinct runtime and migration DB credentials.
   `MESSAGE_SAFETY_CONFIG_ADMIN_DATABASE_URL` may migrate/activate policy while
   `MESSAGE_SAFETY_DATABASE_URL` cannot; `BITRIX_SYNC_MIGRATION_DATABASE_URL`
@@ -76,13 +79,15 @@ Compose first:
   [`arch-10-deployment.md`](../../../../architectory/arch-10-deployment.md) §6.
 3. **Images** — build and push `han-message-safety`, `han-bitrix-sync`; record
   immutable digests for every `*_IMAGE` in `.env.example` (nginx, redis, clamav,
-   otel-collector).
+   otel-collector, Redis exporter and nginx exporter).
 4. **Selectel Secrets Manager** — populate all remote names from
   `deployment/secrets/config.example.json` (DSNs, tokens, S3 read-only keys,
-   `REDIS_SAFETY_ACL`, internal TLS PEM for `8443`). Dedicated VM2 IAM principal
-   with read-only access to those names only.
+   `REDIS_SAFETY_ACL`, `REDIS_EXPORTER_PASSWORD`, internal TLS PEM for `8443`).
+   Dedicated VM2 IAM principal with read-only access to those names only.
 5. **S3 quarantine bucket** and SigNoz OTLP endpoint — non-secret values in
-   `.env`.
+   `.env`. Current self-hosted SigNoz accepts private plaintext OTLP without
+   authentication; do not provision a fake auth secret or mandatory non-empty
+   header.
 6. **Internal TLS** — internal-CA certificate with SAN = VM2 private DNS; PEM
   stored in Secrets Manager, not in the release tree.

@@ -136,6 +141,11 @@ Copy `.env.example` → `.env`, install loader config as
 password with `systemd-creds`, edit nginx allow-lists. Details:
 [`RUNBOOK.ru.md`](RUNBOOK.ru.md) §4.

+The local Collector receives logs directly over OTLP (no `filelog`), scrapes
+only itself plus the Redis/nginx exporters, and collects host metrics through
+read-only `/hostfs`. Exporters and nginx `stub_status` use internal networks and
+`expose` only; they have no host-published ports.
+
 ## 5. PostgreSQL CA and initial public TLS

 Install managed PostgreSQL CA under `/etc/han/ca`, issue Let's Encrypt cert for
@@ -37,8 +37,12 @@ deployment-артефакты: `VM2_services/codebase/services/`. Локальн
 7. `REDIS_SAFETY_ACL` — полный ACL-файл, а не просто пароль. Он должен
   открывать неаутентифицированный `PING` только для health и
   защищённого паролем пользователя `safety`, ограниченного необходимыми
-   ключами/командами `han:safety:*`. 
-   (Пароль в `MESSAGE_SAFETY_REDIS_URL` должен совпадать. Используйте `redis/redis-safety.acl.template`, заменив `REPLACE_WITH_LONG_RANDOM_PASSWORD)
+   ключами/командами `han:safety:*`. Добавьте пользователя `exporter` только с
+   `PING`/`INFO`; его пароль в ACL должен в точности совпадать с отдельным
+   `REDIS_EXPORTER_PASSWORD`. Этот secret хранится в формате JSON password map:
+   `{"redis://redis-safety:6379":"<ТОТ_ЖЕ_ПАРОЛЬ>"}`, а не как голая строка.
+   Пароль `safety` в `MESSAGE_SAFETY_REDIS_URL` также должен совпадать с ACL. Используйте
+   `redis/redis-safety.acl.template`, заменив оба плейсхолдера.
 8. Выделите отдельные учётные данные БД для runtime и миграций.
   `MESSAGE_SAFETY_CONFIG_ADMIN_DATABASE_URL` может мигрировать/активировать
   политику, а `MESSAGE_SAFETY_DATABASE_URL` — нет; `BITRIX_SYNC_MIGRATION_DATABASE_URL`
@@ -83,12 +87,14 @@ deployment-артефакты: `VM2_services/codebase/services/`. Локальн
   [`arch-10-deployment.md`](../../../../architectory/arch-10-deployment.md) §6.
 3. **Образы** — собрать и push `han-message-safety`, `han-bitrix-sync`;
   получить immutable digest для всех `*_IMAGE` в `.env.example` (nginx, redis,
-   clamav, otel-collector).
+   clamav, otel-collector, Redis exporter, nginx exporter).
 4. **Selectel Secrets Manager** — заполнить все remote names из
   `deployment/secrets/config.example.json` (DSN, tokens, S3 read-only keys,
-   `REDIS_SAFETY_ACL`, internal TLS PEM для `8443`). Отдельный IAM principal
-   VM2 с read-only доступом только к этим именам.
+   `REDIS_SAFETY_ACL`, `REDIS_EXPORTER_PASSWORD`, internal TLS PEM для `8443`).
+   Отдельный IAM principal VM2 с read-only доступом только к этим именам.
 5. **S3 quarantine bucket** и SigNoz OTLP endpoint — значения в `.env`.
+   Текущий self-hosted SigNoz принимает private plaintext OTLP без auth, поэтому
+   не создавайте фиктивный auth-secret или обязательный непустой header.
 6. **Internal TLS** — сертификат внутренней CA с SAN = private DNS VM2;
   PEM хранится в Secrets Manager, не в каталоге релиза.

@@ -301,6 +307,13 @@ DSN, token, password, access/secret key туда не записываются.
 создайте отдельный VM2 IAM principal с read-only доступом только к remote names
 из mapping.

+Локальный Collector принимает traces, metrics и logs напрямую по OTLP; чтение
+Docker JSON через `filelog` не используется. Prometheus receiver собирает только
+метрики самого Collector, `redis-exporter` и `nginx-exporter`, а `hostmetrics` —
+метрики VM через read-only `/hostfs`. Exporter-контейнеры имеют только `expose`
+во внутренних сетях и не публикуют host ports. В nginx endpoint
+`/stub_status` слушает только внутренний `8081`.
+
 Зашифруйте пароль Selectel service user через systemd credentials, не помещая
 его в аргументы или history:

@@ -59,7 +59,7 @@ if [ -f "$ENV_FILE" ]; then
    true|false) ;;
    *) fail "OTEL_REMOTE_TLS_INSECURE must be exactly true or false" ;;
  esac
-  for image_key in MESSAGE_SAFETY_IMAGE BITRIX_SYNC_IMAGE NGINX_IMAGE REDIS_IMAGE CLAMAV_IMAGE OTEL_COLLECTOR_IMAGE; do
+  for image_key in MESSAGE_SAFETY_IMAGE BITRIX_SYNC_IMAGE NGINX_IMAGE REDIS_IMAGE CLAMAV_IMAGE OTEL_COLLECTOR_IMAGE REDIS_EXPORTER_IMAGE NGINX_EXPORTER_IMAGE; do
    image=$(/usr/bin/awk -F= -v key="$image_key" '$1 == key {print substr($0, index($0, "=") + 1)}' "$ENV_FILE")
    echo "$image" | /usr/bin/grep -Eq '@sha256:[0-9a-f]{64}$' ||
      fail "$image_key must be pinned by sha256 digest"
@@ -70,6 +70,25 @@ if [ -f "$ENV_FILE" ]; then
  esac
 fi

+collector_config="$ROOT/observability/otel-collector.yaml"
+[ -f "$collector_config" ] || fail "OpenTelemetry Collector config is missing"
+if [ -f "$collector_config" ]; then
+  /usr/bin/grep -Fq 'service.namespace, value: han-chat' "$collector_config" ||
+    fail "Collector must enforce service.namespace=han-chat"
+  /usr/bin/grep -Fq 'receivers: [otlp, prometheus, hostmetrics]' "$collector_config" ||
+    fail "Collector metrics pipeline is incomplete"
+  /usr/bin/grep -Fq 'receivers: [otlp]' "$collector_config" ||
+    fail "Collector direct OTLP logs pipeline is missing"
+  ! /usr/bin/grep -Fq 'filelog' "$collector_config" ||
+    fail "Collector filelog receiver is forbidden for direct OTLP logging"
+  /usr/bin/grep -Fq 'redis-exporter:9121' "$collector_config" ||
+    fail "Collector Redis exporter scrape target is missing"
+  /usr/bin/grep -Fq 'nginx-exporter:9113' "$collector_config" ||
+    fail "Collector nginx exporter scrape target is missing"
+  /usr/bin/grep -Fq 'tail_sampling:' "$collector_config" ||
+    fail "Collector tail sampling is missing"
+fi
+
 bitrix_allowlist="$ROOT/nginx/allowlists/bitrix-webhook-allowlist.conf"
 private_allowlist="$ROOT/nginx/allowlists/private-caller-allowlist.conf"
 for allowlist in "$bitrix_allowlist" "$private_allowlist"; do
@@ -106,7 +125,8 @@ BITRIX_SYNC_CRM_REST_WEBHOOK_URL
 BITRIX_SYNC_CONTACT_RECEIVER_TOKEN
 BITRIX_SYNC_ALERT_RECEIVER_TOKEN
 BITRIX_SYNC_SERVICE_TOKEN
-REDIS_SAFETY_ACL'
+REDIS_SAFETY_ACL
+REDIS_EXPORTER_PASSWORD'

 if [ -f "$MANIFEST" ]; then
  old_ifs=$IFS
@@ -123,6 +143,37 @@ if [ -f "$MANIFEST" ]; then
  done
  IFS=$old_ifs

+  redis_acl=$(/usr/bin/awk -F= \
+    '$1 == "REDIS_SAFETY_ACL" {print substr($0, index($0, "=") + 1)}' \
+    "$MANIFEST")
+  redis_exporter_password_file=$(/usr/bin/awk -F= \
+    '$1 == "REDIS_EXPORTER_PASSWORD" {print substr($0, index($0, "=") + 1)}' \
+    "$MANIFEST")
+  if [ -f "$redis_acl" ] && [ -f "$redis_exporter_password_file" ]; then
+    /usr/bin/grep -Eq '^user exporter reset on >[^[:space:]]+ -@all \+ping \+info$' "$redis_acl" ||
+      fail "Redis ACL must contain the restricted exporter user"
+    if exporter_password=$(/usr/bin/python3 -c '
+import json
+import sys
+
+target = "redis://redis-safety:6379"
+with open(sys.argv[1], encoding="utf-8") as source:
+    values = json.load(source)
+if set(values) != {target}:
+    raise SystemExit("password map must contain exactly redis://redis-safety:6379")
+password = values[target]
+if not isinstance(password, str) or not password or any(char.isspace() for char in password):
+    raise SystemExit("password must be a non-empty whitespace-free string")
+print(password, end="")
+' "$redis_exporter_password_file"); then
+      /usr/bin/grep -Fq -- ">$exporter_password " "$redis_acl" ||
+        fail "Redis exporter password map does not match REDIS_SAFETY_ACL"
+      unset exporter_password
+    else
+      fail "REDIS_EXPORTER_PASSWORD must be a valid redis_exporter JSON password map"
+    fi
+  fi
+
  internal_cert=$(/usr/bin/awk -F= \
    '$1 == "VM2_INTERNAL_TLS_CERTIFICATE" {print substr($0, index($0, "=") + 1)}' \
    "$MANIFEST")
@@ -90,6 +90,11 @@
      "remote": "vm2/REDIS_SAFETY_ACL",
      "consumers": ["redis-safety"],
      "max_bytes": 4096
+    },
+    "REDIS_EXPORTER_PASSWORD": {
+      "remote": "vm2/REDIS_EXPORTER_PASSWORD",
+      "consumers": ["redis-exporter"],
+      "max_bytes": 1024
    }
  }
 }