Внедрение KESL на ВМ2 + замена CLAMAV на KESL
This commit is contained in:
mi
@@ -7,7 +7,6 @@ MESSAGE_SAFETY_IMAGE=<registry>/han-message-safety@sha256:<digest>
|
||||
BITRIX_SYNC_IMAGE=<registry>/han-bitrix-sync@sha256:<digest>
|
||||
NGINX_IMAGE=nginxinc/nginx-unprivileged@sha256:<reviewed-digest>
|
||||
REDIS_IMAGE=redis@sha256:<reviewed-digest>
|
||||
CLAMAV_IMAGE=clamav/clamav@sha256:<reviewed-digest>
|
||||
OTEL_COLLECTOR_IMAGE=otel/opentelemetry-collector-contrib@sha256:<reviewed-digest>
|
||||
REDIS_EXPORTER_IMAGE=oliver006/redis_exporter@sha256:<reviewed-digest>
|
||||
NGINX_EXPORTER_IMAGE=nginx/nginx-prometheus-exporter@sha256:<reviewed-digest>
|
||||
@@ -20,8 +19,7 @@ MESSAGE_SAFETY_HOST=0.0.0.0
|
||||
MESSAGE_SAFETY_PORT=8080
|
||||
MESSAGE_SAFETY_WORKER_CONCURRENCY=5
|
||||
MESSAGE_SAFETY_DNS_RESOLVERS=<vpc-resolver-ip>
|
||||
MESSAGE_SAFETY_CLAMAV_HOST=clamd
|
||||
MESSAGE_SAFETY_CLAMAV_PORT=3310
|
||||
MESSAGE_SAFETY_ANTIVIRUS_SOCKET=/run/han-kesl/scan.sock
|
||||
MESSAGE_SAFETY_ARTIFACTS_DIR=/app/app/artifacts
|
||||
MESSAGE_SAFETY_MODE_FILE=/etc/han-chat/message-safety-mode.env
|
||||
|
||||
|
||||
@@ -46,7 +46,9 @@ The full step-by-step procedure with gates and copy-paste commands is in
|
||||
role. Migration credentials are mounted only into the `ops` profile jobs.
|
||||
9. The setup script leaves UFW egress open for bootstrap. Before production,
|
||||
constrain egress through Selectel SG/NAT/proxy to the approved PostgreSQL,
|
||||
S3, Secrets Manager, Bitrix24, DNS/NTP, SigNoz and ClamAV destinations.
|
||||
S3, Secrets Manager, Bitrix24, DNS/NTP and SigNoz destinations. Host KESL
|
||||
receives only update-source egress approved by
|
||||
[`deployment/kesl/RUNBOOK.KESL.ru.md`](kesl/RUNBOOK.KESL.ru.md).
|
||||
Registry/package access exists only during controlled maintenance windows.
|
||||
|
||||
## Who runs what
|
||||
@@ -78,8 +80,9 @@ Compose first:
|
||||
`bitrix_sync`, separate migration/runtime DSNs; see
|
||||
[`arch-10-deployment.md`](../../../../architectory/arch-10-deployment.md) §6.
|
||||
3. **Images** — build and push `han-message-safety`, `han-bitrix-sync`; record
|
||||
immutable digests for every `*_IMAGE` in `.env.example` (nginx, redis, clamav,
|
||||
otel-collector, Redis exporter and nginx exporter).
|
||||
immutable digests for every `*_IMAGE` in `.env.example` (nginx, redis,
|
||||
otel-collector, Redis exporter and nginx exporter). `clamd`/`freshclam` are
|
||||
absent from Compose; KESL 12.4 standalone and its broker run on the host.
|
||||
4. **Selectel Secrets Manager** — populate all remote names from
|
||||
`deployment/secrets/config.example.json` (DSNs, tokens, S3 read-only keys,
|
||||
`REDIS_SAFETY_ACL`, `REDIS_EXPORTER_PASSWORD`, internal TLS PEM for `8443`).
|
||||
@@ -157,6 +160,13 @@ Install managed PostgreSQL CA under `/etc/han/ca`, issue Let's Encrypt cert for
|
||||
Execute gates **in order** 1 → 2 → 3 → 4 → 5 → 6 → 7 → 8 → 9. Do not enable
|
||||
`han-processing.service` until Gate 5 completes successfully.
|
||||
|
||||
Before Message Safety starts in Gate 5, complete the operator KESL runbook,
|
||||
then enable/start `han-kesl-scan-broker.socket` and verify KESL/socket status
|
||||
plus `root:han-message-safety 0660` on `/run/han-kesl/scan.sock`. Only then
|
||||
start Message Safety. The broker is a custom integration: exact output/exit
|
||||
semantics of `kesl-control --scan-file --action Inform`, cleanup and throughput
|
||||
must pass on the target VM2.
|
||||
|
||||
| Gate | Purpose |
|
||||
| --- | --- |
|
||||
| 1 | Secrets materialized via `han-secrets-vm2.service` |
|
||||
@@ -195,8 +205,9 @@ Safety cutover on VM1 or Bitrix sync enablement.
|
||||
briefly.
|
||||
3. Record release evidence: `han-vm2-compose config --images`, `ps`, certbot
|
||||
timer, unit journals — without secret values.
|
||||
4. Configure operational monitoring (unhealthy/restart/OOM, TLS expiry, ClamAV
|
||||
signature age, OTEL queue, disk/RAM, MOCK mode, private Safety API).
|
||||
4. Configure operational monitoring (unhealthy/restart/OOM, TLS expiry, KESL
|
||||
version/database date, hourly update, broker/socket status, OTEL queue,
|
||||
disk/RAM, MOCK mode, private Safety API).
|
||||
5. Proceed to controlled Message Safety cutover on VM1 — see
|
||||
[`module-10-deployment-vm2.md`](../../../documentation/module-10-deployment-vm2.md) §13.
|
||||
6. Keep `BITRIX_SYNC_ENABLED=false` and `BITRIX_SYNC_MODE=disabled`; Bitrix
|
||||
@@ -209,8 +220,8 @@ Details: [`RUNBOOK.ru.md`](RUNBOOK.ru.md) §7.
|
||||
## Failure policy
|
||||
|
||||
- Safety dependency failure is fail-closed: VM1 must not send/promote content.
|
||||
- Stale/unavailable ClamAV signatures disable file capability only; they never
|
||||
convert a scan error to allow.
|
||||
- A stale/unavailable KESL database or broker error disables file capability
|
||||
only; scan errors remain retryable and eventually return `503`, never allow.
|
||||
- Redis loss may remove acceleration but PostgreSQL remains authoritative.
|
||||
- OTEL outage queues within the bounded volume and must not change verdicts.
|
||||
- Rollback does not downgrade schemas, delete durable tasks/mappings, or run
|
||||
@@ -234,10 +245,12 @@ health, and restores the previous mode on failure. MOCK has no timeout: keep a
|
||||
high-severity alert active until explicit `standard`, then verify normal
|
||||
text/link/file capabilities and an EICAR canary.
|
||||
|
||||
## Known image exceptions
|
||||
## Host KESL and broker
|
||||
|
||||
ClamAV images may require UID/path adjustments after validating the exact
|
||||
digest. Do not weaken `read_only`, capabilities or mounts globally: document
|
||||
the smallest writable signature/runtime paths and compensate with network and
|
||||
resource limits. `freshclam` alone receives signature-CDN egress; `clamd`
|
||||
receives none.
|
||||
KESL 12.4 standalone and the root-owned fail-closed broker are not Compose
|
||||
images. The Message Safety worker receives only
|
||||
`/run/han-kesl/scan.sock`, not `kesl-control`, the Docker socket or host-root
|
||||
access. Runtime records `scanner_engine=kesl`; `signatures_version` is the hash
|
||||
of KESL version plus database date. Installation, hourly database updates,
|
||||
socket permissions, clean/EICAR/error/stale gates and rollback follow
|
||||
[`deployment/kesl/RUNBOOK.KESL.ru.md`](kesl/RUNBOOK.KESL.ru.md).
|
||||
|
||||
@@ -51,8 +51,9 @@ deployment-артефакты: `VM2_services/codebase/services/`. Локальн
|
||||
`ops`.
|
||||
9. Setup оставляет исходящий трафик UFW открытым на bootstrap-окно. До
|
||||
production ограничьте egress правилами Selectel SG/NAT/proxy до
|
||||
утверждённых PostgreSQL, S3, Secrets Manager, Bitrix24, DNS/NTP, SigNoz и
|
||||
источников ClamAV. Registry/package repositories оставляйте только на
|
||||
утверждённых PostgreSQL, S3, Secrets Manager, Bitrix24, DNS/NTP и SigNoz.
|
||||
Для host KESL разрешите только источники обновления из
|
||||
[`deployment/kesl/RUNBOOK.KESL.ru.md`](kesl/RUNBOOK.KESL.ru.md). Registry/package repositories оставляйте только на
|
||||
controlled maintenance window.
|
||||
|
||||
## Кто что выполняет
|
||||
@@ -87,7 +88,8 @@ deployment-артефакты: `VM2_services/codebase/services/`. Локальн
|
||||
[`arch-10-deployment.md`](../../../../architectory/arch-10-deployment.md) §6.
|
||||
3. **Образы** — собрать и push `han-message-safety`, `han-bitrix-sync`;
|
||||
получить immutable digest для всех `*_IMAGE` в `.env.example` (nginx, redis,
|
||||
clamav, otel-collector, Redis exporter, nginx exporter).
|
||||
otel-collector, Redis exporter, nginx exporter). `clamd`/`freshclam` в
|
||||
Compose отсутствуют: KESL 12.4 standalone и broker устанавливаются на host.
|
||||
4. **Selectel Secrets Manager** — заполнить все remote names из
|
||||
`deployment/secrets/config.example.json` (DSN, tokens, S3 read-only keys,
|
||||
`REDIS_SAFETY_ACL`, `REDIS_EXPORTER_PASSWORD`, internal TLS PEM для `8443`).
|
||||
@@ -526,7 +528,7 @@ REVOKE USAGE ON SCHEMA han_app FROM <BITRIX_SYNC_MIGRATION_ROLE>;
|
||||
новый монотонный номер и отдельные значения `--actor`/`--approved-by`; повторно
|
||||
активировать старую версию нельзя. Alembic downgrade запрещён.
|
||||
|
||||
При обновлении ClamAV policy образ Message Safety должен содержать согласованные
|
||||
При обновлении KESL policy образ Message Safety должен содержать согласованные
|
||||
seed и schema: seed `max_signature_age_hours=240`, schema maximum `720`
|
||||
(30 дней). После обновления immutable image digest создайте новую config
|
||||
version; существующую active version не редактируйте и не активируйте повторно:
|
||||
@@ -548,7 +550,7 @@ NEXT_VERSION='<СЛЕДУЮЩИЙ_МОНОТОННЫЙ_НОМЕР>'
|
||||
/usr/local/sbin/han-vm2-compose up -d --no-deps --force-recreate \
|
||||
message-safety-api message-safety-worker
|
||||
/usr/local/sbin/han-vm2-compose ps \
|
||||
message-safety-api message-safety-worker clamd freshclam
|
||||
message-safety-api message-safety-worker
|
||||
unset NEXT_VERSION
|
||||
```
|
||||
|
||||
@@ -588,8 +590,15 @@ open-file limit быть не должно. Ошибка отсутствующ
|
||||
Под `root` на VM2:
|
||||
|
||||
```sh
|
||||
cd /opt/han-chat/services
|
||||
# Сначала полностью выполните operator runbook:
|
||||
# deployment/kesl/RUNBOOK.KESL.ru.md
|
||||
systemctl enable --now han-kesl-scan-broker.socket
|
||||
systemctl --no-pager status kesl han-kesl-scan-broker.socket
|
||||
test -S /run/han-kesl/scan.sock
|
||||
stat -c '%U:%G %a %n' /run/han-kesl/scan.sock
|
||||
|
||||
/usr/local/sbin/han-vm2-compose up -d redis-safety otel-collector
|
||||
/usr/local/sbin/han-vm2-compose up -d freshclam clamd
|
||||
/usr/local/sbin/han-vm2-compose up -d \
|
||||
message-safety-api message-safety-worker
|
||||
/usr/local/sbin/han-vm2-compose up -d \
|
||||
@@ -598,6 +607,13 @@ open-file limit быть не должно. Ошибка отсутствующ
|
||||
/usr/local/sbin/han-vm2-compose ps
|
||||
```
|
||||
|
||||
До запуска Message Safety operator KESL runbook обязан подтвердить KESL 12.4
|
||||
standalone, успешное ежечасное обновление, допустимый database date и canary.
|
||||
Socket должен иметь `root:han-message-safety 0660` и не быть доступен посторонним
|
||||
UID. Broker является custom
|
||||
integration: неизвестный output/exit code `kesl-control --scan-file --action Inform`
|
||||
считается scanner error, а не clean. Формат и throughput проверяются на target VM2.
|
||||
|
||||
`otel-collector` автоматически запускает одноразовый `otel-queue-init`. Он
|
||||
выставляет владельца persistent queue `10001:10001` и завершается с кодом `0`;
|
||||
сам Collector стартует только после этого.
|
||||
@@ -892,7 +908,7 @@ journalctl --no-pager -u han-processing.service -u han-secrets-vm2.service
|
||||
|
||||
- container unhealthy/restart/OOM;
|
||||
- срок TLS;
|
||||
- возраст ClamAV signatures;
|
||||
- KESL version/database date, результат ежечасного update и broker/socket status;
|
||||
- OTEL queue/export errors;
|
||||
- disk/RAM;
|
||||
- активный MOCK mode;
|
||||
@@ -1107,8 +1123,9 @@ plan.
|
||||
|
||||
- Отказ зависимости Safety — fail-closed: VM1 не должна отправлять/продвигать
|
||||
контент.
|
||||
- Устаревшие/недоступные сигнатуры ClamAV отключают только файловую
|
||||
capability; ошибка сканирования никогда не превращается в allow.
|
||||
- Устаревшая/недоступная база KESL или ошибка broker отключают только файловую
|
||||
capability; ошибка сканирования никогда не превращается в allow и ведёт к
|
||||
retry/`503`.
|
||||
- Потеря Redis может убрать ускорение, но PostgreSQL остаётся источником
|
||||
истины.
|
||||
- Сбой OTEL ставит в очередь в пределах ограниченного тома и не должен
|
||||
@@ -1134,10 +1151,12 @@ han-message-safety-mode mock --text-free false --file-free false
|
||||
таймаута: держите high-severity alert активным до явного `standard`, затем
|
||||
проверьте нормальные text/link/file capabilities и EICAR-canary.
|
||||
|
||||
## Известные исключения по образам
|
||||
## Host KESL и broker
|
||||
|
||||
Образы ClamAV могут потребовать корректировок UID/path после валидации
|
||||
точного digest. Не ослабляйте `read_only`, capabilities или mounts глобально:
|
||||
задокументируйте минимальные writable пути для сигнатур/runtime и
|
||||
компенсируйте сетевыми и ресурсными лимитами. Egress к signature-CDN
|
||||
получает только `freshclam`; `clamd` — нет.
|
||||
KESL 12.4 standalone и root-owned fail-closed broker не являются Compose
|
||||
образами. Message Safety worker получает только Unix socket
|
||||
`/run/han-kesl/scan.sock`; доступ к `kesl-control`, Docker socket и host root
|
||||
ему не выдаётся. `scanner_engine=kesl`, а `signatures_version` вычисляется как
|
||||
hash KESL version + database date. Установка, ежечасное обновление, права
|
||||
socket, clean/EICAR/error/stale gates и rollback выполняются строго по
|
||||
[`deployment/kesl/RUNBOOK.KESL.ru.md`](kesl/RUNBOOK.KESL.ru.md).
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
[Unit]
|
||||
Description=HAN Processing VM2 root Compose stack
|
||||
Requires=docker.service han-secrets-vm2.service
|
||||
After=docker.service han-secrets-vm2.service network-online.target
|
||||
Requires=docker.service han-secrets-vm2.service han-kesl-scan-broker.socket
|
||||
After=docker.service han-secrets-vm2.service han-kesl-scan-broker.socket network-online.target
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
|
||||
@@ -0,0 +1,95 @@
|
||||
# Карта доказательств KESL на ВМ2
|
||||
|
||||
Заполняется оператором после `RUNBOOK.KESL.ru.md`. Не включать activation code,
|
||||
secrets, ПД, S3 object key, file bytes, полный checksum или EICAR.
|
||||
|
||||
## 1. Изменение
|
||||
|
||||
- Change ID / окно:
|
||||
- Оператор / approvers Security, Service, Operations:
|
||||
- Hostname, Ubuntu, kernel, architecture:
|
||||
- KESL package/version, SHA-256, источник:
|
||||
- HAN release SHA и image digests:
|
||||
- Коммерческая сборка не заявлена сертифицированной ФСТЭК: да / нет
|
||||
- KSN decision и правовое основание:
|
||||
|
||||
## 2. Baseline и stop conditions
|
||||
|
||||
- VM2 containers healthy/running:
|
||||
- Public/private smoke:
|
||||
- CPU, available RAM, swap activity, disk/IO wait:
|
||||
- Safety p95/p99, error rate, queue age:
|
||||
- Redis latency/blocked clients:
|
||||
- Restart/OOM, OTEL queue:
|
||||
- Утверждённые пороги и rollback approver:
|
||||
|
||||
## 3. АВЗ.1
|
||||
|
||||
Норма: Приказ ФСТЭК №21, приложение АВЗ.1; п. 8.6 — обнаружение
|
||||
вредоносных программ/информации и реагирование.
|
||||
|
||||
- [ ] `kesl.service` active, лицензия действительна.
|
||||
- [ ] File Threat Protection task 1 = `Started`.
|
||||
- [ ] `ActionOnThreat=DisinfectDeleteIfNotPossible`, `ScanArchived=No`.
|
||||
- [ ] fanotify on-access обнаружил и обработал разрешённый EICAR.
|
||||
- [ ] Событие detection/action присутствует в KESL events.
|
||||
- [ ] Исключения ограничены фактическими hot-data mountpoint.
|
||||
- [ ] Broker staging отсутствует в `ExcludedFromScanScope`.
|
||||
- [ ] Broker подтверждает `scanned >= 1`, `skipped = 0`, `errors = 0`.
|
||||
- [ ] Firewall, smoke и health после Block успешны.
|
||||
- [ ] 24 часа без неприемлемой деградации/OOM/restart/5xx.
|
||||
|
||||
Артефакты/время/результат:
|
||||
|
||||
## 4. АВЗ.2
|
||||
|
||||
Норма: Приказ ФСТЭК №21, приложение АВЗ.2 — обновление базы признаков
|
||||
вредоносных компьютерных программ.
|
||||
|
||||
- [ ] Update task 6 вручную завершилась успешно.
|
||||
- [ ] Базы загружены, дата актуальна.
|
||||
- [ ] Schedule = Hourly.
|
||||
- [ ] Наблюдён последующий автоматический successful update.
|
||||
- [ ] Alert на update failure / unloaded / age >240h / license failure.
|
||||
- [ ] Назначен ежедневный контроль и owner.
|
||||
|
||||
Артефакты/время последнего automatic update:
|
||||
|
||||
## 5. Message Safety scan-broker
|
||||
|
||||
- [ ] Socket `root:han-message-safety:0660`, TCP listener отсутствует.
|
||||
- [ ] Clean corpus → `clean`; Message Safety final `200 allow`.
|
||||
- [ ] EICAR → `infected`; Message Safety sticky `403 deny`.
|
||||
- [ ] KESL stopped/timeout/unknown output → retry/terminal `503`, не allow/deny.
|
||||
- [ ] `scanner_engine=kesl`.
|
||||
- [ ] `signatures_version` меняется при обновлении KESL databases.
|
||||
- [ ] Старый cache не используется после смены signatures version.
|
||||
- [ ] Logs/traces не содержат bytes, object key, checksum, filename или secrets.
|
||||
- [ ] 5 slots / 2 files per second gate пройден.
|
||||
|
||||
Результаты corpus/load и ссылки на безопасные метрики:
|
||||
|
||||
## 6. РСБ и АНЗ.2
|
||||
|
||||
- [ ] Определены и защищены KESL detection/remediation/update/license events.
|
||||
- [ ] Определены место/срок хранения и экспорт/регламент просмотра.
|
||||
- [ ] Версия KESL и upgrade lifecycle контролируются.
|
||||
- [ ] Kernel/Docker/KESL upgrade требует compatibility pilot + evidence delta.
|
||||
|
||||
## 7. Cutover и rollback
|
||||
|
||||
- [ ] До cutover KESL+ClamAV coexistence не нарушило resource gates.
|
||||
- [ ] После cutover `clamd`/`freshclam` отсутствуют, старый egress закрыт.
|
||||
- [ ] Предыдущая совместимая release/config пара зафиксирована.
|
||||
- [ ] Desk check: application rollback возвращает ClamAV release.
|
||||
- [ ] Desk check: минимальный host rollback — `kesl-control --stop-task 1`.
|
||||
- [ ] Rollback не удаляет Docker volumes и не использует `down -v`.
|
||||
|
||||
## 8. Итог
|
||||
|
||||
- АВЗ.1: принято / не принято; ограничения:
|
||||
- АВЗ.2: принято / не принято; ограничения:
|
||||
- Message Safety KESL cutover: принято / не принято:
|
||||
- Residual risk custom broker / parser / throughput:
|
||||
- Operations / Security / Service owner, ФИО, подпись, дата:
|
||||
- Следующий review:
|
||||
@@ -0,0 +1,288 @@
|
||||
# KESL 12.4 standalone на ВМ2 Processing
|
||||
|
||||
Операторский runbook для Ubuntu 24.04 ВМ2. Команды выполняет персональная роль
|
||||
`admin` через `sudo` в отдельное maintenance window. Репозиторий не устанавливает,
|
||||
не активирует и не запускает KESL автоматически.
|
||||
|
||||
Цели:
|
||||
|
||||
1. АВЗ.1: обнаружение и реагирование через host File Threat Protection.
|
||||
2. АВЗ.2: автоматическое обновление баз.
|
||||
3. Замена ClamAV в Message Safety локальным fail-closed scan-broker.
|
||||
|
||||
KESL 12.4 commercial не заявляется как сертифицированная ФСТЭК сборка.
|
||||
|
||||
## 0. Approvals и stop conditions
|
||||
|
||||
До окна зафиксировать change ID, оператора, Security/Service/Operations approvers,
|
||||
hostname, ОС/ядро, package version/hash/source, лицензию, release SHA и rollback.
|
||||
EULA, Privacy Policy и KSN согласовать отдельно. Activation code, ключи и secrets
|
||||
не помещать в репозиторий, историю shell, chat или evidence.
|
||||
|
||||
Остановиться, если:
|
||||
|
||||
- ОС/архитектура/ядро отсутствуют в матрице KESL 12.4;
|
||||
- установлен другой AV или неизвестный KESL;
|
||||
- свободно менее 10 ГБ, нет 4 ГБ swap либо baseline ВМ2 уже деградирован;
|
||||
- KESL+ClamAV coexistence оставляет менее 2 ГБ available memory, вызывает
|
||||
устойчивый swap IO, OOM/restart, IO wait >10% за 5 минут;
|
||||
- p95 Safety/Redis вырос более чем на 20%, queue age >30 секунд;
|
||||
- package hash/source не подтверждены, лицензия недействительна или базы не загружены.
|
||||
|
||||
Не совмещать с deploy, миграциями, backup, TLS renewal, Docker restart или reboot.
|
||||
|
||||
## 1. Read-only baseline
|
||||
|
||||
```sh
|
||||
date -Is
|
||||
hostnamectl
|
||||
uname -a
|
||||
dpkg --print-architecture
|
||||
free -h
|
||||
swapon --show
|
||||
df -hT / /var/lib/docker /tmp
|
||||
df -ih / /var/lib/docker /tmp
|
||||
systemctl is-active docker fail2ban ufw han-processing.service
|
||||
dpkg-query -W -f='${Package}\t${Version}\t${Status}\n' \
|
||||
kesl kesl-gui kav4fs 2>/dev/null || true
|
||||
/usr/local/sbin/han-vm2-compose ps
|
||||
docker stats --no-stream
|
||||
iptables -S HAN-CHAT-VM2
|
||||
ufw status verbose
|
||||
```
|
||||
|
||||
Сохранить baseline CPU/RAM/swap/IO, Safety latency/error/queue, Redis latency,
|
||||
container restart/OOM, OTEL queue и внешний/private smoke. Получить фактические
|
||||
mountpoint только для `redis-safety-data` и `otel-queue` через
|
||||
`docker volume inspect`; prefix Compose не угадывать.
|
||||
|
||||
## 2. Дистрибутив и установка
|
||||
|
||||
Использовать amd64 DEB KESL 12.4 из официального канала. Хеши пилота ВМ1 не
|
||||
считать универсальными: записать фактические SHA-256 DEB/ISO и, если поставщик
|
||||
публикует ФИКС/ГОСТ, проверить соответствующим алгоритмом.
|
||||
|
||||
Проверка файла:
|
||||
```
|
||||
apt install rhash
|
||||
ISO='/var/lib/han-deploy/incoming/049-16-d-01.iso'
|
||||
command -v rhash
|
||||
rhash --gost "$ISO"
|
||||
```
|
||||
|
||||
Должно быть 6a94b16afad211e8b9be5ec86f5379184f2b3a9e5869843fe763e2796e5ac1d3
|
||||
|
||||
Если совпадёт, извлеките DEB:
|
||||
```
|
||||
install -d -m 0700 -o root -g root \
|
||||
/root/kesl-install /mnt/kesl-iso
|
||||
|
||||
mount -o ro,loop "$ISO" /mnt/kesl-iso
|
||||
|
||||
find /mnt/kesl-iso -type f \
|
||||
\( -iname 'kesl_*_amd64.deb' -o -iname '*.deb' \) |
|
||||
sort
|
||||
```
|
||||
|
||||
Затем:
|
||||
```
|
||||
DEB='/mnt/kesl-iso/kesl/kesl_12.4.0-1225_amd64.deb'
|
||||
|
||||
dpkg-deb -f "$DEB" Package Version Architecture
|
||||
sha256sum "$DEB"
|
||||
|
||||
install -m 0600 -o root -g root \
|
||||
"$DEB" /root/kesl-install/kesl.deb
|
||||
|
||||
umount /mnt/kesl-iso
|
||||
rmdir /mnt/kesl-iso
|
||||
```
|
||||
|
||||
Ожидаемые свойства: kesl, версия 12.4.x, архитектура amd64. После этого можно выполнять:
|
||||
```
|
||||
apt-get install /root/kesl-install/kesl.deb
|
||||
```
|
||||
|
||||
Создать `/root/kesl-install/autoinstall.ini` mode `0600`:
|
||||
|
||||
```ini
|
||||
KSVLA_MODE=No
|
||||
ENDPOINT_AGENT_MODE=No
|
||||
EULA_AGREED=<Yes_AFTER_APPROVAL>
|
||||
PRIVACY_POLICY_AGREED=<Yes_AFTER_APPROVAL>
|
||||
USE_KSN=<Yes_OR_No_AFTER_APPROVAL>
|
||||
GROUP_CLEAN=Yes
|
||||
LOCALE=ru_RU.UTF-8
|
||||
INSTALL_LICENSE=None
|
||||
UPDATER_SOURCE=KLServers
|
||||
UPDATE_EXECUTE=No
|
||||
KERNEL_SRCS_INSTALL=No
|
||||
USE_GUI=No
|
||||
CONFIGURE_SELINUX=No
|
||||
DISABLE_PROTECTION=Yes
|
||||
INTERCEPTOR_MODE=UseFanotify
|
||||
ENABLE_TRACES_ON_FIRST_STARTUP=No
|
||||
```
|
||||
|
||||
```sh
|
||||
chmod 0600 /root/kesl-install/autoinstall.ini
|
||||
/opt/kaspersky/kesl/bin/kesl-setup.pl \
|
||||
--autoinstall=/root/kesl-install/autoinstall.ini
|
||||
systemctl --no-pager status kesl
|
||||
kesl-control --app-info --json
|
||||
kesl-control --get-task-list
|
||||
```
|
||||
|
||||
При `INSTALL_LICENSE=None` setup может вернуть 71; не запускать повторно, если
|
||||
`kesl.service` active и компоненты защиты остановлены. Активацию выполнить без
|
||||
history, затем удалить переменную:
|
||||
|
||||
```sh
|
||||
set +o history
|
||||
unset HISTFILE
|
||||
read -rsp 'KESL activation code: ' KESL_CODE; echo
|
||||
kesl-control --add-active-key "$KESL_CODE"
|
||||
unset KESL_CODE
|
||||
set -o history
|
||||
kesl-control -L --query
|
||||
```
|
||||
|
||||
## 3. Resource budget до первой проверки
|
||||
|
||||
Defaults `ScanMemoryLimit=8192` и `MaxMemory=auto` не принимать на ВМ2 8 ГБ.
|
||||
Стартовый кандидат — `ScanMemoryLimit=1024`, `MaxMemory=2048MB`; окончательные
|
||||
значения определяются baseline и coexistence с ещё работающим ClamAV.
|
||||
|
||||
```sh
|
||||
kesl-control --get-app-settings \
|
||||
--file /root/kesl-install/app-settings.before.ini
|
||||
kesl-control --set-app-settings \
|
||||
UseOnDemandCPULimit=Yes OnDemandCPULimit=<APPROVED_VALUE>
|
||||
```
|
||||
|
||||
`ScanMemoryLimit`/`MaxMemory` менять только по официальной процедуре в секции
|
||||
`[General]` `/var/opt/kaspersky/kesl/common/kesl.ini` при остановленном KESL.
|
||||
|
||||
## 4. АВЗ.2 — базы
|
||||
|
||||
ID подтвердить через `--get-task-list`.
|
||||
|
||||
```sh
|
||||
kesl-control --get-settings 6
|
||||
kesl-control --start-task 6 -W
|
||||
kesl-control --get-task-state 6
|
||||
kesl-control --app-info --json
|
||||
START="$(LC_ALL=C date +'%Y/%b/%d %H:%M:%S;1')"
|
||||
kesl-control --set-schedule 6 RuleType=Hourly \
|
||||
"StartTime=${START}" RunMissedStartRules=No RandomInterval=0
|
||||
kesl-control --get-schedule 6
|
||||
```
|
||||
|
||||
До приёмки наблюдать хотя бы один автоматический hourly cycle. Alert: ошибка
|
||||
Update, базы старше 240 часов, базы не загружены, лицензия недействительна.
|
||||
|
||||
## 5. Host protection — АВЗ.1
|
||||
|
||||
Добавить только фактические hot-data mountpoint Redis Safety и OTEL queue.
|
||||
Не исключать `/var/lib/docker`, release, incoming, secrets, `/tmp` или `/`.
|
||||
Не исключать staging broker `/var/lib/han-kesl-scan/staging`: исключение task 1
|
||||
распространяется и на явный `Scan_File`, поэтому файл будет пропущен. Если
|
||||
исключение уже добавлено, удалить его до проверки broker:
|
||||
|
||||
```sh
|
||||
kesl-control --set-settings 1 \
|
||||
--del-exclusion /var/lib/han-kesl-scan/staging
|
||||
```
|
||||
|
||||
```sh
|
||||
kesl-control --get-settings 1 \
|
||||
--file /root/kesl-install/file-threat.before.ini
|
||||
kesl-control --set-settings 1 \
|
||||
--add-exclusion <REDIS_SAFETY_DATA_MOUNTPOINT>
|
||||
kesl-control --set-settings 1 \
|
||||
--add-exclusion <OTEL_QUEUE_MOUNTPOINT>
|
||||
kesl-control --start-task 1
|
||||
kesl-control --set-settings 1 \
|
||||
ActionOnThreat=DisinfectDeleteIfNotPossible ScanArchived=No
|
||||
kesl-control --get-task-state 1
|
||||
kesl-control --get-settings 1
|
||||
```
|
||||
|
||||
Проверять события, health, firewall и ресурсы каждые 15 минут. При деградации:
|
||||
`kesl-control --stop-task 1`.
|
||||
|
||||
## 6. Scan-broker
|
||||
|
||||
После активации release повторить host setup для установки root-owned unit,
|
||||
затем проверить файлы и включить socket:
|
||||
|
||||
```sh
|
||||
systemd-analyze verify \
|
||||
/etc/systemd/system/han-kesl-scan-broker.service \
|
||||
/etc/systemd/system/han-kesl-scan-broker.socket
|
||||
systemctl enable --now han-kesl-scan-broker.socket
|
||||
systemctl is-active kesl han-kesl-scan-broker.socket
|
||||
stat -c '%U:%G:%a %n' /run/han-kesl/scan.sock
|
||||
/usr/local/libexec/han-kesl-scan-broker --probe
|
||||
journalctl --since '-10 min' --no-pager -u han-kesl-scan-broker.service
|
||||
```
|
||||
|
||||
Ожидается `root:han-message-safety:660`. Broker не имеет TCP port, принимает
|
||||
не более 5 MiB, не получает S3 key/имя/ПД, запускает фиксированный Scan_File
|
||||
`Inform` и возвращает только verdict/version. Неизвестный вывод KESL — error.
|
||||
Для `Scan_File` код возврата `72` означает обнаружение угрозы и принимается
|
||||
только вместе с итогом `detected > 0`; любое противоречие обрабатывается как
|
||||
`scanner_unavailable`.
|
||||
Broker service намеренно не содержит `Requires=kesl.service`: иначе обращение
|
||||
к socket автоматически запустит остановленный KESL и сделает fail-closed
|
||||
проверку недостоверной. Порядок запуска обеспечивается `After=kesl.service`.
|
||||
|
||||
## 7. Приёмка и cutover
|
||||
|
||||
EICAR разрешён только письменно Security и не хранится в repository/evidence.
|
||||
Отдельно выполнить:
|
||||
|
||||
1. host on-access: обнаружение и реагирование task 1;
|
||||
2. broker: clean → clean, EICAR → infected, остановленный KESL → error;
|
||||
3. 5 concurrent broker slots и corpus 2 file/s; p95 final ≤60 s, public wait
|
||||
≤300 s, queue age p95 ≤5 s;
|
||||
4. Message Safety: clean `200`, malware `403`, scanner error `503`, verdict sticky;
|
||||
5. smoke ВМ2/ВМ1, firewall unchanged, 24 часа без OOM/restart/5xx/degradation.
|
||||
|
||||
Только после gates активировать новую config version, release без
|
||||
`clamd`/`freshclam`, затем подтвердить отсутствие их контейнеров/volumes и
|
||||
закрыть старый signature-CDN egress.
|
||||
|
||||
## 8. Evidence
|
||||
|
||||
Заполнить `EVIDENCE.AVZ.ru.md` без activation code, secrets, file bytes, S3 key,
|
||||
полного checksum и EICAR. Сохранить безопасные excerpts:
|
||||
|
||||
```sh
|
||||
systemctl is-active kesl han-kesl-scan-broker.socket
|
||||
kesl-control --app-info --json
|
||||
kesl-control --get-task-state 1
|
||||
kesl-control --get-settings 1
|
||||
kesl-control --get-schedule 6
|
||||
kesl-control -E --query -n 100 --reverse --json
|
||||
/usr/local/sbin/han-vm2-compose ps
|
||||
```
|
||||
|
||||
## 9. Rollback
|
||||
|
||||
Application rollback: вернуть предыдущие совместимые image/config/release с
|
||||
`clamd`, не создавать новый task id для уже pending request. Host KESL оставить,
|
||||
если он не причина инцидента.
|
||||
|
||||
Если проблема в host protection:
|
||||
|
||||
```sh
|
||||
kesl-control --stop-task 1
|
||||
# если KESL не отвечает:
|
||||
systemctl stop kesl
|
||||
```
|
||||
|
||||
`apt-get purge kesl` — только по решению approver. Не выполнять compose
|
||||
`down -v`, не удалять Redis/OTEL volumes и не менять firewall ради обхода.
|
||||
|
||||
После rollback повторить smoke, resource checks и зафиксировать incident.
|
||||
@@ -0,0 +1,43 @@
|
||||
[Unit]
|
||||
Description=HAN local KESL scan broker
|
||||
Requires=han-kesl-scan-broker.socket
|
||||
After=kesl.service
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=root
|
||||
Group=han-message-safety
|
||||
ExecStart=/usr/local/libexec/han-kesl-scan-broker
|
||||
Environment=HAN_KESL_SCAN_TIMEOUT_SEC=60
|
||||
Environment=HAN_KESL_SCAN_CONCURRENCY=5
|
||||
StateDirectory=han-kesl-scan
|
||||
StateDirectoryMode=0700
|
||||
UMask=0077
|
||||
NoNewPrivileges=yes
|
||||
PrivateTmp=yes
|
||||
PrivateDevices=yes
|
||||
ProtectSystem=strict
|
||||
ProtectHome=yes
|
||||
ProtectKernelTunables=yes
|
||||
ProtectKernelModules=yes
|
||||
ProtectKernelLogs=yes
|
||||
ProtectControlGroups=yes
|
||||
ProtectClock=yes
|
||||
ProtectHostname=yes
|
||||
RestrictAddressFamilies=AF_UNIX
|
||||
RestrictNamespaces=yes
|
||||
RestrictRealtime=yes
|
||||
RestrictSUIDSGID=yes
|
||||
LockPersonality=yes
|
||||
MemoryDenyWriteExecute=yes
|
||||
SystemCallArchitectures=native
|
||||
ReadWritePaths=/var/lib/han-kesl-scan
|
||||
CapabilityBoundingSet=
|
||||
AmbientCapabilities=
|
||||
LimitCORE=0
|
||||
LimitNOFILE=128
|
||||
TasksMax=64
|
||||
MemoryMax=256M
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@@ -0,0 +1,15 @@
|
||||
[Unit]
|
||||
Description=HAN KESL scan broker socket
|
||||
Before=han-processing.service
|
||||
|
||||
[Socket]
|
||||
ListenStream=/run/han-kesl/scan.sock
|
||||
SocketUser=root
|
||||
SocketGroup=han-message-safety
|
||||
SocketMode=0660
|
||||
DirectoryMode=0750
|
||||
RemoveOnStop=true
|
||||
Service=han-kesl-scan-broker.service
|
||||
|
||||
[Install]
|
||||
WantedBy=sockets.target
|
||||
@@ -0,0 +1 @@
|
||||
d /run/han-kesl 0750 root han-message-safety -
|
||||
@@ -0,0 +1,310 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Local, fail-closed bridge between Message Safety and host KESL."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import socket
|
||||
import struct
|
||||
import sys
|
||||
import tempfile
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
MAX_HEADER_BYTES = 4096
|
||||
MAX_FILE_BYTES = 5 * 1024 * 1024
|
||||
MAX_RESPONSE_BYTES = 16 * 1024
|
||||
DEFAULT_TIMEOUT_SECONDS = 60.0
|
||||
KESL_CONTROL = Path("/opt/kaspersky/kesl/bin/kesl-control")
|
||||
STAGING_DIR = Path("/var/lib/han-kesl-scan/staging")
|
||||
|
||||
_SUMMARY_PATTERNS = {
|
||||
"scanned": re.compile(
|
||||
r"(?im)^\s*(?:Scanned objects|Objects scanned|Проверенные объекты)"
|
||||
r"\s*:\s*(\d+)\s*$"
|
||||
),
|
||||
"detected": re.compile(
|
||||
r"(?im)^\s*(?:Total detected objects|Всего обнаружено объектов)"
|
||||
r"\s*:\s*(\d+)\s*$"
|
||||
),
|
||||
"errors": re.compile(
|
||||
r"(?im)^\s*(?:Scan errors|Ошибки проверки)\s*:\s*(\d+)\s*$"
|
||||
),
|
||||
"skipped": re.compile(
|
||||
r"(?im)^\s*(?:Skipped objects|Objects skipped|Пропущено объектов)"
|
||||
r"\s*:\s*(\d+)\s*$"
|
||||
),
|
||||
}
|
||||
_THREAT_RE = re.compile(r"(?im)^\s*(?:Threat|Detect name)\s*:\s*(.{1,256})\s*$")
|
||||
|
||||
|
||||
class ProtocolError(ValueError):
|
||||
pass
|
||||
|
||||
|
||||
class KeslError(RuntimeError):
|
||||
pass
|
||||
|
||||
|
||||
def _json_bytes(payload: dict[str, Any]) -> bytes:
|
||||
body = json.dumps(payload, ensure_ascii=True, separators=(",", ":")).encode()
|
||||
if len(body) > MAX_RESPONSE_BYTES:
|
||||
raise KeslError("response exceeds protocol limit")
|
||||
return struct.pack(">I", len(body)) + body
|
||||
|
||||
|
||||
async def _read_frame(reader: asyncio.StreamReader) -> tuple[dict[str, Any], bytes]:
|
||||
(header_size,) = struct.unpack(">I", await reader.readexactly(4))
|
||||
if header_size < 2 or header_size > MAX_HEADER_BYTES:
|
||||
raise ProtocolError("invalid header size")
|
||||
try:
|
||||
header = json.loads((await reader.readexactly(header_size)).decode("utf-8"))
|
||||
except (UnicodeDecodeError, json.JSONDecodeError) as exc:
|
||||
raise ProtocolError("invalid header") from exc
|
||||
if not isinstance(header, dict) or set(header) - {"op", "size"}:
|
||||
raise ProtocolError("invalid header fields")
|
||||
operation = header.get("op")
|
||||
if operation == "status":
|
||||
if "size" in header:
|
||||
raise ProtocolError("status request cannot contain a body")
|
||||
return header, b""
|
||||
if operation != "scan" or not isinstance(header.get("size"), int):
|
||||
raise ProtocolError("unsupported operation")
|
||||
size = header["size"]
|
||||
if isinstance(size, bool) or size < 1 or size > MAX_FILE_BYTES:
|
||||
raise ProtocolError("invalid file size")
|
||||
return header, await reader.readexactly(size)
|
||||
|
||||
|
||||
def parse_scan_output(output: str, *, exit_code: int = 0) -> tuple[str, str | None]:
|
||||
"""Parse the documented KESL Scan_File summary, rejecting format drift."""
|
||||
if exit_code not in (0, 72):
|
||||
raise KeslError(f"unexpected KESL scan exit code: {exit_code}")
|
||||
summary: dict[str, int] = {}
|
||||
for name, pattern in _SUMMARY_PATTERNS.items():
|
||||
match = pattern.search(output)
|
||||
if not match:
|
||||
raise KeslError(f"KESL scan summary field is missing: {name}")
|
||||
summary[name] = int(match.group(1))
|
||||
if summary["scanned"] < 1:
|
||||
raise KeslError("KESL did not scan the submitted object")
|
||||
if summary["errors"] or summary["skipped"]:
|
||||
raise KeslError("KESL scan completed with errors or skipped objects")
|
||||
detected = summary["detected"]
|
||||
if (exit_code == 72) != (detected > 0):
|
||||
raise KeslError("KESL exit code contradicts scan summary")
|
||||
if detected == 0:
|
||||
return "clean", None
|
||||
threat = _THREAT_RE.search(output)
|
||||
return "infected", threat.group(1).strip() if threat else "detected"
|
||||
|
||||
|
||||
def _flatten_json(value: Any, prefix: str = "") -> dict[str, Any]:
|
||||
result: dict[str, Any] = {}
|
||||
if isinstance(value, dict):
|
||||
for key, item in value.items():
|
||||
normalized = re.sub(r"[^\w]+", "_", str(key).casefold()).strip("_")
|
||||
result.update(_flatten_json(item, f"{prefix}_{normalized}".strip("_")))
|
||||
else:
|
||||
result[prefix] = value
|
||||
return result
|
||||
|
||||
|
||||
def parse_app_info(output: str) -> dict[str, Any]:
|
||||
try:
|
||||
flattened = _flatten_json(json.loads(output))
|
||||
except json.JSONDecodeError as exc:
|
||||
raise KeslError("invalid KESL app-info JSON") from exc
|
||||
|
||||
def find(*suffixes: str) -> Any:
|
||||
for suffix in suffixes:
|
||||
for key, value in flattened.items():
|
||||
if key == suffix or key.endswith(f"_{suffix}"):
|
||||
return value
|
||||
raise KeslError(f"KESL app-info field is missing: {suffixes[0]}")
|
||||
|
||||
version = str(find("version", "application_version", "версия")).strip()
|
||||
databases_loaded = find(
|
||||
"databases_loaded",
|
||||
"application_databases_loaded",
|
||||
"базы_приложения_загружены",
|
||||
)
|
||||
databases_date = str(
|
||||
find(
|
||||
"databases_date",
|
||||
"last_release_date_of_databases",
|
||||
"database_date",
|
||||
"дата_последнего_выпуска_баз_приложения",
|
||||
)
|
||||
).strip()
|
||||
license_info = str(
|
||||
find(
|
||||
"license_info",
|
||||
"license_status",
|
||||
"key_status",
|
||||
"license_information",
|
||||
"информация_о_лицензии_приложения",
|
||||
)
|
||||
).strip()
|
||||
if databases_loaded not in (True, "Yes", "yes", "true", "Да", "да", 1):
|
||||
raise KeslError("KESL databases are not loaded")
|
||||
if not version or not databases_date or databases_date.upper() == "N/A":
|
||||
raise KeslError("KESL version or database date is unavailable")
|
||||
if "valid" not in license_info.lower() and "действ" not in license_info.lower():
|
||||
raise KeslError("KESL license is not valid")
|
||||
signature_source = f"{version}\0{databases_date}".encode()
|
||||
return {
|
||||
"status": "ready",
|
||||
"engine_version": version,
|
||||
"databases_date": databases_date,
|
||||
"signatures_version": "sha256:" + hashlib.sha256(signature_source).hexdigest(),
|
||||
}
|
||||
|
||||
|
||||
async def _run_kesl(
|
||||
*arguments: str,
|
||||
timeout: float,
|
||||
allowed_returncodes: frozenset[int] = frozenset({0}),
|
||||
) -> tuple[str, int]:
|
||||
if not KESL_CONTROL.is_file():
|
||||
raise KeslError("KESL control utility is unavailable")
|
||||
process = await asyncio.create_subprocess_exec(
|
||||
str(KESL_CONTROL),
|
||||
*arguments,
|
||||
stdout=asyncio.subprocess.PIPE,
|
||||
stderr=asyncio.subprocess.PIPE,
|
||||
env={**os.environ, "LC_ALL": "C", "LANG": "C", "LANGUAGE": "C"},
|
||||
)
|
||||
try:
|
||||
stdout, stderr = await asyncio.wait_for(process.communicate(), timeout)
|
||||
except TimeoutError:
|
||||
process.kill()
|
||||
await process.wait()
|
||||
raise KeslError("KESL command timed out") from None
|
||||
if process.returncode not in allowed_returncodes:
|
||||
detail = stderr.decode("utf-8", "replace").strip()[:256]
|
||||
raise KeslError(f"KESL command failed: {detail or process.returncode}")
|
||||
return stdout.decode("utf-8", "replace"), process.returncode
|
||||
|
||||
|
||||
class Broker:
|
||||
def __init__(self, *, timeout: float, concurrency: int) -> None:
|
||||
self.timeout = timeout
|
||||
self.slots = asyncio.Semaphore(concurrency)
|
||||
|
||||
async def status(self) -> dict[str, Any]:
|
||||
output, _ = await _run_kesl(
|
||||
"--app-info", "--json", timeout=min(self.timeout, 10.0)
|
||||
)
|
||||
return parse_app_info(output)
|
||||
|
||||
async def scan(self, body: bytes) -> dict[str, Any]:
|
||||
async with self.slots:
|
||||
status_before = await self.status()
|
||||
STAGING_DIR.mkdir(mode=0o700, parents=True, exist_ok=True)
|
||||
descriptor, name = tempfile.mkstemp(prefix="scan-", dir=STAGING_DIR)
|
||||
path = Path(name)
|
||||
try:
|
||||
os.fchmod(descriptor, 0o600)
|
||||
with os.fdopen(descriptor, "wb", closefd=True) as target:
|
||||
target.write(body)
|
||||
target.flush()
|
||||
os.fsync(target.fileno())
|
||||
output, exit_code = await _run_kesl(
|
||||
"--scan-file",
|
||||
str(path),
|
||||
"--action",
|
||||
"Inform",
|
||||
timeout=self.timeout,
|
||||
allowed_returncodes=frozenset({0, 72}),
|
||||
)
|
||||
verdict, threat = parse_scan_output(output, exit_code=exit_code)
|
||||
status_after = await self.status()
|
||||
if (
|
||||
status_before["signatures_version"]
|
||||
!= status_after["signatures_version"]
|
||||
):
|
||||
raise KeslError("KESL databases changed during scan")
|
||||
return {
|
||||
"status": "completed",
|
||||
"verdict": verdict,
|
||||
"threat": threat,
|
||||
"engine_version": status_after["engine_version"],
|
||||
"signatures_version": status_after["signatures_version"],
|
||||
}
|
||||
finally:
|
||||
path.unlink(missing_ok=True)
|
||||
|
||||
async def handle(self, reader: asyncio.StreamReader, writer: asyncio.StreamWriter) -> None:
|
||||
try:
|
||||
header, body = await asyncio.wait_for(_read_frame(reader), 10.0)
|
||||
result = await (self.status() if header["op"] == "status" else self.scan(body))
|
||||
except (ProtocolError, asyncio.IncompleteReadError, TimeoutError):
|
||||
result = {"status": "error", "error": "invalid_request"}
|
||||
except KeslError as exc:
|
||||
print(f"KESL scanner unavailable: {exc}", file=sys.stderr, flush=True)
|
||||
result = {"status": "error", "error": "scanner_unavailable"}
|
||||
except Exception:
|
||||
result = {"status": "error", "error": "internal_error"}
|
||||
writer.write(_json_bytes(result))
|
||||
await writer.drain()
|
||||
writer.close()
|
||||
await writer.wait_closed()
|
||||
|
||||
|
||||
def _activation_socket() -> socket.socket:
|
||||
if int(os.environ.get("LISTEN_FDS", "0")) != 1 or os.getpid() != int(
|
||||
os.environ.get("LISTEN_PID", "0")
|
||||
):
|
||||
raise SystemExit("exactly one systemd activation socket is required")
|
||||
descriptor = socket.fromfd(3, socket.AF_UNIX, socket.SOCK_STREAM)
|
||||
descriptor.setblocking(False)
|
||||
return descriptor
|
||||
|
||||
|
||||
async def serve() -> None:
|
||||
timeout = float(os.environ.get("HAN_KESL_SCAN_TIMEOUT_SEC", DEFAULT_TIMEOUT_SECONDS))
|
||||
concurrency = int(os.environ.get("HAN_KESL_SCAN_CONCURRENCY", "5"))
|
||||
if not 1 <= concurrency <= 16 or not 1 <= timeout <= 300:
|
||||
raise SystemExit("invalid broker limits")
|
||||
server = await asyncio.start_unix_server(
|
||||
Broker(timeout=timeout, concurrency=concurrency).handle,
|
||||
sock=_activation_socket(),
|
||||
limit=MAX_FILE_BYTES + MAX_HEADER_BYTES + 4,
|
||||
)
|
||||
async with server:
|
||||
await server.serve_forever()
|
||||
|
||||
|
||||
async def probe() -> None:
|
||||
reader, writer = await asyncio.wait_for(
|
||||
asyncio.open_unix_connection("/run/han-kesl/scan.sock"), 5.0
|
||||
)
|
||||
try:
|
||||
header = json.dumps({"op": "status"}, separators=(",", ":")).encode()
|
||||
writer.write(struct.pack(">I", len(header)) + header)
|
||||
await writer.drain()
|
||||
(size,) = struct.unpack(">I", await asyncio.wait_for(reader.readexactly(4), 10.0))
|
||||
if size < 2 or size > MAX_RESPONSE_BYTES:
|
||||
raise SystemExit("invalid broker response")
|
||||
response = json.loads((await asyncio.wait_for(reader.readexactly(size), 10.0)).decode())
|
||||
if response.get("status") != "ready":
|
||||
print(json.dumps(response, ensure_ascii=True, sort_keys=True), file=sys.stderr)
|
||||
raise SystemExit("KESL broker is not ready")
|
||||
print(json.dumps(response, ensure_ascii=True, sort_keys=True))
|
||||
finally:
|
||||
writer.close()
|
||||
await writer.wait_closed()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if not sys.argv[1:]:
|
||||
asyncio.run(serve())
|
||||
elif sys.argv[1:] == ["--probe"]:
|
||||
asyncio.run(probe())
|
||||
else:
|
||||
raise SystemExit("usage: han-kesl-scan-broker [--probe]")
|
||||
@@ -0,0 +1,62 @@
|
||||
# Reference decisions for KESL 12.4 on HAN Processing VM2.
|
||||
# Do not import this partial file into KESL. Export the installed-build
|
||||
# settings, review them, and apply individual values from RUNBOOK.KESL.ru.md.
|
||||
|
||||
[deployment]
|
||||
product_major_minor=12.4
|
||||
mode=standard_standalone
|
||||
gui=disabled
|
||||
update_source=KLServers
|
||||
interceptor=fanotify
|
||||
network_features=disabled
|
||||
ksn=<Yes_OR_No_AFTER_SECURITY_AND_LEGAL_APPROVAL>
|
||||
|
||||
[resource_budget]
|
||||
# VM2 starts at 8 GiB RAM. Values require measured baseline and coexistence gate.
|
||||
scan_memory_limit_mb=<1024_OR_APPROVED_VALUE>
|
||||
max_memory=<2048MB_OR_APPROVED_VALUE>
|
||||
use_on_demand_cpu_limit=Yes
|
||||
on_demand_cpu_limit_percent=<APPROVED_VALUE>
|
||||
|
||||
[update_task_6]
|
||||
rule_type=Hourly
|
||||
required_result=completed_successfully
|
||||
required_bases_loaded=Yes
|
||||
stale_bases_alert_hours=240
|
||||
|
||||
[file_threat_protection_task_1]
|
||||
steady_state=Started
|
||||
action_on_threat=DisinfectDeleteIfNotPossible
|
||||
scan_archived=No
|
||||
|
||||
[message_safety_broker]
|
||||
socket=/run/han-kesl/scan.sock
|
||||
max_file_bytes=5242880
|
||||
scan_action=Inform
|
||||
fail_closed=Yes
|
||||
staging=/var/lib/han-kesl-scan/staging
|
||||
# Do not add staging to ExcludedFromScanScope: KESL Scan_File can skip it.
|
||||
staging_exclusion=forbidden
|
||||
explicit_scan_before_allow=required
|
||||
|
||||
[allowed_hot_data_exclusions]
|
||||
item_0000=<REDIS_SAFETY_DATA_MOUNTPOINT>
|
||||
item_0001=<OTEL_QUEUE_MOUNTPOINT>
|
||||
|
||||
[forbidden_broad_exclusions]
|
||||
item_0000=/var/lib/docker
|
||||
item_0001=/var/lib/docker/overlay2
|
||||
item_0002=/opt/han-chat
|
||||
item_0003=/var/lib/han-deploy/incoming
|
||||
item_0004=/etc/han
|
||||
item_0005=/run/han-chat
|
||||
item_0006=/tmp
|
||||
item_0007=/
|
||||
|
||||
[acceptance]
|
||||
host_on_access_eicar=required
|
||||
broker_clean_eicar_error=required
|
||||
automatic_hourly_update_observed=required
|
||||
message_safety_200_403_503=required
|
||||
performance_two_files_per_second=required
|
||||
resource_comparison_24h=required
|
||||
@@ -0,0 +1,140 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import importlib.util
|
||||
import json
|
||||
import struct
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
import pytest
|
||||
|
||||
MODULE_PATH = Path(__file__).with_name("kesl_scan_broker.py")
|
||||
SPEC = importlib.util.spec_from_file_location("kesl_scan_broker", MODULE_PATH)
|
||||
assert SPEC and SPEC.loader
|
||||
broker = importlib.util.module_from_spec(SPEC)
|
||||
sys.modules[SPEC.name] = broker
|
||||
SPEC.loader.exec_module(broker)
|
||||
|
||||
|
||||
def test_scan_output_is_strict_and_fail_closed() -> None:
|
||||
clean = (
|
||||
"Scanned objects: 1\n"
|
||||
"Total detected objects: 0\n"
|
||||
"Scan errors: 0\n"
|
||||
"Skipped objects: 0\n"
|
||||
)
|
||||
assert broker.parse_scan_output(clean) == ("clean", None)
|
||||
assert broker.parse_scan_output(
|
||||
"Threat: EICAR-Test-File\n"
|
||||
"Scanned objects: 1\n"
|
||||
"Total detected objects: 1\n"
|
||||
"Scan errors: 0\n"
|
||||
"Skipped objects: 0\n",
|
||||
exit_code=72,
|
||||
) == ("infected", "EICAR-Test-File")
|
||||
with pytest.raises(broker.KeslError):
|
||||
broker.parse_scan_output("scan allegedly completed")
|
||||
|
||||
|
||||
def test_scan_output_accepts_russian_summary_and_rejects_skipped_file() -> None:
|
||||
clean = (
|
||||
"Проверенные объекты: 1\n"
|
||||
"Всего обнаружено объектов: 0\n"
|
||||
"Ошибки проверки: 0\n"
|
||||
"Пропущено объектов: 0\n"
|
||||
)
|
||||
assert broker.parse_scan_output(clean) == ("clean", None)
|
||||
|
||||
skipped = clean.replace("Проверенные объекты: 1", "Проверенные объекты: 0").replace(
|
||||
"Пропущено объектов: 0", "Пропущено объектов: 1"
|
||||
)
|
||||
with pytest.raises(broker.KeslError):
|
||||
broker.parse_scan_output(skipped)
|
||||
|
||||
|
||||
def test_scan_output_rejects_exit_code_summary_mismatch() -> None:
|
||||
infected = (
|
||||
"Scanned objects: 1\n"
|
||||
"Total detected objects: 1\n"
|
||||
"Scan errors: 0\n"
|
||||
"Skipped objects: 0\n"
|
||||
)
|
||||
clean = infected.replace("Total detected objects: 1", "Total detected objects: 0")
|
||||
with pytest.raises(broker.KeslError):
|
||||
broker.parse_scan_output(infected, exit_code=0)
|
||||
with pytest.raises(broker.KeslError):
|
||||
broker.parse_scan_output(clean, exit_code=72)
|
||||
|
||||
|
||||
def test_app_info_builds_versioned_signature() -> None:
|
||||
raw = json.dumps(
|
||||
{
|
||||
"product info": {
|
||||
"version": "12.4.0.1225",
|
||||
"databases_loaded": True,
|
||||
"databases_date": "2026-09-07T11:25:00+00:00",
|
||||
"license_info": "The key is valid",
|
||||
}
|
||||
}
|
||||
)
|
||||
status = broker.parse_app_info(raw)
|
||||
assert status["status"] == "ready"
|
||||
assert status["engine_version"] == "12.4.0.1225"
|
||||
assert str(status["signatures_version"]).startswith("sha256:")
|
||||
|
||||
|
||||
def test_app_info_accepts_kesl_12_4_control_json() -> None:
|
||||
raw = json.dumps(
|
||||
{
|
||||
"Application databases loaded": "Yes",
|
||||
"Application license information": "The key is valid",
|
||||
"Last release date of databases": "2026-09-07 15:41:00",
|
||||
"Version": "12.4.0.1225",
|
||||
}
|
||||
)
|
||||
status = broker.parse_app_info(raw)
|
||||
assert status["status"] == "ready"
|
||||
assert status["engine_version"] == "12.4.0.1225"
|
||||
assert status["databases_date"] == "2026-09-07 15:41:00"
|
||||
|
||||
|
||||
def test_app_info_accepts_russian_kesl_12_4_control_json() -> None:
|
||||
raw = json.dumps(
|
||||
{
|
||||
"Базы приложения загружены": "Да",
|
||||
"Версия": "12.4.0.1225",
|
||||
"Дата последнего выпуска баз приложения": "2026-09-07 15:41:00",
|
||||
"Информация о лицензии приложения": "Ключ действителен",
|
||||
}
|
||||
)
|
||||
status = broker.parse_app_info(raw)
|
||||
assert status["status"] == "ready"
|
||||
assert status["engine_version"] == "12.4.0.1225"
|
||||
assert status["databases_date"] == "2026-09-07 15:41:00"
|
||||
|
||||
|
||||
def test_app_info_rejects_license_and_database_failures() -> None:
|
||||
base = {
|
||||
"version": "12.4",
|
||||
"databases_loaded": True,
|
||||
"databases_date": "2026-09-07T11:25:00+00:00",
|
||||
"license_info": "The key is valid",
|
||||
}
|
||||
for key, value in (("databases_loaded", False), ("license_info", "Expired")):
|
||||
candidate = dict(base)
|
||||
candidate[key] = value
|
||||
with pytest.raises(broker.KeslError):
|
||||
broker.parse_app_info(json.dumps(candidate))
|
||||
|
||||
|
||||
def test_protocol_rejects_oversized_file() -> None:
|
||||
async def exercise() -> None:
|
||||
header = json.dumps({"op": "scan", "size": broker.MAX_FILE_BYTES + 1}).encode()
|
||||
reader = asyncio.StreamReader()
|
||||
reader.feed_data(struct.pack(">I", len(header)) + header)
|
||||
reader.feed_eof()
|
||||
with pytest.raises(broker.ProtocolError):
|
||||
await broker._read_frame(reader)
|
||||
|
||||
asyncio.run(exercise())
|
||||
@@ -21,6 +21,24 @@ fail() {
|
||||
fail "root-owned Message Safety mode file is missing; initialize standard mode"
|
||||
/usr/bin/getent group han-message-safety | /usr/bin/awk -F: '$3 == 10001 {found=1} END {exit !found}' ||
|
||||
fail "han-message-safety group with GID 10001 is missing"
|
||||
[ -x /opt/kaspersky/kesl/bin/kesl-control ] ||
|
||||
fail "KESL 12.4 control utility is missing"
|
||||
/usr/bin/systemctl is-active --quiet kesl.service ||
|
||||
fail "kesl.service is not active"
|
||||
/usr/bin/systemctl is-active --quiet han-kesl-scan-broker.socket ||
|
||||
fail "KESL scan broker socket is not active"
|
||||
[ "$(/usr/bin/stat -c '%U:%G:%a' /run/han-kesl 2>/dev/null || true)" = \
|
||||
"root:han-message-safety:750" ] ||
|
||||
fail "KESL broker runtime directory must be root:han-message-safety 0750"
|
||||
[ -S /run/han-kesl/scan.sock ] || fail "KESL scan broker Unix socket is missing"
|
||||
if [ -S /run/han-kesl/scan.sock ]; then
|
||||
[ "$(/usr/bin/stat -c '%U:%G:%a' /run/han-kesl/scan.sock)" = \
|
||||
"root:han-message-safety:660" ] ||
|
||||
fail "KESL broker socket must be root:han-message-safety 0660"
|
||||
fi
|
||||
[ -x /usr/local/libexec/han-kesl-scan-broker ] &&
|
||||
/usr/local/libexec/han-kesl-scan-broker --probe >/dev/null ||
|
||||
fail "KESL scan broker readiness probe failed"
|
||||
|
||||
public_tls_dir=/var/lib/han-chat/public-tls
|
||||
/usr/bin/getent group han-nginx-tls | /usr/bin/awk -F: '$3 == 11001 {found=1} END {exit !found}' ||
|
||||
@@ -59,7 +77,7 @@ if [ -f "$ENV_FILE" ]; then
|
||||
true|false) ;;
|
||||
*) fail "OTEL_REMOTE_TLS_INSECURE must be exactly true or false" ;;
|
||||
esac
|
||||
for image_key in MESSAGE_SAFETY_IMAGE BITRIX_SYNC_IMAGE NGINX_IMAGE REDIS_IMAGE CLAMAV_IMAGE OTEL_COLLECTOR_IMAGE REDIS_EXPORTER_IMAGE NGINX_EXPORTER_IMAGE; do
|
||||
for image_key in MESSAGE_SAFETY_IMAGE BITRIX_SYNC_IMAGE NGINX_IMAGE REDIS_IMAGE OTEL_COLLECTOR_IMAGE REDIS_EXPORTER_IMAGE NGINX_EXPORTER_IMAGE; do
|
||||
image=$(/usr/bin/awk -F= -v key="$image_key" '$1 == key {print substr($0, index($0, "=") + 1)}' "$ENV_FILE")
|
||||
echo "$image" | /usr/bin/grep -Eq '@sha256:[0-9a-f]{64}$' ||
|
||||
fail "$image_key must be pinned by sha256 digest"
|
||||
@@ -224,7 +242,11 @@ fi
|
||||
for protected in \
|
||||
"$ROOT/docker-compose.yml" \
|
||||
"$ROOT/deployment/han-message-safety-mode" \
|
||||
"$ROOT/deployment/han-processing.service"
|
||||
"$ROOT/deployment/han-processing.service" \
|
||||
"$ROOT/deployment/kesl/kesl_scan_broker.py" \
|
||||
"$ROOT/deployment/kesl/han-kesl-scan-broker.service" \
|
||||
"$ROOT/deployment/kesl/han-kesl-scan-broker.socket" \
|
||||
"$ROOT/deployment/kesl/han-kesl-scan-broker.tmpfiles.conf"
|
||||
do
|
||||
[ -f "$protected" ] || continue
|
||||
owner=$(/usr/bin/stat -c '%U:%G' "$protected")
|
||||
|
||||
@@ -603,6 +603,10 @@ install_release_helpers_if_possible() {
|
||||
install -m 0755 -o root -g root \
|
||||
"${deployment}/han-message-safety-mode" \
|
||||
/usr/local/sbin/han-message-safety-mode
|
||||
install -d -m 0755 -o root -g root /usr/local/libexec
|
||||
install -m 0750 -o root -g root \
|
||||
"${deployment}/kesl/kesl_scan_broker.py" \
|
||||
/usr/local/libexec/han-kesl-scan-broker
|
||||
install -d -m 0755 -o root -g root /etc/letsencrypt/renewal-hooks/deploy
|
||||
install -m 0755 -o root -g root \
|
||||
"${deployment}/scripts/ssl-renew-deploy-hook.sh" \
|
||||
@@ -613,6 +617,16 @@ install_release_helpers_if_possible() {
|
||||
install -m 0644 -o root -g root \
|
||||
"${deployment}/han-processing.service" \
|
||||
/etc/systemd/system/han-processing.service
|
||||
install -m 0644 -o root -g root \
|
||||
"${deployment}/kesl/han-kesl-scan-broker.service" \
|
||||
/etc/systemd/system/han-kesl-scan-broker.service
|
||||
install -m 0644 -o root -g root \
|
||||
"${deployment}/kesl/han-kesl-scan-broker.socket" \
|
||||
/etc/systemd/system/han-kesl-scan-broker.socket
|
||||
install -m 0644 -o root -g root \
|
||||
"${deployment}/kesl/han-kesl-scan-broker.tmpfiles.conf" \
|
||||
/etc/tmpfiles.d/han-kesl-scan-broker.conf
|
||||
systemd-tmpfiles --create /etc/tmpfiles.d/han-kesl-scan-broker.conf
|
||||
safety_sudoers="$(mktemp)"
|
||||
sed 's/\r$//' "${deployment}/deploy-message-safety-mode.sudoers" >"$safety_sudoers"
|
||||
chmod 0440 "$safety_sudoers"
|
||||
|
||||
@@ -22,8 +22,7 @@ x-message-safety-environment: &message-safety-environment
|
||||
MESSAGE_SAFETY_PORT: ${MESSAGE_SAFETY_PORT:-8080}
|
||||
MESSAGE_SAFETY_WORKER_CONCURRENCY: ${MESSAGE_SAFETY_WORKER_CONCURRENCY:-5}
|
||||
MESSAGE_SAFETY_DNS_RESOLVERS: ${MESSAGE_SAFETY_DNS_RESOLVERS:?set trusted DNS resolvers}
|
||||
MESSAGE_SAFETY_CLAMAV_HOST: ${MESSAGE_SAFETY_CLAMAV_HOST:-clamd}
|
||||
MESSAGE_SAFETY_CLAMAV_PORT: ${MESSAGE_SAFETY_CLAMAV_PORT:-3310}
|
||||
MESSAGE_SAFETY_ANTIVIRUS_SOCKET: /run/han-kesl/scan.sock
|
||||
MESSAGE_SAFETY_ARTIFACTS_DIR: ${MESSAGE_SAFETY_ARTIFACTS_DIR:-/app/app/artifacts}
|
||||
MESSAGE_SAFETY_MODE_FILE: /etc/han-chat/message-safety-mode.env
|
||||
PG_CA_FILE: /run/config/postgresql-ca.pem
|
||||
@@ -169,49 +168,6 @@ services:
|
||||
mem_limit: 64m
|
||||
cpus: 0.25
|
||||
|
||||
clamd:
|
||||
<<: *hardening
|
||||
image: ${CLAMAV_IMAGE:?set immutable ClamAV image digest}
|
||||
entrypoint: ["/init-unprivileged"]
|
||||
user: "100:101"
|
||||
command: ["clamd", "--foreground=true"]
|
||||
volumes:
|
||||
- clamav-signatures:/var/lib/clamav:ro
|
||||
- clamav-runtime:/run/clamav
|
||||
tmpfs:
|
||||
- /tmp:rw,noexec,nosuid,nodev,size=64m
|
||||
- /var/log/clamav:rw,noexec,nosuid,nodev,size=32m,uid=100,gid=101,mode=0750
|
||||
networks: [backend]
|
||||
expose: ["3310"]
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "clamdscan --ping 1 >/dev/null 2>&1"]
|
||||
interval: 30s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
start_period: 60s
|
||||
pids_limit: 300
|
||||
mem_limit: 2g
|
||||
cpus: 2.0
|
||||
|
||||
freshclam:
|
||||
<<: *hardening
|
||||
image: ${CLAMAV_IMAGE:?set immutable ClamAV image digest}
|
||||
entrypoint: ["/init-unprivileged"]
|
||||
user: "100:101"
|
||||
command: ["freshclam", "--daemon", "--foreground", "--checks=12"]
|
||||
volumes:
|
||||
- clamav-signatures:/var/lib/clamav
|
||||
- clamav-runtime:/run/clamav
|
||||
tmpfs:
|
||||
- /tmp:rw,noexec,nosuid,nodev,size=64m
|
||||
- /var/log/clamav:rw,noexec,nosuid,nodev,size=32m,uid=100,gid=101,mode=0750
|
||||
healthcheck:
|
||||
disable: true
|
||||
networks: [signature-egress]
|
||||
pids_limit: 100
|
||||
mem_limit: 256m
|
||||
cpus: 0.5
|
||||
|
||||
otel-queue-init:
|
||||
<<: *hardening
|
||||
image: ${REDIS_IMAGE:?set immutable Redis image digest}
|
||||
@@ -270,6 +226,7 @@ services:
|
||||
MESSAGE_SAFETY_PROCESS_ROLE: api
|
||||
volumes:
|
||||
- /etc/han-chat/message-safety-mode.env:/etc/han-chat/message-safety-mode.env:ro
|
||||
- /run/han-kesl:/run/han-kesl:ro
|
||||
- *postgres-ca-volume
|
||||
tmpfs:
|
||||
- /tmp:rw,noexec,nosuid,nodev,size=64m
|
||||
@@ -306,6 +263,7 @@ services:
|
||||
MESSAGE_SAFETY_PROCESS_ROLE: worker
|
||||
volumes:
|
||||
- /etc/han-chat/message-safety-mode.env:/etc/han-chat/message-safety-mode.env:ro
|
||||
- /run/han-kesl:/run/han-kesl:ro
|
||||
- *postgres-ca-volume
|
||||
tmpfs:
|
||||
- /tmp:rw,noexec,nosuid,nodev,size=256m
|
||||
@@ -322,8 +280,6 @@ services:
|
||||
depends_on:
|
||||
redis-safety:
|
||||
condition: service_healthy
|
||||
clamd:
|
||||
condition: service_healthy
|
||||
pids_limit: 400
|
||||
mem_limit: 2g
|
||||
cpus: 2.0
|
||||
@@ -465,13 +421,10 @@ networks:
|
||||
internal: true
|
||||
safety-egress:
|
||||
bitrix-egress:
|
||||
signature-egress:
|
||||
telemetry-egress:
|
||||
|
||||
volumes:
|
||||
redis-safety-data:
|
||||
clamav-signatures:
|
||||
clamav-runtime:
|
||||
otel-queue:
|
||||
|
||||
secrets:
|
||||
|
||||
@@ -30,8 +30,16 @@ message-safety-config activate --version 1 --approved-by security-owner
|
||||
|
||||
`docker-compose.fragment.yml` is an include fragment for the root VM2 Compose. It publishes no
|
||||
host port, runs API and worker as UID 10001 with a read-only filesystem, drops all capabilities,
|
||||
and mounts only service-specific secret files. The root project owns networks/secrets and the
|
||||
root-owned emergency mode file.
|
||||
and mounts only service-specific secret files plus the broker Unix socket
|
||||
`/run/han-kesl/scan.sock` into the worker. The root project owns networks/secrets and the
|
||||
root-owned emergency mode file. KESL 12.4 standalone and its root-owned fail-closed broker are
|
||||
host services, not Compose services; `clamd` and `freshclam` are absent from Compose.
|
||||
|
||||
The broker invokes the fixed host command `kesl-control --scan-file --action Inform`. A clean
|
||||
result may continue to allow, an infected result denies, and scanner errors, unknown output,
|
||||
timeouts or a stale KESL database remain retryable and eventually return `503` rather than
|
||||
allowing content. Runtime records `scanner_engine=kesl`; `signatures_version` is the hash of KESL
|
||||
version plus database date. KESL database updates run hourly under the operator KESL runbook.
|
||||
|
||||
## External release gates
|
||||
|
||||
@@ -41,8 +49,10 @@ target environment verifies them:
|
||||
- Selectel S3 supports version-specific `GetObject`, signed conditional ETag behavior, bucket
|
||||
versioning, checksum metadata, virtual-host addressing and a read-only IAM policy without
|
||||
list/write/delete.
|
||||
- ClamAV engine/signature metadata is supplied to readiness and task cache keys; freshclam
|
||||
activate/reload, signature-age alarms and clean/EICAR/malformed corpora pass on VM2.
|
||||
- KESL version/database date is supplied through the broker to readiness and task cache keys;
|
||||
hourly update, stale-database alarms and clean/EICAR/malformed corpora pass on VM2.
|
||||
- The broker is a custom integration: exact `kesl-control` output/exit semantics, socket
|
||||
permissions, cleanup and throughput must pass gates on the target VM2 with KESL 12.4.
|
||||
- HEIF native decoding and PDF parser sandbox resource limits pass the approved corpus. The
|
||||
in-process detector is bounded by 5 MiB and validates active/encrypted PDF markers, but OS-level
|
||||
CPU/memory/wall-time isolation must be enforced by the worker container and target runtime.
|
||||
|
||||
@@ -96,6 +96,7 @@ def create_app(service: SafetyService, token: str) -> FastAPI:
|
||||
|
||||
@app.get("/health/ready")
|
||||
async def ready() -> JSONResponse:
|
||||
await service.refresh_antivirus()
|
||||
mode = "mock" if service.mode.mock else "standard"
|
||||
components = {
|
||||
"postgres": "ok",
|
||||
|
||||
@@ -2,9 +2,9 @@
|
||||
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["schema_version", "rules_bundle_ref", "detector_manifest_ref", "task", "rate", "retention", "cache", "link", "clamav", "file_policy"],
|
||||
"required": ["schema_version", "rules_bundle_ref", "detector_manifest_ref", "task", "rate", "retention", "cache", "link", "antivirus", "file_policy"],
|
||||
"properties": {
|
||||
"schema_version": {"const": 1},
|
||||
"schema_version": {"const": 2},
|
||||
"rules_bundle_ref": {"type": "string", "pattern": "^rules-[0-9]{4}-[0-9]{2}-[0-9]{2}$"},
|
||||
"detector_manifest_ref": {"const": "detector-2026-08-03"},
|
||||
"task": {
|
||||
@@ -48,9 +48,13 @@
|
||||
"pipeline_timeout_sec": {"type": "number", "exclusiveMinimum": 0, "maximum": 5}
|
||||
}
|
||||
},
|
||||
"clamav": {
|
||||
"type": "object", "additionalProperties": false, "required": ["scan_timeout_sec", "max_signature_age_hours"],
|
||||
"properties": {"scan_timeout_sec": {"type": "integer", "minimum": 1, "maximum": 120}, "max_signature_age_hours": {"type": "integer", "minimum": 1, "maximum": 720}}
|
||||
"antivirus": {
|
||||
"type": "object", "additionalProperties": false, "required": ["engine", "scan_timeout_sec", "max_signature_age_hours"],
|
||||
"properties": {
|
||||
"engine": {"const": "kesl"},
|
||||
"scan_timeout_sec": {"type": "integer", "minimum": 1, "maximum": 300},
|
||||
"max_signature_age_hours": {"type": "integer", "minimum": 1, "maximum": 720}
|
||||
}
|
||||
},
|
||||
"file_policy": {
|
||||
"type": "object", "additionalProperties": false, "required": ["enabled_mime_types", "max_size_bytes"],
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
schema_version: 1
|
||||
schema_version: 2
|
||||
rules_bundle_ref: rules-2026-01-01
|
||||
detector_manifest_ref: detector-2026-08-03
|
||||
task:
|
||||
@@ -21,8 +21,9 @@ link:
|
||||
url_max_length: 2048
|
||||
dns_lookup_timeout_sec: 1
|
||||
pipeline_timeout_sec: 2
|
||||
clamav:
|
||||
scan_timeout_sec: 45
|
||||
antivirus:
|
||||
engine: kesl
|
||||
scan_timeout_sec: 60
|
||||
max_signature_age_hours: 240
|
||||
file_policy:
|
||||
enabled_mime_types:
|
||||
|
||||
@@ -7,6 +7,8 @@ import json
|
||||
import struct
|
||||
from collections.abc import AsyncIterator
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC, datetime
|
||||
from email.utils import parsedate_to_datetime
|
||||
from pathlib import Path
|
||||
from typing import Protocol
|
||||
|
||||
@@ -33,7 +35,24 @@ class ObjectReader(Protocol):
|
||||
|
||||
|
||||
class Antivirus(Protocol):
|
||||
async def scan(self, chunks: AsyncIterator[bytes]) -> str | None: ...
|
||||
async def scan(
|
||||
self, chunks: AsyncIterator[bytes], *, scan_timeout: float
|
||||
) -> AntivirusScanResult: ...
|
||||
|
||||
async def status(self) -> AntivirusStatus: ...
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class AntivirusScanResult:
|
||||
threat: str | None
|
||||
signatures_version: str
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class AntivirusStatus:
|
||||
engine_version: str
|
||||
signatures_version: str
|
||||
databases_date: datetime
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
@@ -119,65 +138,101 @@ async def collect_and_hash(
|
||||
return bytes(body), digest.digest()
|
||||
|
||||
|
||||
class ClamAvInstream:
|
||||
def __init__(self, host: str, port: int, timeout: float = 45.0) -> None:
|
||||
self.host, self.port, self.timeout = host, port, timeout
|
||||
class KeslSocketScanner:
|
||||
MAX_HEADER = 4096
|
||||
MAX_RESPONSE = 16 * 1024
|
||||
|
||||
async def scan(self, chunks: AsyncIterator[bytes]) -> str | None:
|
||||
def __init__(self, socket_path: Path, timeout: float = 60.0) -> None:
|
||||
self.socket_path = socket_path
|
||||
self.timeout = timeout
|
||||
|
||||
async def _request(
|
||||
self,
|
||||
header: dict[str, object],
|
||||
body: bytes = b"",
|
||||
*,
|
||||
request_timeout: float | None = None,
|
||||
) -> dict[str, object]:
|
||||
encoded = json.dumps(header, separators=(",", ":")).encode()
|
||||
if len(encoded) > self.MAX_HEADER:
|
||||
raise DependencyFailure("KESL request header is too large")
|
||||
|
||||
async def operation() -> dict[str, object]:
|
||||
reader, writer = await asyncio.open_unix_connection(str(self.socket_path))
|
||||
try:
|
||||
writer.write(struct.pack(">I", len(encoded)) + encoded + body)
|
||||
await writer.drain()
|
||||
(size,) = struct.unpack(">I", await reader.readexactly(4))
|
||||
if size < 2 or size > self.MAX_RESPONSE:
|
||||
raise DependencyFailure("invalid KESL broker response size")
|
||||
value = json.loads((await reader.readexactly(size)).decode())
|
||||
if not isinstance(value, dict):
|
||||
raise DependencyFailure("invalid KESL broker response")
|
||||
return value
|
||||
finally:
|
||||
writer.close()
|
||||
await writer.wait_closed()
|
||||
|
||||
try:
|
||||
return await asyncio.wait_for(operation(), request_timeout or self.timeout)
|
||||
except (OSError, TimeoutError, asyncio.IncompleteReadError, json.JSONDecodeError) as exc:
|
||||
raise DependencyFailure("KESL unavailable") from exc
|
||||
|
||||
async def scan(
|
||||
self, chunks: AsyncIterator[bytes], *, scan_timeout: float
|
||||
) -> AntivirusScanResult:
|
||||
body = bytearray()
|
||||
async for chunk in chunks:
|
||||
body.extend(chunk)
|
||||
with tracer.start_as_current_span(
|
||||
"message_safety.clamav.scan",
|
||||
attributes={"server.address.type": "clamav"},
|
||||
"message_safety.antivirus.scan",
|
||||
attributes={"message_safety.antivirus.engine": "kesl"},
|
||||
):
|
||||
return await self._scan(chunks)
|
||||
|
||||
async def _scan(self, chunks: AsyncIterator[bytes]) -> str | None:
|
||||
async def operation() -> str | None:
|
||||
reader, writer = await asyncio.open_connection(self.host, self.port)
|
||||
try:
|
||||
writer.write(b"zINSTREAM\0")
|
||||
async for chunk in chunks:
|
||||
writer.write(struct.pack(">I", len(chunk)) + chunk)
|
||||
await writer.drain()
|
||||
writer.write(struct.pack(">I", 0))
|
||||
await writer.drain()
|
||||
result = await reader.readuntil(b"\0")
|
||||
text = result.rstrip(b"\0").decode("utf-8", "replace")
|
||||
if text.endswith(" OK"):
|
||||
return None
|
||||
if text.endswith(" FOUND"):
|
||||
return text.rsplit(": ", 1)[-1].removesuffix(" FOUND")
|
||||
raise DependencyFailure("invalid ClamAV response")
|
||||
finally:
|
||||
writer.close()
|
||||
await writer.wait_closed()
|
||||
|
||||
try:
|
||||
return await asyncio.wait_for(operation(), self.timeout)
|
||||
except (OSError, TimeoutError) as exc:
|
||||
raise DependencyFailure("ClamAV unavailable") from exc
|
||||
|
||||
async def signatures_version(self) -> str:
|
||||
with tracer.start_as_current_span("message_safety.clamav.version"):
|
||||
return await self._signatures_version()
|
||||
|
||||
async def _signatures_version(self) -> str:
|
||||
try:
|
||||
reader, writer = await asyncio.wait_for(
|
||||
asyncio.open_connection(self.host, self.port), 2.0
|
||||
value = await self._request(
|
||||
{"op": "scan", "size": len(body)},
|
||||
bytes(body),
|
||||
request_timeout=scan_timeout,
|
||||
)
|
||||
if value.get("status") != "completed" or value.get("verdict") not in {
|
||||
"clean",
|
||||
"infected",
|
||||
}:
|
||||
raise DependencyFailure("KESL scan failed")
|
||||
signatures = value.get("signatures_version")
|
||||
if not isinstance(signatures, str) or not signatures.startswith("sha256:"):
|
||||
raise DependencyFailure("KESL signatures version is missing")
|
||||
threat = value.get("threat")
|
||||
if value["verdict"] == "infected" and not isinstance(threat, str):
|
||||
raise DependencyFailure("KESL infected verdict has no threat")
|
||||
return AntivirusScanResult(
|
||||
threat=threat if isinstance(threat, str) else None,
|
||||
signatures_version=signatures,
|
||||
)
|
||||
|
||||
async def status(self) -> AntivirusStatus:
|
||||
with tracer.start_as_current_span(
|
||||
"message_safety.antivirus.status",
|
||||
attributes={"message_safety.antivirus.engine": "kesl"},
|
||||
):
|
||||
value = await self._request({"op": "status"})
|
||||
if value.get("status") != "ready":
|
||||
raise DependencyFailure("KESL is not ready")
|
||||
try:
|
||||
database_date = datetime.fromisoformat(
|
||||
str(value["databases_date"]).replace("Z", "+00:00")
|
||||
)
|
||||
except ValueError:
|
||||
try:
|
||||
writer.write(b"zVERSION\0")
|
||||
await writer.drain()
|
||||
raw = await asyncio.wait_for(reader.readuntil(b"\0"), 2.0)
|
||||
finally:
|
||||
writer.close()
|
||||
await writer.wait_closed()
|
||||
except (OSError, TimeoutError) as exc:
|
||||
raise DependencyFailure("ClamAV unavailable") from exc
|
||||
value = raw.rstrip(b"\0")
|
||||
if not value.startswith(b"ClamAV ") or len(value) > 512:
|
||||
raise DependencyFailure("invalid ClamAV version response")
|
||||
return "sha256:" + hashlib.sha256(value).hexdigest()
|
||||
database_date = parsedate_to_datetime(str(value["databases_date"]))
|
||||
except (TypeError, ValueError) as exc:
|
||||
raise DependencyFailure("invalid KESL database date") from exc
|
||||
if database_date.tzinfo is None:
|
||||
database_date = database_date.replace(tzinfo=UTC)
|
||||
engine = value.get("engine_version")
|
||||
signatures = value.get("signatures_version")
|
||||
if not isinstance(engine, str) or not isinstance(signatures, str):
|
||||
raise DependencyFailure("incomplete KESL status")
|
||||
return AntivirusStatus(engine, signatures, database_date.astimezone(UTC))
|
||||
|
||||
|
||||
async def one_chunk(data: bytes) -> AsyncIterator[bytes]:
|
||||
|
||||
@@ -8,7 +8,7 @@ from app.adapters import TrustedDnsResolver
|
||||
from app.api import create_app
|
||||
from app.config import ActiveConfig, validate_config
|
||||
from app.db import engine_and_sessions
|
||||
from app.file_pipeline import ClamAvInstream, DependencyFailure
|
||||
from app.file_pipeline import DependencyFailure, KeslSocketScanner
|
||||
from app.repository import Repository
|
||||
from app.service import SafetyService
|
||||
from app.settings import BootstrapSettings, EmergencyMode
|
||||
@@ -29,13 +29,14 @@ async def build_runtime() -> tuple[object, object]:
|
||||
resolver = TrustedDnsResolver(
|
||||
[item.strip() for item in settings.dns_resolvers.split(",") if item.strip()]
|
||||
)
|
||||
clamav = ClamAvInstream(settings.clamav_host, settings.clamav_port)
|
||||
antivirus = KeslSocketScanner(settings.antivirus_socket)
|
||||
if mode.mock:
|
||||
signatures_version = "unavailable"
|
||||
files_ready = False
|
||||
else:
|
||||
try:
|
||||
signatures_version = await clamav.signatures_version()
|
||||
status = await antivirus.status()
|
||||
signatures_version = status.signatures_version
|
||||
files_ready = True
|
||||
except DependencyFailure:
|
||||
signatures_version = "unavailable"
|
||||
@@ -47,6 +48,7 @@ async def build_runtime() -> tuple[object, object]:
|
||||
resolver,
|
||||
files_ready=files_ready,
|
||||
signatures_version=signatures_version,
|
||||
antivirus=antivirus,
|
||||
)
|
||||
app = create_app(service, settings.service_token.get_secret_value())
|
||||
instrument_fastapi(app)
|
||||
|
||||
@@ -90,7 +90,7 @@ class Repository:
|
||||
FileVerdictCache.config_version == config.version,
|
||||
FileVerdictCache.rules_version == config.rules_version,
|
||||
FileVerdictCache.detector_version == config.detector.version,
|
||||
FileVerdictCache.scanner_engine == "clamav",
|
||||
FileVerdictCache.scanner_engine == "kesl",
|
||||
FileVerdictCache.signatures_version == signatures_version,
|
||||
FileVerdictCache.expires_at > func.now(),
|
||||
)
|
||||
@@ -262,9 +262,29 @@ class Repository:
|
||||
return result.rowcount == 1
|
||||
|
||||
async def finish(
|
||||
self, task_id: uuid.UUID, owner: str, generation: int, *, allow: bool, rule_id: str
|
||||
self,
|
||||
task_id: uuid.UUID,
|
||||
owner: str,
|
||||
generation: int,
|
||||
*,
|
||||
allow: bool,
|
||||
rule_id: str,
|
||||
signatures_version: str | None = None,
|
||||
) -> bool:
|
||||
now = datetime.now(UTC)
|
||||
values = {
|
||||
"status": TaskStatus.allowed if allow else TaskStatus.denied,
|
||||
"verdict": "allow" if allow else "deny",
|
||||
"rule_id": rule_id,
|
||||
"reason_code": None if allow else "message_blocked",
|
||||
"finished_at": now,
|
||||
"purge_after": now + timedelta(days=30),
|
||||
"updated_at": now,
|
||||
"lease_owner": None,
|
||||
"lease_until": None,
|
||||
}
|
||||
if signatures_version is not None:
|
||||
values["signatures_version"] = signatures_version
|
||||
async with self.sessions.begin() as session:
|
||||
result = await session.execute(
|
||||
update(SafetyTask)
|
||||
@@ -275,17 +295,7 @@ class Repository:
|
||||
SafetyTask.lease_generation == generation,
|
||||
SafetyTask.lease_until > func.now(),
|
||||
)
|
||||
.values(
|
||||
status=TaskStatus.allowed if allow else TaskStatus.denied,
|
||||
verdict="allow" if allow else "deny",
|
||||
rule_id=rule_id,
|
||||
reason_code=None if allow else "message_blocked",
|
||||
finished_at=now,
|
||||
purge_after=now + timedelta(days=30),
|
||||
updated_at=now,
|
||||
lease_owner=None,
|
||||
lease_until=None,
|
||||
)
|
||||
.values(**values)
|
||||
)
|
||||
if result.rowcount == 1:
|
||||
await session.execute(
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import uuid
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from time import perf_counter
|
||||
@@ -14,7 +15,7 @@ from app.db import (
|
||||
TaskStatus,
|
||||
TextRulesCache,
|
||||
)
|
||||
from app.file_pipeline import validate_metadata
|
||||
from app.file_pipeline import Antivirus, DependencyFailure, validate_metadata
|
||||
from app.fingerprint import fingerprint
|
||||
from app.normalization import normalize_text
|
||||
from app.rate_limit import ConservativeRateLimiter
|
||||
@@ -52,6 +53,7 @@ class SafetyService:
|
||||
links_ready: bool = True,
|
||||
files_ready: bool = True,
|
||||
signatures_version: str = "unverified",
|
||||
antivirus: Antivirus | None = None,
|
||||
) -> None:
|
||||
self.repository = repository
|
||||
self.config = config
|
||||
@@ -60,11 +62,40 @@ class SafetyService:
|
||||
self.links_ready = links_ready
|
||||
self.files_ready = files_ready
|
||||
self.signatures_version = signatures_version
|
||||
self.antivirus = antivirus
|
||||
self._antivirus_checked_at = 0.0
|
||||
self._antivirus_check_lock = asyncio.Lock()
|
||||
self.rate_limiter = ConservativeRateLimiter(
|
||||
config.document["rate"]["text_rps"], config.document["rate"]["file_rps"]
|
||||
)
|
||||
record_runtime_state("mock" if mode.mock else "standard", config.version)
|
||||
|
||||
async def refresh_antivirus(self) -> bool:
|
||||
if self.mode.mock:
|
||||
self.files_ready = False
|
||||
return False
|
||||
if self.antivirus is None:
|
||||
return self.files_ready
|
||||
if perf_counter() - self._antivirus_checked_at < 5:
|
||||
return self.files_ready
|
||||
async with self._antivirus_check_lock:
|
||||
if perf_counter() - self._antivirus_checked_at < 5:
|
||||
return self.files_ready
|
||||
try:
|
||||
status = await self.antivirus.status()
|
||||
maximum_age = timedelta(
|
||||
hours=self.config.document["antivirus"]["max_signature_age_hours"]
|
||||
)
|
||||
if datetime.now(UTC) - status.databases_date > maximum_age:
|
||||
raise DependencyFailure("KESL databases are stale")
|
||||
self.signatures_version = status.signatures_version
|
||||
self.files_ready = True
|
||||
except DependencyFailure:
|
||||
self.signatures_version = "unavailable"
|
||||
self.files_ready = False
|
||||
self._antivirus_checked_at = perf_counter()
|
||||
return self.files_ready
|
||||
|
||||
def _verdict(
|
||||
self,
|
||||
allow: bool,
|
||||
@@ -222,7 +253,7 @@ class SafetyService:
|
||||
)
|
||||
|
||||
async def _check_file(self, request: FileCheck, digest: bytes) -> Verdict | Pending:
|
||||
if not self.files_ready:
|
||||
if not await self.refresh_antivirus():
|
||||
raise CapabilityUnavailable("files")
|
||||
rule = validate_metadata(
|
||||
request.attachment,
|
||||
@@ -274,7 +305,7 @@ class SafetyService:
|
||||
declared_checksum=request.attachment.checksum,
|
||||
rules_version=self.config.rules_version,
|
||||
detector_version=self.config.detector.version,
|
||||
scanner_engine="clamav",
|
||||
scanner_engine="kesl",
|
||||
signatures_version=self.signatures_version,
|
||||
origin_trace_id=trace_id,
|
||||
origin_span_id=span_id,
|
||||
|
||||
@@ -32,8 +32,10 @@ class BootstrapSettings(BaseSettings):
|
||||
default=5, ge=1, le=32, alias="MESSAGE_SAFETY_WORKER_CONCURRENCY"
|
||||
)
|
||||
dns_resolvers: str = Field(default="", alias="MESSAGE_SAFETY_DNS_RESOLVERS")
|
||||
clamav_host: str = Field(default="clamd", alias="MESSAGE_SAFETY_CLAMAV_HOST")
|
||||
clamav_port: int = Field(default=3310, ge=1, le=65535, alias="MESSAGE_SAFETY_CLAMAV_PORT")
|
||||
antivirus_socket: Path = Field(
|
||||
default=Path("/run/han-kesl/scan.sock"),
|
||||
alias="MESSAGE_SAFETY_ANTIVIRUS_SOCKET",
|
||||
)
|
||||
s3_endpoint_url: str = Field(alias="SELECTEL_S3_ENDPOINT_URL")
|
||||
s3_bucket: str = Field(alias="SELECTEL_S3_BUCKET_QUARANTINE")
|
||||
artifacts_dir: Path = Field(
|
||||
|
||||
@@ -12,8 +12,9 @@ from app.config import validate_config
|
||||
from app.contracts import Attachment
|
||||
from app.db import FileVerdictCache, SafetyAudit, engine_and_sessions
|
||||
from app.file_pipeline import (
|
||||
ClamAvInstream,
|
||||
Antivirus,
|
||||
DependencyFailure,
|
||||
KeslSocketScanner,
|
||||
ObjectChanged,
|
||||
collect_and_hash,
|
||||
detect_format,
|
||||
@@ -34,7 +35,7 @@ tracer = trace.get_tracer("message-safety.worker")
|
||||
|
||||
class Worker:
|
||||
def __init__(
|
||||
self, repository: Repository, reader: S3VersionReader, antivirus: ClamAvInstream, artifacts
|
||||
self, repository: Repository, reader: S3VersionReader, antivirus: Antivirus, artifacts
|
||||
) -> None:
|
||||
self.repository, self.reader, self.antivirus, self.artifacts = (
|
||||
repository,
|
||||
@@ -94,10 +95,15 @@ class Worker:
|
||||
)
|
||||
record_dependency("s3", "get_object", "success")
|
||||
rule = detect_format(body, attachment.mime_type)
|
||||
signatures_version = task.signatures_version
|
||||
if not rule:
|
||||
malware = await self.antivirus.scan(one_chunk(body))
|
||||
record_dependency("clamav", "scan", "success")
|
||||
rule = "file.malware_detected" if malware else None
|
||||
scan = await self.antivirus.scan(
|
||||
one_chunk(body),
|
||||
scan_timeout=row.config["antivirus"]["scan_timeout_sec"],
|
||||
)
|
||||
signatures_version = scan.signatures_version
|
||||
record_dependency("antivirus", "scan", "success")
|
||||
rule = "file.malware_detected" if scan.threat else None
|
||||
with tracer.start_as_current_span("message_safety.worker.finalize"):
|
||||
finished = await self.repository.finish(
|
||||
task.id,
|
||||
@@ -105,6 +111,7 @@ class Worker:
|
||||
task.lease_generation,
|
||||
allow=rule is None,
|
||||
rule_id=rule or "safety.all_checks_passed",
|
||||
signatures_version=signatures_version,
|
||||
)
|
||||
if finished:
|
||||
record_worker("allow" if rule is None else "deny", task_age)
|
||||
@@ -116,7 +123,7 @@ class Worker:
|
||||
rules_version=task.rules_version,
|
||||
detector_version=task.detector_version,
|
||||
scanner_engine=task.scanner_engine,
|
||||
signatures_version=task.signatures_version,
|
||||
signatures_version=signatures_version,
|
||||
verdict="allow" if rule is None else "deny",
|
||||
rule_id=rule or "safety.all_checks_passed",
|
||||
reason_code=None if rule is None else "message_blocked",
|
||||
@@ -192,7 +199,7 @@ async def serve() -> None:
|
||||
settings.s3_access_key.get_secret_value(),
|
||||
settings.s3_secret_key.get_secret_value(),
|
||||
),
|
||||
ClamAvInstream(settings.clamav_host, settings.clamav_port),
|
||||
KeslSocketScanner(settings.antivirus_socket),
|
||||
settings.artifacts_dir,
|
||||
)
|
||||
async with asyncio.TaskGroup() as group:
|
||||
|
||||
@@ -1,5 +1,6 @@
|
||||
from __future__ import annotations
|
||||
|
||||
from datetime import UTC, datetime, timedelta
|
||||
from uuid import uuid4
|
||||
|
||||
import httpx
|
||||
@@ -7,6 +8,7 @@ import pytest
|
||||
|
||||
from app.api import create_app
|
||||
from app.db import TaskStatus
|
||||
from app.file_pipeline import AntivirusStatus
|
||||
from app.repository import ConflictError
|
||||
from app.service import SafetyService
|
||||
from app.settings import EmergencyMode
|
||||
@@ -54,6 +56,15 @@ class ForbiddenResolver:
|
||||
raise AssertionError("MOCK must not call DNS")
|
||||
|
||||
|
||||
class StaleAntivirus:
|
||||
async def status(self):
|
||||
return AntivirusStatus(
|
||||
"12.4",
|
||||
"sha256:" + "a" * 64,
|
||||
datetime.now(UTC) - timedelta(hours=241),
|
||||
)
|
||||
|
||||
|
||||
def body(kind: str, message_id=None) -> dict:
|
||||
value = {
|
||||
"message_id": str(message_id or uuid4()),
|
||||
@@ -188,3 +199,27 @@ async def test_final_task_response_keeps_task_config_snapshot(active_config) ->
|
||||
|
||||
assert final.status_code == 200
|
||||
assert final.json()["config_version"] == 99
|
||||
|
||||
|
||||
async def test_stale_kesl_databases_disable_only_files(active_config) -> None:
|
||||
repo = FakeRepository()
|
||||
service = SafetyService(
|
||||
repo,
|
||||
active_config,
|
||||
EmergencyMode(),
|
||||
ForbiddenResolver(),
|
||||
antivirus=StaleAntivirus(),
|
||||
)
|
||||
async with httpx.AsyncClient(
|
||||
transport=httpx.ASGITransport(app=create_app(service, "secret")),
|
||||
base_url="http://test",
|
||||
headers={"X-Service-Token": "secret"},
|
||||
) as client:
|
||||
file_response = await client.post(
|
||||
"/internal/safety/v2/messages/check", json=body("file")
|
||||
)
|
||||
text_response = await client.post(
|
||||
"/internal/safety/v2/messages/check", json=body("text")
|
||||
)
|
||||
assert file_response.status_code == 503
|
||||
assert text_response.status_code == 200
|
||||
|
||||
@@ -35,12 +35,13 @@ def test_config_cross_field_and_manifest_subset(artifacts: Path) -> None:
|
||||
validate_config(bad, artifacts)
|
||||
|
||||
|
||||
def test_clamav_signature_age_policy_bounds(artifacts: Path) -> None:
|
||||
def test_kesl_signature_age_policy_bounds(artifacts: Path) -> None:
|
||||
document = seed(artifacts)
|
||||
assert document["clamav"]["max_signature_age_hours"] == 240
|
||||
document["clamav"]["max_signature_age_hours"] = 720
|
||||
assert document["antivirus"]["engine"] == "kesl"
|
||||
assert document["antivirus"]["max_signature_age_hours"] == 240
|
||||
document["antivirus"]["max_signature_age_hours"] = 720
|
||||
validate_config(document, artifacts)
|
||||
document["clamav"]["max_signature_age_hours"] = 721
|
||||
document["antivirus"]["max_signature_age_hours"] = 721
|
||||
with pytest.raises(ValidationError):
|
||||
validate_config(document, artifacts)
|
||||
|
||||
|
||||
@@ -1,14 +1,24 @@
|
||||
from __future__ import annotations
|
||||
|
||||
import asyncio
|
||||
import hashlib
|
||||
import io
|
||||
import json
|
||||
import struct
|
||||
from pathlib import Path
|
||||
from uuid import UUID
|
||||
|
||||
import pytest
|
||||
from PIL import Image
|
||||
|
||||
from app.contracts import Attachment
|
||||
from app.file_pipeline import ObjectChanged, collect_and_hash, detect_format
|
||||
from app.file_pipeline import (
|
||||
KeslSocketScanner,
|
||||
ObjectChanged,
|
||||
collect_and_hash,
|
||||
detect_format,
|
||||
one_chunk,
|
||||
)
|
||||
|
||||
|
||||
def image_bytes(format_name: str) -> bytes:
|
||||
@@ -71,3 +81,65 @@ async def test_authoritative_stream_hash_and_size() -> None:
|
||||
assert body == data and digest == hashlib.sha256(data).digest()
|
||||
with pytest.raises(ObjectChanged):
|
||||
await collect_and_hash(Reader(data), attachment(data, size=len(data) + 1), max_size=100)
|
||||
|
||||
|
||||
class FakeWriter:
|
||||
def __init__(self) -> None:
|
||||
self.request = bytearray()
|
||||
|
||||
def write(self, value: bytes) -> None:
|
||||
self.request.extend(value)
|
||||
|
||||
async def drain(self) -> None:
|
||||
return None
|
||||
|
||||
def close(self) -> None:
|
||||
return None
|
||||
|
||||
async def wait_closed(self) -> None:
|
||||
return None
|
||||
|
||||
|
||||
def framed(value: dict[str, object]):
|
||||
body = json.dumps(value).encode()
|
||||
reader = __import__("asyncio").StreamReader()
|
||||
reader.feed_data(struct.pack(">I", len(body)) + body)
|
||||
reader.feed_eof()
|
||||
return reader
|
||||
|
||||
|
||||
async def test_kesl_socket_clean_infected_and_status(monkeypatch, tmp_path: Path) -> None:
|
||||
responses = [
|
||||
{
|
||||
"status": "completed",
|
||||
"verdict": "clean",
|
||||
"threat": None,
|
||||
"engine_version": "12.4",
|
||||
"signatures_version": "sha256:" + "a" * 64,
|
||||
},
|
||||
{
|
||||
"status": "completed",
|
||||
"verdict": "infected",
|
||||
"threat": "EICAR-Test-File",
|
||||
"engine_version": "12.4",
|
||||
"signatures_version": "sha256:" + "b" * 64,
|
||||
},
|
||||
{
|
||||
"status": "ready",
|
||||
"engine_version": "12.4",
|
||||
"databases_date": "2026-09-07T11:25:00+00:00",
|
||||
"signatures_version": "sha256:" + "c" * 64,
|
||||
},
|
||||
]
|
||||
|
||||
async def connect(_):
|
||||
return framed(responses.pop(0)), FakeWriter()
|
||||
|
||||
monkeypatch.setattr(asyncio, "open_unix_connection", connect, raising=False)
|
||||
scanner = KeslSocketScanner(tmp_path / "scan.sock")
|
||||
clean = await scanner.scan(one_chunk(b"clean"), scan_timeout=1)
|
||||
infected = await scanner.scan(one_chunk(b"eicar"), scan_timeout=1)
|
||||
status = await scanner.status()
|
||||
assert clean.threat is None
|
||||
assert infected.threat == "EICAR-Test-File"
|
||||
assert status.engine_version == "12.4"
|
||||
|
||||
Reference in New Issue
Block a user