- Уменьшено число последовательных синхронизаций до доставки: ordinary
MAM append использует один sync вместо четырёх, SM checkpoint — один вместо двух. Checksummed redo journals сохраняют commit-before-ACK и восстанавливают native archive/SM JSON exports после аварии. - Ограничены рост журналов, ротация и tombstone GC. Старый `.rappend` и legacy JSON импортируются. Retain/save/purge не возвращают удалённые записи. - Recovery синхронизирует валидный journal перед публикацией состояния; проверка checksum не заменяет durability barrier. EIO при adoption блокирует startup или offline handoff. Новые fault fixtures проверяют сохранность журнала и retry. - Replay SM проверяет checksum/JSON всех frames, но создаёт и валидирует полный Record только для последнего snapshot, сокращая работу с superseded queues. - `message-processing-slow` содержит времена MAM отправителя/получателя, доставки, carbons, offline/push и handled, а также число ресурсов/копий. Payload и секреты не добавляются в логи. - Добавлена offline команда `storage-checkpoint CONFIG`: durable экспорт и передача archive writer/подготовка отката на legacy Rust. Не запускайте другого writer и не удаляйте journals до успешного завершения команды. - Новые crash, partial-write, corruption, EIO, recovery, handoff и latency регрессии включены в GitHub Actions через verify-local.sh; отчёты сохраняются. Измерения, ограничения и эксплуатация: [docs/DURABLE-MESSAGE-IO.md](docs/DURABLE-MESSAGE-IO.md), [docs/VALIDATION-0.8.3.md](docs/VALIDATION-0.8.3.md).
This commit is contained in:
1 parent
e9329ccbb2
commit
bff609097b
53 files changed
+8924
-1792
No files matched your search
@@ -85,6 +85,9 @@ jobs:
|
||||
/tmp/push-latency.json
|
||||
/tmp/push-offline-latency.json
|
||||
/tmp/bind2-archive-latency.json
|
||||
/tmp/slow-storage-latency.json
|
||||
/tmp/slow-storage-crash.json
|
||||
/tmp/journal-recovery-sync.json
|
||||
if-no-files-found: warn
|
||||
retention-days: 14
|
||||
|
||||
|
||||
@@ -28,7 +28,7 @@
|
||||
В нём сейчас нет `.git`. Если следующая среда содержит Git checkout, используйте
|
||||
его состояние и инструкции, не создавайте второй расходящийся источник истины.
|
||||
|
||||
Версия Rust source — **0.8.2**. В0.8.0 добавлены durable MUC/SM, затем CI
|
||||
Версия Rust source — **0.8.3**. В0.8.0 добавлены durable MUC/SM, затем CI
|
||||
и этот handoff;0.8.1 исправляет задержку сообщений. См. docs/MESSAGE-LATENCY.md.
|
||||
Перед передачей исходники упаковываются в ZIP; актуальный архив должен содержать
|
||||
этот файл и `.github/workflows/ci.yml`. В среде ChatGPT сохраняйте новый архив
|
||||
@@ -237,3 +237,64 @@ bind2_archive_latency_network.py, archive_edges.rs, session_routing_isolation.rs
|
||||
В передаваемый ZIP включайте AGENTS.md, workflow, vendor reference, исходники,
|
||||
документацию и scripts; исключайте target, demo data, bin, .git, __pycache__, pyc
|
||||
и секреты. Сохраняйте executable bits shell scripts и проверяйте содержимое ZIP.
|
||||
|
||||
## Durable message IO 0.8.3
|
||||
|
||||
Новый пользовательский log 0.8.2 содержит три local message-processing-slow
|
||||
4792–5549 мс. Таймер начинается внутри server после XML, но старые поля не
|
||||
доказывают причину IO. Контролируемый slow-sync fixture воспроизводит задержку
|
||||
1:1 в несколько секунд. MAM 4 sync/append и SM 2 sync/checkpoint заменены
|
||||
ограниченными checksummed redo journals. Обычная операция теперь требует 1 sync;
|
||||
создание, recovery и rotation дороже. Точные результаты — VALIDATION-0.8.3.md
|
||||
и DURABLE-MESSAGE-IO.md. Не приписывайте loopback причину production без phase logs.
|
||||
|
||||
SM `.wal` авторитетен, `.json` — readable export. Replay выполняется до JSON parse.
|
||||
Неполный trailing frame отбрасывается; неполный baseline или полная corruption
|
||||
блокируют startup. Durable tombstone предшествует delete. GC сначала делает
|
||||
sync каталога с JSON deletions, затем unlink WAL и sync каталога.
|
||||
Ordinary commit: append + sync_data, потом JSON export без sync. Ошибка export
|
||||
после commit не откатывает память; store блокирует дальнейшие записи до recovery.
|
||||
Precommit/sync error также блокирует store. Не возвращайте 2 sync на каждый
|
||||
checkpoint и не удаляйте redo без checkpoint. Общий SM mutex пока сохранён.
|
||||
Slot limit 8 MiB, total 256 MiB; WAL rotation 16 MiB плюс один slot, aggregate
|
||||
512 MiB. Не допускайте роста каждого из тысяч slots без aggregate cap.
|
||||
|
||||
Archive `.rredo`: native baseline и bounded anchor, checksummed record frames.
|
||||
Rotation после 128 frames/1 MiB плюс одна запись до 16 MiB. Cache ограничен 1024;
|
||||
изменение metadata вызывает перепроверку. Accepted records восстанавливаются
|
||||
даже при потере dirty native pages. Unjournaled native suffix не доказывает
|
||||
external append: truncate до committed end. Legacy `.rappend` ещё восстанавливается.
|
||||
Save/retain сначала durably фиксируют old native и удаление redo, затем
|
||||
делают atomic replacement. Смена inode при копировании/backup НЕ отменяет redo: journal
|
||||
остаётся авторитетным. Любой внешний writer, даже atomic replace, требует handoff. Purge включает rredo.
|
||||
Recovery повторно инвалидирует `.lidx`: unlink на hot path отдельно не dirsynced.
|
||||
Anchor проверяет только последние 4 KiB baseline, не всю старую историю.
|
||||
|
||||
Перед сменой writer/откатом остановите сервер и новым binary выполните
|
||||
`rust-xmpp storage-checkpoint CONFIG`. Внешний data-directory lock обязателен.
|
||||
`archive::checkpoint_all` и `session_store::checkpoint_exports` — только offline.
|
||||
Сначала sync native/current JSON и каталога, потом remove journal и sync каталога.
|
||||
Graceful stop сам по себе не означает handoff. Native Prosody не импортирует
|
||||
Rust SM sessions. Не редактируйте JSON export для expiry fixtures: используйте
|
||||
короткий настоящий timeout или проверяйте journal recovery. Не удаляйте WAL,
|
||||
чтобы получить зелёный тест. Native oracle перед Prosody modification явно
|
||||
вызывает checkpoint_all(root).
|
||||
|
||||
Новые tests: archive_redo.rs, sm_journal.rs, slow_storage_latency_network.py,
|
||||
slow_storage_crash_network.py, journal_recovery_sync_network.py, fixtures/slow_sync.c. Interposer действует только
|
||||
на изолированный fixture data_path. Обычный режим выполняет настоящий
|
||||
fsync/fdatasync; отдельный XMPP_SYNC_FAIL_SUFFIX вводит EIO только для fault test.
|
||||
Crash fixture проверяет journals и оба MAM архива до reauth после sender SM ACK
|
||||
и SIGKILL. Readable JSON без redo не подтверждает durability. Phase fields —
|
||||
server.rs и DURABLE-MESSAGE-IO.md.
|
||||
|
||||
В восстановленной среде native prefix теперь `/tmp/xmpp-native/usr/usr`, а не
|
||||
старый `/tmp/xmpp-native/usr`. `/tmp/xmpp083-env.sh` содержит env текущего прогона;
|
||||
это не portable config. Toolchain переустановлен; tests используют Rust 1.99.1.
|
||||
|
||||
Итог 0.8.3: 334 ordinary passed, 11 ignored; 9 native и 1 large-archive ignored
|
||||
выполнены отдельно. Все network checks и обе нагрузки 500 TLS прошли.
|
||||
Final binary SHA-256: 8ceb8de3a2d02bc4399b8ae817dff7e3c3fe9bff83dc0c6669b7f19210a3847a.
|
||||
Подробные samples/границы — docs/VALIDATION-0.8.3.md. Checksum и JSON syntax всех frames
|
||||
проверяются при SM replay, но Record создаётся только для последнего
|
||||
snapshot. Recovery WAL sync до публикации обязателен; EIO fixture защищает это.
|
||||
@@ -0,0 +1,25 @@
|
||||
# 0.8.3 — durable IO личных сообщений
|
||||
|
||||
- Уменьшено число последовательных синхронизаций до доставки: ordinary MAM
|
||||
append использует один sync вместо четырёх, SM checkpoint — один вместо двух.
|
||||
Checksummed redo journals сохраняют commit-before-ACK и восстанавливают
|
||||
native archive/SM JSON exports после аварии.
|
||||
- Ограничены рост журналов, ротация и tombstone GC. Старый `.rappend` и legacy
|
||||
JSON импортируются. Retain/save/purge не возвращают удалённые записи.
|
||||
- Recovery синхронизирует валидный journal перед публикацией состояния; проверка
|
||||
checksum не заменяет durability barrier. EIO при adoption блокирует startup
|
||||
или offline handoff. Новые fault fixtures проверяют сохранность журнала и retry.
|
||||
- Replay SM проверяет checksum/JSON всех frames, но создаёт и валидирует полный
|
||||
Record только для последнего snapshot, сокращая работу с superseded queues.
|
||||
- `message-processing-slow` содержит времена MAM отправителя/получателя,
|
||||
доставки, carbons, offline/push и handled, а также число ресурсов/копий.
|
||||
Payload и секреты не добавляются в логи.
|
||||
- Добавлена offline команда `storage-checkpoint CONFIG`: durable экспорт и
|
||||
передача archive writer/подготовка отката на legacy Rust. Не запускайте другого
|
||||
writer и не удаляйте journals до успешного завершения команды.
|
||||
- Новые crash, partial-write, corruption, EIO, recovery, handoff и latency
|
||||
регрессии включены в GitHub Actions через verify-local.sh; отчёты сохраняются.
|
||||
|
||||
Измерения, ограничения и эксплуатация:
|
||||
[docs/DURABLE-MESSAGE-IO.md](docs/DURABLE-MESSAGE-IO.md),
|
||||
[docs/VALIDATION-0.8.3.md](docs/VALIDATION-0.8.3.md).
|
||||
Generated
+1
-1
@@ -1111,7 +1111,7 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "rust-xmpp"
|
||||
version = "0.8.2"
|
||||
version = "0.8.3"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"axum",
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
[package]
|
||||
name = "rust-xmpp"
|
||||
version = "0.8.2"
|
||||
version = "0.8.3"
|
||||
edition = "2024"
|
||||
description = "Experimental Rust XMPP server with Prosody-compatible storage"
|
||||
license = "MIT"
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
# Rust XMPP 0.8.2
|
||||
# Rust XMPP 0.8.3
|
||||
|
||||
Сервер на Rust, разрабатываемый для замены **Prosody 13.0 nightly 100,
|
||||
cafd74b9c5ea** с предоставленными модулями. Читает существующий Lua-конфиг,
|
||||
@@ -251,3 +251,10 @@ recovery Rust служит собственный журнал. Перед от
|
||||
Push routing больше не блокирует отправителя/SM ACK; CSI defaults немедленно
|
||||
передают bodyless OMEMO и receipts. Bind2/MAM metadata не сканируют всю историю.
|
||||
[Причины, тесты и конфигурация](docs/PERSONAL-MESSAGE-LATENCY.md).
|
||||
|
||||
## Durable IO личных сообщений в0.8.3
|
||||
|
||||
Сокращены последовательные sync MAM/SM через checksummed redo journals,
|
||||
добавлены замеры этапов медленной обработки и crash/replay проверки.
|
||||
Перед передачей файлов Prosody/откатом выполните offline `storage-checkpoint`.
|
||||
[Durability, команда handoff и диагностика](docs/DURABLE-MESSAGE-IO.md).
|
||||
@@ -166,3 +166,9 @@ ad-hoc destroy/set-affiliation. Общие и host `modules_disabled` приме
|
||||
Ограничения журнала, native-экспорта и отката:
|
||||
[MUC-RECOVERY.md](MUC-RECOVERY.md). Итоги проверки:
|
||||
[VALIDATION-0.8.0.md](VALIDATION-0.8.0.md).
|
||||
|
||||
## Durable IO0.8.3
|
||||
|
||||
MAM/SM используют authoritative redo sidecars. Перед сменой writer или откатом
|
||||
на старую версию требуется offline `storage-checkpoint CONFIG`; после аварии
|
||||
нельзя просто удалить sidecars. Подробности — [DURABLE-MESSAGE-IO.md](DURABLE-MESSAGE-IO.md).
|
||||
@@ -0,0 +1,103 @@
|
||||
# Личные сообщения и durable IO в 0.8.3
|
||||
|
||||
В присланном журнале сервера 0.8.2 есть три `message-processing-slow` с 4792–5549 мс,
|
||||
route=local. Этот таймер начинается после приёма XML и заканчивается после
|
||||
`mark_handled`: ожидание происходит внутри обработчика. Старые события не
|
||||
содержат разбивки по этапам и сами по себе не доказывают причину медленного IO.
|
||||
Исходный пользовательский журнал не включён в архив проекта.
|
||||
|
||||
## Изменение записи
|
||||
|
||||
До 0.8.3 append в MAM требовал 4 последовательных sync: rollback marker file+dir,
|
||||
native list, удаление marker+dir. Обычное сохранение SM требовало 2: JSON file+dir.
|
||||
В TLS fixture1:1 с двумя аккаунтами, четырьмя SM-ресурсами и carbons это обычно
|
||||
16 sync на сообщение. Увеличение задержки каждого настоящего fsync/fdatasync
|
||||
внутри изолированного data_path воспроизводит задержку в несколько секунд.
|
||||
Interposer не пропускает sync и не меняет системные настройки/production.
|
||||
|
||||
В 0.8.3 принятые данные сначала фиксируются в журнале восстановления с checksum:
|
||||
|
||||
- MAM `.rredo`: native `.list` сохраняет формат Prosody; обычный append требует
|
||||
одного sync журнала. Replay восстанавливает недостающий native suffix и
|
||||
отбрасывает незавершённый хвост. Полностью записанная повреждённая запись
|
||||
вызывает ошибку. Ротация —128 frames/1 MiB плюс максимум одна 16 MiB запись.
|
||||
- SM `.wal`: authoritative полные snapshots и tombstones; readable JSON —
|
||||
восстанавливаемый экспорт. Обычное сохранение требует одного sync_data.
|
||||
Ротация —16 MiB плюс максимум один snapshot; aggregate WAL cap 512 MiB,
|
||||
logical slots по-прежнему 256 MiB/8 MiB на slot/4096 stanzas и 4096 sessions.
|
||||
Tombstone GC сначала синхронизирует удаление JSON, затем удаляет WAL.
|
||||
|
||||
Recovery синхронизирует полностью проверенный журнал перед публикацией
|
||||
восстановленного состояния: полный frame после SIGKILL может остаться
|
||||
только в page cache. Checksum подтверждает целостность, но не запись на диск. Ошибка этого sync запрещает startup/handoff.
|
||||
|
||||
Создание журналов, восстановление и ротация дополнительно синхронизируют файлы
|
||||
и каталог. Экономия относится к обычной записи, не каждому возможному запросу.
|
||||
Отключения durability или периодической записи вместо commit-before-ACK нет.
|
||||
При неопределённой ошибке SM IO store блокирует дальнейшие изменения до
|
||||
восстановления. После успешного WAL commit ошибка экспорта не откатывает память.
|
||||
Queues/archives не образуют общей транзакции с сетью: гарантия replay остаётся
|
||||
at least once, а не exactly once между всеми хранилищами.
|
||||
|
||||
## Передача данных Prosody и откат версии
|
||||
|
||||
После SIGKILL/OS crash сначала используйте 0.8.3 для восстановления redo. Не
|
||||
удаляйте `.wal`/`.rredo` вручную и не редактируйте JSON export как источник истины.
|
||||
Для смены writer или отката на старую версию остановите сервер и выполните:
|
||||
|
||||
```bash
|
||||
rust-xmpp storage-checkpoint /etc/prosody/prosody.cfg.lua
|
||||
```
|
||||
|
||||
Команда берёт data-directory lock и SM store lock, восстанавливает журналы,
|
||||
синхронизирует native archives и актуальные JSON, затем durably удаляет redo.
|
||||
При ошибке передача владения не завершена; разберите ошибку, не запускайте
|
||||
другой writer. Перед работой с production данными сохраните обычную резервную
|
||||
копию всего data_path. Native Prosody не понимает resumable SM snapshots Rust;
|
||||
checkpoint SM обеспечивает совместимый JSON для старого Rust, не переносит
|
||||
TLS/TCP/SM-сессию в Prosody. Простой graceful stop не заменяет эту явную команду.
|
||||
Нельзя одновременно изменять `.list` средствами Prosody и Rust. Неjournaled
|
||||
in-place append при активном redo считается незавершённым и может быть отброшен.
|
||||
При восстановлении согласованной копии всего data_path журнал остаётся
|
||||
авторитетным независимо от смены inode файла. Любая внешняя модификация,
|
||||
включая atomic replacement, требует handoff. Внутренний rewrite сначала
|
||||
фиксирует старый native snapshot и удаление redo, затем атомарно заменяет native
|
||||
файл: авария в промежутке сохраняет старый snapshot, после замены — старый/новый.
|
||||
Редактирование baseline in place без handoff не поддержано. Anchor проверяет
|
||||
последние 4 KiB baseline, не является checksum всего старого архива.
|
||||
|
||||
## Диагностика
|
||||
|
||||
`message-processing-slow` по-прежнему имеет threshold 250 мс, duration_ms, route
|
||||
и connection. Новые поля в integer milliseconds:
|
||||
|
||||
|Поле|Измеряемый этап|
|
||||
|---|---|
|
||||
|pre_routing_ms|Работа до личного MAM; для MUC/ранних выходов — обработка до handled|
|
||||
|sender_mam_ms|Политика и архивирование отправителя|
|
||||
|recipient_mam_ms|Проверка аккаунта, политика и архивирование получателя|
|
||||
|delivery_ms|Маршрутизация, targets, mutex и transmit|
|
||||
|carbons_ms|Выбор ресурсов и отправка carbon|
|
||||
|offline_push_ms|Offline commit и подготовка/enqueue push|
|
||||
|handled_ms|Session mutex и durable mark_handled|
|
||||
|
||||
Неисполненные этапы равны 0. target_count — выбранные локальные ресурсы,
|
||||
carbon_count — успешные carbon transmit. Это серверные этапы, не время
|
||||
отображения клиентом. Тексты сообщений, SASL и токены не логируются.
|
||||
|
||||
## Проверки и границы
|
||||
|
||||
`tests/slow_storage_latency_network.py` — реальный STARTTLS, MAM/carbons/PEP,
|
||||
контролируемый sync delay и счётчик операций. `tests/slow_storage_crash_network.py`
|
||||
— sender SM ACK, немедленный SIGKILL, проверка committed journals/обоих архивов
|
||||
до повторной аутентификации и replay всех ресурсов. Rust regressions проверяют
|
||||
partial tails, corruption, ротацию, tombstones, migration и offline handoff.
|
||||
Отдельный `journal_recovery_sync_network.py` вводит scoped EIO для WAL/redo
|
||||
и проверяет отказ startup/handoff без потери journal и успешный повтор.
|
||||
Точные измерения и результаты этой сборки — [VALIDATION.md](VALIDATION.md).
|
||||
|
||||
Общий mutex SM store, обязательное durable IO, большие pending snapshots,
|
||||
полные MAM queries/maintenance, PEP storage и federation всё ещё имеют стоимость.
|
||||
Медленный диск не становится быстрым от сокращения числа sync. Проверка на
|
||||
loopback не устанавливает причину каждого production warning; новые phase
|
||||
fields позволяют проверить это на целевой системе.
|
||||
@@ -80,7 +80,10 @@ Native state без подтверждающих Rust-метаданных са
|
||||
восстановление. Это не универсальное слияние параллельных записей двух серверов.
|
||||
|
||||
Для отката на Prosody остановите Rust и сохраните резервную копию всего
|
||||
`data_path`. Prosody использует native-экспорт и не читает Rust-журнал. Перед
|
||||
`data_path`. Начиная с0.8.3 перед сменой writer выполните новым binary
|
||||
`rust-xmpp storage-checkpoint CONFIG`: он фиксирует MAM redo и SM JSON exports.
|
||||
Это не перенос Rust SM-сессий в Prosody и не универсальная конверсия MUC journal.
|
||||
Подробности — [DURABLE-MESSAGE-IO.md](DURABLE-MESSAGE-IO.md). Prosody использует native-экспорт и не читает Rust-журнал. Перед
|
||||
возвратом с Prosody на Rust согласуйте данные и устраните устаревший
|
||||
`rust_muc_checkpoint` в остановленном экземпляре: иначе неизменившиеся native
|
||||
файлы могут быть приняты за экспорт предыдущего поколения Rust. Удаление
|
||||
|
||||
@@ -100,3 +100,8 @@ SM Store пока использует общий mutex для durable slot IO;
|
||||
и внешняя федерация имеют собственную стоимость. Производственная квалификация
|
||||
клиентов и полная замена Prosody остаются за рамками этих fixtures.
|
||||
Итоговые тесты/сырые отчёты: VALIDATION-0.8.2.md и validation-logs-0.8.2/.
|
||||
|
||||
## Продолжение0.8.3
|
||||
|
||||
Новые логи показали seconds-level серверную обработку. Добавлены redo MAM/SM
|
||||
и разбивка времени на этапы: [DURABLE-MESSAGE-IO.md](DURABLE-MESSAGE-IO.md).
|
||||
+21
-14
@@ -1,13 +1,14 @@
|
||||
79f3f5cd46255826869c9151151d3471dd7139c2ea8fb2da4b2b6fbea113f0cf .github/workflows/ci.yml
|
||||
d9732ba55b93d2907cd1a2a0fa20bc1f46e0f2970de0e9f2bf730048212cfc31 AGENTS.md
|
||||
f8e6a12039b067ed5af42320bca42fb0555d9a5f438d1652860a5be4534469d1 Cargo.lock
|
||||
4e8d7ee328f044be284e0cb436f13a468adcac3058bbb1be381f8bff96603e5a Cargo.toml
|
||||
9984392a77fcbb6d0ee70b536ecdcfc4349101bb300a509e32353aaa1134979e .github/workflows/ci.yml
|
||||
6593266e57a718762fbca26fd893fcabf1ea842dcc53dd0fd11844b2def2a543 AGENTS.md
|
||||
b83b20e991af384b349825ca39aad157d8b412d0143cc34c68567ca011bccf58 Cargo.lock
|
||||
3ecc46532a7fb1a231dbbd5337221d0c0c32ba36574e74655854f1a1d54d4581 Cargo.toml
|
||||
6be2fbcd1da8a07dda7daf3ddd727b38f363751ebace9c0b87c282f8961cffed Dockerfile
|
||||
7136ea911bb6b6d352f703f80efeebde921028ea9dc39cd0d2cd55ea465f8c37 examples/config.json
|
||||
8a65739eb5c24f10b9f2c509f0bd050fb1115706e6a41fdf804b9e0e44f583fd examples/config.toml
|
||||
ca86274bcf4dc5daebbf69c9f003854afca92954719aff068f810e0a3d615dea examples/config.yaml
|
||||
bbc2591581d382fef5fb918a480e6c338c7e3e3f2025df5509f51708534631e7 examples/converse-origin.fragment.yaml
|
||||
66ea9e2b58f6a39b70911b9981cae5e99625483af7a41168e24923464de8ae56 examples/reverse-proxy.fragment.yaml
|
||||
d13306be0ecda81ce779baa290ffb417380d525928477f16200884b5169cca33 rust-toolchain.toml
|
||||
c08bd5923906f4d2163a258c8b68dbef39ea6a3e07032f67e5c12d3d9ca81c5e scripts/ci/image-tags.py
|
||||
8820d262f8018a822b27eb9e5f0d3b9ca0c1272001164ca8a59fcd13911a5426 scripts/ci/prepare-minio.sh
|
||||
5bcd646c6c9159bbdce3a4eadce4d80d6c7422748e47c6ad04e5769efb4f2920 scripts/ci/prepare-reference.sh
|
||||
@@ -25,10 +26,10 @@ b1018e1b279c89f6cf14b4017e4f1b14e9165afef2f668cec6ac8bbe0c710d85 scripts/integr
|
||||
e08ac6136cce27cda2ddbd510da0f25750766ab0150a9f4b8f87c953276d16e9 scripts/prosody_compat/oracle.lua
|
||||
2c7a2c946885205e9df38b97305073a8ac72f58ac8ff84fc6f9af320c35bbc96 scripts/prosody_compat/verify.sh
|
||||
1b9ed063daf3e0320819b4fe0135c96dd27e49877044aa2f5093b8ab4fbda41d scripts/smoke.py
|
||||
c32bc7ed2d5f954a2b3de06db31fb1580378259edfc8ce574aa5cd5719cc2a59 scripts/verify-local.sh
|
||||
2cd5e17cdf0902ceaaca9b6e8d36ca766e33e66a06784ec97ad99165571eaf91 scripts/verify-local.sh
|
||||
a2e47f547ab9349f84ee6c4f8e51c147b7110b41a82c17cdb938027a84d82ef3 src/account_modules.rs
|
||||
5637f816586f0613b83e14d0ed478bd539f325b70ccbfc9fdf893adbe494453d src/admin.rs
|
||||
7b844cbe377b057555436378ab8bd784f99b85e7c9992cb3c5a044f40326951f src/archive.rs
|
||||
1b2011fb38482eba69dc245c367ddf496089d0cbde8a000b1ff5e9bbf8c3dea8 src/archive.rs
|
||||
e15b079a420227ee59e3b7ac305783c51eb59d76a8fb5dd12760547b0f812107 src/auth.rs
|
||||
83ac35a4c9d20debf006fe80fba8e423c09806fb2526ff384b188ca28d612a5e src/config.rs
|
||||
175ab07f17ba14c51c76857c4905b0a587d8be68e2be5179456e9c71a523c487 src/csi.rs
|
||||
@@ -43,7 +44,7 @@ c604a0a764ffac6bae0c9aa15c9ef7022d35cb1f8c9159eb66257114445621c5 src/http_trans
|
||||
5b27b6c719537af919bd6fe089574eba5e2c7bbe0d32249164914a4bb57a1427 src/lib.rs
|
||||
a21e1119ea0271b63935e92eb106ab6076f0035e6d3a06c21a352f152b9b37db src/logging.rs
|
||||
58ce2a9daa36ac26b39c8f263a16f4fa8cc4e57c91bbbf5b24a6aad3f1a326e0 src/lua.rs
|
||||
a264e2765a31c8c096462551aca229530cad356e39e338370d1a0bdd97860680 src/main.rs
|
||||
f103c15e11d61b5818e9a9491b8533b904cade7b3654c481cc61d3a7d0bb8173 src/main.rs
|
||||
68d97d96aa3a370474c144ccfdbf16109c46a6b14123d55d30464237ae29a958 src/mam.rs
|
||||
f17b06fe5b76381b5cdbfc2b668107f6c2b0b456c05b7a7709bea6a284531ff5 src/muc.rs
|
||||
c7ba46a140b9dce65cc573176265d6f77049ae657a70ffe59611a1c0f3ef89af src/muc_admin.rs
|
||||
@@ -56,10 +57,10 @@ f09a6479b76c9c601a169b5f1f9cf1364e82df5ae27e4bab8d1b1428b5cad457 src/preauth.rs
|
||||
8dbb7a6b43a5f0562a775df42a959ab8badc466135a8b32da86cd79615b5acbf src/pubsub.rs
|
||||
913cd033f60a68f6c7cf9b3adbaf76db1c5c9bc88a0599196d373b16b64c0a1e src/push.rs
|
||||
bec477f2520af6a9d7ca54e43eb12158b53e3cf09057653e504dfdeab956885e src/sasl.rs
|
||||
a445dcab6d0275861b08412a4bd622cb72c3f462ebba5eaedad360e740a6513c src/server.rs
|
||||
c7450f9058528e3030df559840f87dae591a2b88be8503872956e9b78435d916 src/session_store.rs
|
||||
1bcc6995e221f316bd4765eeb79c2ad67c9fe9e93d4254e6f6679244fc733e6f src/server.rs
|
||||
c37f6c478d45b3c0741f0b1bed95f499e67d4d3eb50b78fe606e2b518170e45f src/session_store.rs
|
||||
1d4e890315886bc4fd0b824e034a5c1833a1f2c1347e8c3ca77254e13a2acab9 src/sessions.rs
|
||||
71d7ce1036a091d8c27016ea7ce4cffedf904dfca9d1cc4891b7dc66a1e06a4e src/storage.rs
|
||||
c9fb5d18da148a6366a18afb86eaee26a67d20c7b6664e425e612d44e79a2988 src/storage.rs
|
||||
462b00edb2c1e02dd34524bafe4f82b5ef902028b44310ba31b085d31f4dc2aa src/tls_binding.rs
|
||||
a78592aa79d8f01c22af1457d0c3f5922863ccd3e8ebb1aa05e043cc44579b8a src/upload_migration.rs
|
||||
f9b023565711e0217238b53fb678c5fb14849cdcd2efcd0fce249018274fad69 src/wire.rs
|
||||
@@ -68,6 +69,7 @@ b4eb7a230570dfde0fd0d301a6ace36559a5eae3e1402e42c76f014f61739c24 tests/_wire_cl
|
||||
0645fda01bede4362d998ef22aefdd2c02578dd2c2aae3878d0cff87b3679835 tests/android_sasl_network.py
|
||||
ad225ade463f55bfd0734e67f5306e274cd9aab59e10e91469bf7263547cdf79 tests/archive_edges.rs
|
||||
366b315521bd4f4448c0bac17bf74ddb77a728ff41fc48e29d46377625e0330f tests/archive_mam.rs
|
||||
df8530d69beaf36c34e78f0f050930b7e0d6a296c22ca543dfc7533180ba0543 tests/archive_redo.rs
|
||||
cb6b7bbcd1bc7ee01a4ccf892d4b6074765685c56c4b8bb46e0381b834577b6a tests/auth.rs
|
||||
011bbdd00c8ff8b648076d133086c395f6784cdcbb0c9f4e1efcbc15c1e79d9c tests/bind2_archive_latency_network.py
|
||||
b3c3470f0b9b64584abf98c26a4578b0b90089086d00f7b07187aa5f6db6a417 tests/cli.rs
|
||||
@@ -78,6 +80,7 @@ d4d8d982c6f4ab96a809b598ea9ead7f64fa9d916547f1eb9d797aa71db64fdb tests/data_loc
|
||||
be18a83fb5884cfdbd2ddc542cb25fd07b70f139cbdce1ae67562568df42a2e2 tests/debug_legacy.rs
|
||||
63753c2e3524e8e7826bc2d48e2bd78e36e1cc56fde9da19950d5c35e1c38f77 tests/federation_tls.rs
|
||||
1aaf06ba872cb5dbe0545afffdd8efddd0eae7305de06f3b3b75c19289581865 tests/fixtures/prosody.cfg.lua
|
||||
33a8a5d9980bb9aa1162a1e3e562a9d889edd3d2f96a7073e14acc8e504bf9c7 tests/fixtures/slow_sync.c
|
||||
19d570b92075d1355257c3798e4c52e404cc356dfe5dfd0c8fb384e4e7150682 tests/gajim_discovery_network.py
|
||||
4398bf0639c31f0df328f0b0f1a5df17f2e69d2df84514845bba882146de06bc tests/gajim_features.rs
|
||||
a346e70af3657fbf56e0bb1c39c427f630de8c349e6858b8933c52f42a21ab46 tests/http_altconnect.rs
|
||||
@@ -86,6 +89,7 @@ f1ea3eb78d5c716f4281cf17cb41b9fb22e08dbb4cf08ed9c8477c79b7d2382b tests/http_pro
|
||||
7c599c83eb999cd8776f0ce6811db8f478366b840311f2e513823f70b5d7cd30 tests/http_transport.rs
|
||||
278db32ba0dec882b0af8f1969da87670ff3d2d749d32320154dabef51cff191 tests/http_upload.rs
|
||||
da2fc65e931ec74fbcdcd26d04f2cb073e2dc1e354139135166c79580973a477 tests/iq_errors.rs
|
||||
ec993a6e31d38f82abd13bd46ad99286cf13e8abea63c0385c848037a12b0cd4 tests/journal_recovery_sync_network.py
|
||||
05160f17f8098466a67c9b069c165dc9d94a84ab03a49896169f110baf8e869a tests/logging.rs
|
||||
dfc3429e3e755d4c99ec1bd8ee6fb0f1f6aaf86414d5adaca7e9e3b18c7fe3f7 tests/logging_network.py
|
||||
0141961eadf9246ce287d7ee0dbbbf5243fd6e1332e09ccd2f6a591be419c2c3 tests/message_latency_network.py
|
||||
@@ -98,7 +102,7 @@ dfc3429e3e755d4c99ec1bd8ee6fb0f1f6aaf86414d5adaca7e9e3b18c7fe3f7 tests/logging_
|
||||
a8a01faf51bf82bce6cbb2616cdaabb6337047d1c5f17d5f39975c0528a1d5ae tests/muc_flow.rs
|
||||
b24bd920f096576c2009ff1ccc0366eb8734c5f0b2feb39f04ad7087268a94b9 tests/muc_history.rs
|
||||
1b34ab721eec69295e1274e44bfd108c0e117dd383624cb6c98d851eb20e7a9a tests/muc_latency_storage.rs
|
||||
1f3df5f3f39143adc2b2a419fac0cd74b2229337ffa86570261d1f6ac61137a1 tests/muc_native.rs
|
||||
18f18362f26710043b51168675daee91ae88c071fb44e48210b49686011f8722 tests/muc_native.rs
|
||||
3df2736ccfd25a6cab290df80ceb7474ea44bb02af6cf768a9216d6b8f7f8630 tests/muc_optional_admin.rs
|
||||
5dc93407a658b45e504b89e26cfc8f495cd28ad90d0a308b7d228a472435ff9f tests/muc_protocol.rs
|
||||
ac1caa62a3cd91e74bd7100a5e542c404e97aec61a34e9627900b458c2ee3806 tests/muc_recovery.rs
|
||||
@@ -124,7 +128,7 @@ b8dd71798f96931f7ab4489cdb1e258556f41411815ef38d3a03e80e4e8fbdfc tests/push_dis
|
||||
44c8e642f725be2292dfc5282feac8772ac34824927b63058f03f7c98285e5b6 tests/push_latency_network.py
|
||||
abf5e6a32ab542ccba5861eaaf7aed9a86b8306abfe2c329646d54e43176dc0c tests/reference/COPYING
|
||||
1ecd0a5c122910cbd066922c95ff0c32e945a07e2c9b0a37a30e5f035b5ecea6 tests/reference/configmanager.lua
|
||||
f407c6565fa73b6653e7e73204daa65ae1d87111e82c6b96154aea76497ad65d tests/reference_prosody.rs
|
||||
78fb639a334846c884f2d438407d7fa8491778b695f9e6218a161ebf8acaeced tests/reference_prosody.rs
|
||||
5ab78e9319d6004d65d35cd4d04dbb99d4ea5ffd8da00473d874bb26000fce35 tests/resource_rebind.py
|
||||
2e79de5ef8d3be81ac8f715bb96797b6e8e39cdcd61c13d8d873a96b53cb52e4 tests/resource_sessions.rs
|
||||
cefd1001c9619456402223fa89e449f65599c4397f523860e965ca192a4fd915 tests/sasl2_bind.py
|
||||
@@ -135,9 +139,12 @@ a7d1519f4f44cc7bfad3d20bcfdf455e92abdd084feed493ffc9e0267a387740 tests/sasl_net
|
||||
4748fa227743686f78a0fa0364735d2dda5995f8d70a79ad03c1aa1decb42978 tests/sasl_retry_network.py
|
||||
d8882b8aabd6ff603c81fbbc259eefa68772afb2b3eec2e003ac29efd65d146b tests/sasl_web_network.py
|
||||
7d69c14ced7884450b9a1ff63cd984cdb9dfe948d6c56631c7593b6fabdabf7f tests/session_routing_isolation.rs
|
||||
7636b0767fdf24e3d4bde03779ec40309ed2b3301d4e35c6ae5f59aa065e4b2e tests/slow_storage_crash_network.py
|
||||
7492523bb226b68f72b97b028a7fafeb4fb5375b1efc6fb516400279dd0743d4 tests/slow_storage_latency_network.py
|
||||
d9eef5c37fcd99404619d61dbf7a2bf7673087924d93e69b08769e716afe297b tests/sm_durable_slots.rs
|
||||
0b74b2ff47b8ae681b5613bc87e914fdf8e5033156180df06ff2174df2781a5a tests/sm_restart_network.py
|
||||
72f2d60e47e5295b82d497289b78925a64a1d986c70c277015ae228d211d32bb tests/storage_accounts.rs
|
||||
b4c9422cdba8fd133572c0fa3a0ae6cda14edbd14db6d984d76f4409eefd1262 tests/sm_journal.rs
|
||||
8fa5de6a2291a7c35e7bbbc6334d42b4e0836a36bec252d0ef4703e8ee17aecb tests/sm_restart_network.py
|
||||
f76ad9de4c7e0140b1a4c3da06e849e28bb0562cf9b576ed85a5257f7394506b tests/storage_accounts.rs
|
||||
5c38afd5f77b5774552d12c461902ffaae21de11735560c6f0088e3dda8afa58 tests/storage_compat.rs
|
||||
8724991fdf22471952cf1a0fac863abe837069c93d3c8d2fa751af4f6bbc3e38 tests/tls_binding.rs
|
||||
5bea9b5546d3a6ae4fe78d17acbc3f6e6a1d5e1d13b32f2aa88d0e320892b7c5 tests/tls_plus_network.rs
|
||||
|
||||
@@ -0,0 +1,79 @@
|
||||
# Проверка 0.8.3
|
||||
|
||||
2026-10-21, Linux x86_64, Rust 1.99.1. Итоговый единый прогон после заморозки
|
||||
исходников: [verify.txt](validation-logs-0.8.3/verify.txt).
|
||||
[Machine-readable results](validation-results-0.8.3.json).
|
||||
|
||||
|Проверка|Результат|
|
||||
|---|---|
|
||||
|Rust ordinary tests|334 passed, 0 failed, 11 ignored|
|
||||
|fmt / Clippy all-targets -D warnings / release|Успешно|
|
||||
|Новые Rust regressions|13 archive redo + 2 rewrite crash boundaries + 14 SM journal, успешно|
|
||||
|Native reference|8 module/storage + 1 TLS/SASL EXTERNAL federation, успешно|
|
||||
|Archive >128 MiB|Отдельно выполненный ignored streaming test, успешно|
|
||||
|Copy/restore и rewrite|Новый inode, потеря suffix, baseline0/nonzero, crash до/после rename — успешно|
|
||||
|Все network regressions verify-local.sh|Успешно, включая Android, WSS/BOSH, PEP, push, MUC crash/SM resume, upload/migration|
|
||||
|Slow durable IO|0/100 мс: 3 samples/scenario; 200 мс: 1 sample/scenario; exact MAM/carbons/queues и phase logs проверены|
|
||||
|SIGKILL сразу после sender SM ACK|100 мс/sync, 3 сообщения, оба MAM архива, 4 resume, 9 replay, до reauth; успешно|
|
||||
|Recovery sync EIO|WAL/redo: отказ handoff без удаления journal; WAL startup до listeners; retry и данные проверены|
|
||||
|Offline storage-checkpoint|4 SM exports, 2 archives, durable journal removal, повтор идемпотентен|
|
||||
|Общая нагрузка|500 TLS, 100 accounts, 5000 ping, 5000 messages, 10000 MAM, failures=[]|
|
||||
|MUC+SM нагрузка|500 resources, 10 rooms, 25000 presence, 1500 deliveries, SIGKILL и 10 resume; успешно|
|
||||
|
||||
## Личная доставка
|
||||
|
||||
Медианы для 3 samples, при 200 мс — одна проба; числа в миллисекундах.
|
||||
|
||||
|Sync delay / сценарий|Body 0.8.2 → 0.8.3|Sender SM ACK 0.8.2 → 0.8.3|Проб|
|
||||
|---|---:|---:|---:|
|
||||
|0 мс / 1:1|25.11 → 7.87|33.41 → 15.08|3|
|
||||
|0 мс / PEP → 1:1|29.71 → 20.94|32.25 → 21.86|3|
|
||||
|100 мс / 1:1|1232.47 → 419.68|1641.60 → 644.08|3|
|
||||
|100 мс / PEP → 1:1|2683.29 → 1378.23|3195.21 → 1581.98|3|
|
||||
|200 мс / 1:1|2836.64 → 2042.56|3643.30 → 2441.99|1|
|
||||
|200 мс / PEP → 1:1|5782.02 → 2676.28|6595.92 → 3094.95|1|
|
||||
|
||||
0 мс — отдельная последовательная quiet pair, без full verify. Финальные
|
||||
100/200 мс выполнялись при фоновой работе verify; это не изолированный CPU benchmark.
|
||||
Первый append включает создание/migration MAM journal. В обычном warm сообщении
|
||||
6 sync вместо 16; первый — 12. Окна счётчиков могут включать heartbeat, а PEP
|
||||
сценарий включает предшествующую publication. Реальные fsync/fdatasync сохранены;
|
||||
interposer замедляет только isolated data_path, не меняет host mount/DNS.
|
||||
Raw samples, hashes, sync counts и phase fields — validation-logs-0.8.3/*.json.
|
||||
|
||||
В production log 0.8.2 серверный handler занимал 4792–5549 мс; старых полей
|
||||
недостаточно, чтобы доказать конкретный IO stage. Новый таймер разделяет MAM,
|
||||
доставку, carbons и handled, без payload. Эти loopback результаты не доказывают
|
||||
время отображения реальным Conversations/Monocles на wthome.ru.
|
||||
|
||||
## Нагрузка и сопоставимость
|
||||
|
||||
Сопоставимый повтор 0.8.2 в текущей среде: total 151.334 с,
|
||||
join 104.226 с. Финальный 0.8.3: total 128.380 с,
|
||||
join 94.493 с, restart 7.557 с,
|
||||
peak RSS 40.87 MiB. По одной пробе; фоновые условия
|
||||
различаются, вывод о CPU efficiency не делается. Исторические 45.54 с из другой
|
||||
среды не являются baseline этого сравнения. Все счётчики доставки и replay
|
||||
проверены, а не только открытие порта.
|
||||
|
||||
## Durability и native handoff
|
||||
|
||||
SM WAL и archive redo авторитетны после аварии. Recovery durably adopts полный
|
||||
валидный journal перед публикацией; EIO regression выявляла отсутствие barrier
|
||||
в предварительном кандидате a5f5b3a…, теперь проходит на final binary. Отчёты
|
||||
preliminary-recovery-sync-* сохранены именно как предварительные, не результаты
|
||||
0.8.2 или final. Изначальный native MUC fixture делал in-place append без передачи
|
||||
владения: теперь он явно вызывает checkpoint_all перед оригинальным Prosody.
|
||||
Неjournaled suffix при активном redo намеренно не объявляется external commit.
|
||||
Copy/restore с новым inode сохраняет авторитет redo; внутренний rewrite
|
||||
фиксирует старый native и удаление redo перед atomic replacement. Проверены
|
||||
оба crash boundaries, потеря suffix после копирования и обе baseline длины.
|
||||
Перед native writer/откатом требуется остановка и `storage-checkpoint CONFIG`.
|
||||
[Формат, ограничения и эксплуатация](DURABLE-MESSAGE-IO.md).
|
||||
|
||||
Release SHA-256: `8ceb8de3a2d02bc4399b8ae817dff7e3c3fe9bff83dc0c6669b7f19210a3847a`.
|
||||
Один final verify с XMPP_LOAD_CHECK=1 и native env; отдельно archive ignored test
|
||||
и подробные latency/crash/EIO fixtures. Исторические результаты сохранены.
|
||||
Оставшийся AF_UNIX ignored test не запускался; 10 GiB upload и реальный MinIO/AWS,
|
||||
Docker build/run, GitHub/Gitea publish и настоящие APNs/FCM не повторялись.
|
||||
Общие границы полной совместимости: [COMPATIBILITY.md](COMPATIBILITY.md).
|
||||
+68
-40
@@ -1,51 +1,79 @@
|
||||
# Проверка0.8.2
|
||||
# Проверка 0.8.3
|
||||
|
||||
2026-10-21, Linux x86_64, Rust1.99.1. Итоговый единый прогон:
|
||||
[verify.txt](validation-logs-0.8.2/verify.txt). Machine-readable report:
|
||||
[validation-results.json](validation-results.json).
|
||||
2026-10-21, Linux x86_64, Rust 1.99.1. Итоговый единый прогон после заморозки
|
||||
исходников: [verify.txt](validation-logs-0.8.3/verify.txt).
|
||||
[Machine-readable results](validation-results-0.8.3.json).
|
||||
|
||||
|Проверка|Результат|
|
||||
|---|---|
|
||||
|Rust ordinary tests|305passed,0failed,11ignored|
|
||||
|Rust ordinary tests|334 passed, 0 failed, 11 ignored|
|
||||
|fmt / Clippy all-targets -D warnings / release|Успешно|
|
||||
|Новые Rust regressions|6archive endpoints/recovery/limits +2account-lock/duplicateACK — успешно|
|
||||
|Личная доставка|6fixtures:100000MAM/user, roster0/5000/10000,500SMpending, default/native CSI — успешно|
|
||||
|Неподвижный push gateway|Настоящий STARTTLS; hibernating+offline: bothmessagesdurable, senderACKh=4 до ответа gateway; offline сохранён до первого postTLSstream|
|
||||
|Bind2 и MAM metadata|100000records,5reconnects+5IQ, exactIDs/timestamps — успешно|
|
||||
|Сетевые регрессии|Весь verify-local.sh, включаяprivate WSS/BOSH chat, PEP/push,AndroidSASL,MUCcrash/SMrestart,S3/migration — успешно|
|
||||
|Native reference|8module/storage +1TLS/SASL EXTERNAL federation — успешно|
|
||||
|Общая нагрузка|500TLS,100accounts,5000ping,5000messages,10000MAM,failures=[]|
|
||||
|MUC+SM нагрузка|500resources,10rooms,25000presence,1500messages,SIGKILL+10resume — успешно|
|
||||
|Новые Rust regressions|13 archive redo + 2 rewrite crash boundaries + 14 SM journal, успешно|
|
||||
|Native reference|8 module/storage + 1 TLS/SASL EXTERNAL federation, успешно|
|
||||
|Archive >128 MiB|Отдельно выполненный ignored streaming test, успешно|
|
||||
|Copy/restore и rewrite|Новый inode, потеря suffix, baseline0/nonzero, crash до/после rename — успешно|
|
||||
|Все network regressions verify-local.sh|Успешно, включая Android, WSS/BOSH, PEP, push, MUC crash/SM resume, upload/migration|
|
||||
|Slow durable IO|0/100 мс: 3 samples/scenario; 200 мс: 1 sample/scenario; exact MAM/carbons/queues и phase logs проверены|
|
||||
|SIGKILL сразу после sender SM ACK|100 мс/sync, 3 сообщения, оба MAM архива, 4 resume, 9 replay, до reauth; успешно|
|
||||
|Recovery sync EIO|WAL/redo: отказ handoff без удаления journal; WAL startup до listeners; retry и данные проверены|
|
||||
|Offline storage-checkpoint|4 SM exports, 2 archives, durable journal removal, повтор идемпотентен|
|
||||
|Общая нагрузка|500 TLS, 100 accounts, 5000 ping, 5000 messages, 10000 MAM, failures=[]|
|
||||
|MUC+SM нагрузка|500 resources, 10 rooms, 25000 presence, 1500 deliveries, SIGKILL и 10 resume; успешно|
|
||||
|
||||
MUC+SM total 45.54с, peak sampled RSS 40.94MiB.
|
||||
## Личная доставка
|
||||
|
||||
Воспроизведение личного push-stall: прежний0.8.1 за2с сохраняет только первое
|
||||
сообщение и не отвечаетSMACK при зависшем postTLSfeatures gateway. Новый0.8.2
|
||||
сохраняет оба и отвечаетh=4 в том же2сокне; тестовое окно не является latency SLA.
|
||||
Offline ordering baseline проверен на0.8.0; его точный hash указан отдельно.
|
||||
Медианы для 3 samples, при 200 мс — одна проба; числа в миллисекундах.
|
||||
|
||||
Bind2 authenticate→success median2742.353→1.976мс; metadataIQ2437.945→0.355мс.
|
||||
Baseline0.8.0hash74882319… содержит тот же прежний scan path. Все raw samples,
|
||||
IDs, timestamps и hashes — validation-logs-0.8.2/bind2-*.json.
|
||||
|Sync delay / сценарий|Body 0.8.2 → 0.8.3|Sender SM ACK 0.8.2 → 0.8.3|Проб|
|
||||
|---|---:|---:|---:|
|
||||
|0 мс / 1:1|25.11 → 7.87|33.41 → 15.08|3|
|
||||
|0 мс / PEP → 1:1|29.71 → 20.94|32.25 → 21.86|3|
|
||||
|100 мс / 1:1|1232.47 → 419.68|1641.60 → 644.08|3|
|
||||
|100 мс / PEP → 1:1|2683.29 → 1378.23|3195.21 → 1581.98|3|
|
||||
|200 мс / 1:1|2836.64 → 2042.56|3643.30 → 2441.99|1|
|
||||
|200 мс / PEP → 1:1|5782.02 → 2676.28|6595.92 → 3094.95|1|
|
||||
|
||||
Default inactive OMEMO доставлен за2.58–4.06мс, receipts7.36–35.56мс в checked
|
||||
fixtures; explicit[] сохраняет удержание доactive. Намеренный CSI dwell2с
|
||||
отдельно обозначен в отчёте и не выдаётся за время вычисления/доставки.
|
||||
Каждый personalarchive содержит ровно100523records/user. Детали и настройка:
|
||||
[PERSONAL-MESSAGE-LATENCY.md](PERSONAL-MESSAGE-LATENCY.md).
|
||||
0 мс — отдельная последовательная quiet pair, без full verify. Финальные
|
||||
100/200 мс выполнялись при фоновой работе verify; это не изолированный CPU benchmark.
|
||||
Первый append включает создание/migration MAM journal. В обычном warm сообщении
|
||||
6 sync вместо 16; первый — 12. Окна счётчиков могут включать heartbeat, а PEP
|
||||
сценарий включает предшествующую publication. Реальные fsync/fdatasync сохранены;
|
||||
interposer замедляет только isolated data_path, не меняет host mount/DNS.
|
||||
Raw samples, hashes, sync counts и phase fields — validation-logs-0.8.3/*.json.
|
||||
|
||||
Release SHA-256: `130431025c4c7c2f72e28b77a31c83867f857ef8bb56d1f71b10e4b19d7caaf5`. Все новые final reports содержат этот hash.
|
||||
Команда: `XMPP_LOAD_CHECK=1 bash scripts/verify-local.sh` с native env согласно
|
||||
scripts/prosody_compat/README.md. Использованы CARGO_INCREMENTAL=0,
|
||||
CARGO_PROFILE_DEV_DEBUG=0, CARGO_PROFILE_TEST_DEBUG=0. Перед итоговым прогоном
|
||||
устранены stale package linker artifacts очисткой package build outputs;
|
||||
исходники не менялись после финальной сборки. Воспроизведение прошло одним
|
||||
последующим полным прогоном; preliminary compiler errors не являются test failures.
|
||||
В production log 0.8.2 серверный handler занимал 4792–5549 мс; старых полей
|
||||
недостаточно, чтобы доказать конкретный IO stage. Новый таймер разделяет MAM,
|
||||
доставку, carbons и handled, без payload. Эти loopback результаты не доказывают
|
||||
время отображения реальным Conversations/Monocles на wthome.ru.
|
||||
|
||||
Это loopback, не production клиента/диска/сети. Fsync и durable commit до
|
||||
подтверждения сохранены; boundedpushqueue не обещает durable notification delivery.
|
||||
SMglobalIOmutex, полные MAMqueries и maintenance остаются отдельной стоимостью.
|
||||
9nativeignoredtests выполнены; остальные AF_UNIX/largearchiveignoredtests и
|
||||
10GiBtransfers в этом локальном прогоне не запускались (полный CI entrypoint
|
||||
по-прежнему включает их). Старые largeupload результаты не приписываются0.8.2.
|
||||
Общие границы: [COMPATIBILITY.md](COMPATIBILITY.md).
|
||||
## Нагрузка и сопоставимость
|
||||
|
||||
Сопоставимый повтор 0.8.2 в текущей среде: total 151.334 с,
|
||||
join 104.226 с. Финальный 0.8.3: total 128.380 с,
|
||||
join 94.493 с, restart 7.557 с,
|
||||
peak RSS 40.87 MiB. По одной пробе; фоновые условия
|
||||
различаются, вывод о CPU efficiency не делается. Исторические 45.54 с из другой
|
||||
среды не являются baseline этого сравнения. Все счётчики доставки и replay
|
||||
проверены, а не только открытие порта.
|
||||
|
||||
## Durability и native handoff
|
||||
|
||||
SM WAL и archive redo авторитетны после аварии. Recovery durably adopts полный
|
||||
валидный journal перед публикацией; EIO regression выявляла отсутствие barrier
|
||||
в предварительном кандидате a5f5b3a…, теперь проходит на final binary. Отчёты
|
||||
preliminary-recovery-sync-* сохранены именно как предварительные, не результаты
|
||||
0.8.2 или final. Изначальный native MUC fixture делал in-place append без передачи
|
||||
владения: теперь он явно вызывает checkpoint_all перед оригинальным Prosody.
|
||||
Неjournaled suffix при активном redo намеренно не объявляется external commit.
|
||||
Copy/restore с новым inode сохраняет авторитет redo; внутренний rewrite
|
||||
фиксирует старый native и удаление redo перед atomic replacement. Проверены
|
||||
оба crash boundaries, потеря suffix после копирования и обе baseline длины.
|
||||
Перед native writer/откатом требуется остановка и `storage-checkpoint CONFIG`.
|
||||
[Формат, ограничения и эксплуатация](DURABLE-MESSAGE-IO.md).
|
||||
|
||||
Release SHA-256: `8ceb8de3a2d02bc4399b8ae817dff7e3c3fe9bff83dc0c6669b7f19210a3847a`.
|
||||
Один final verify с XMPP_LOAD_CHECK=1 и native env; отдельно archive ignored test
|
||||
и подробные latency/crash/EIO fixtures. Исторические результаты сохранены.
|
||||
Оставшийся AF_UNIX ignored test не запускался; 10 GiB upload и реальный MinIO/AWS,
|
||||
Docker build/run, GitHub/Gitea publish и настоящие APNs/FCM не повторялись.
|
||||
Общие границы полной совместимости: [COMPATIBILITY.md](COMPATIBILITY.md).
|
||||
@@ -0,0 +1,2 @@
|
||||
Checking rust-xmpp v0.8.3 (/workspace/scratch/362577cdf2e0/recovered/rust-xmpp)
|
||||
Finished `dev` profile [unoptimized] target(s) in 14.04s
|
||||
@@ -0,0 +1,35 @@
|
||||
{
|
||||
"status": "pass",
|
||||
"delay_ms_per_sync": 100,
|
||||
"acknowledged_messages": 3,
|
||||
"mam_records_per_account": [
|
||||
3,
|
||||
3
|
||||
],
|
||||
"resumed_resources": 4,
|
||||
"replayed_personal_deliveries": 9,
|
||||
"offline_checkpoint": {
|
||||
"exported_sessions": 4,
|
||||
"native_archives": 2,
|
||||
"journals_removed": true,
|
||||
"second_invocation_idempotent": true
|
||||
},
|
||||
"pending_before_reauthentication": [
|
||||
2,
|
||||
3,
|
||||
4,
|
||||
3
|
||||
],
|
||||
"sync_between_publication_and_sigkill": {
|
||||
"total": 32,
|
||||
"by_store_kind_op": {
|
||||
"mam:directory:fsync": 2,
|
||||
"mam:file:fsync": 10,
|
||||
"pep:directory:fsync": 2,
|
||||
"pep:file:fsync": 2,
|
||||
"sm:file:fdatasync": 16
|
||||
}
|
||||
},
|
||||
"binary_sha256": "8ceb8de3a2d02bc4399b8ae817dff7e3c3fe9bff83dc0c6669b7f19210a3847a",
|
||||
"scope": "loopback STARTTLS; real delayed fsync/fdatasync; sender SM acknowledgement then SIGKILL; authoritative queues and MAM checked before reauthentication; at-least-once replay"
|
||||
}
|
||||
@@ -0,0 +1,9 @@
|
||||
Compiling rust-xmpp v0.8.3 (/workspace/scratch/362577cdf2e0/recovered/rust-xmpp)
|
||||
Finished `test` profile [unoptimized] target(s) in 3.26s
|
||||
Running tests/archive_mam.rs (target/debug/deps/archive_mam-1243a5e0f6034fe2)
|
||||
|
||||
running 1 test
|
||||
test archive_larger_than_128mib_streams ... ok
|
||||
|
||||
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 13 filtered out; finished in 2.76s
|
||||
|
||||
@@ -0,0 +1,62 @@
|
||||
{
|
||||
"clients": 500,
|
||||
"accounts": 100,
|
||||
"rooms": 10,
|
||||
"message_rounds": 3,
|
||||
"concurrency": 20,
|
||||
"binary": "/tmp/xmpp-durable-muc-load-ssixqqk9/rust-xmpp",
|
||||
"binary_sha256": "8ceb8de3a2d02bc4399b8ae817dff7e3c3fe9bff83dc0c6669b7f19210a3847a",
|
||||
"before": {
|
||||
"rss_kib": 9516,
|
||||
"cpu_seconds": 0.01,
|
||||
"fd_count": 12
|
||||
},
|
||||
"connect_seconds": 12.048,
|
||||
"connect_latency": {
|
||||
"samples": 500,
|
||||
"median_ms": 398.044,
|
||||
"p95_ms": 1086.236,
|
||||
"max_ms": 1403.7
|
||||
},
|
||||
"phase": "completed",
|
||||
"status": "pass",
|
||||
"elapsed_seconds": 128.38,
|
||||
"join_seconds": 94.493,
|
||||
"joined": {
|
||||
"rss_kib": 41828,
|
||||
"cpu_seconds": 55.61,
|
||||
"fd_count": 513
|
||||
},
|
||||
"join_presence_deliveries": 25000,
|
||||
"slots_joined": {
|
||||
"count": 500,
|
||||
"bytes": 309790
|
||||
},
|
||||
"message_seconds": 9.585,
|
||||
"room_messages_sent": 30,
|
||||
"room_messages_delivered": 1500,
|
||||
"before_kill": {
|
||||
"rss_kib": 41740,
|
||||
"cpu_seconds": 62.6,
|
||||
"fd_count": 512
|
||||
},
|
||||
"slots_before_kill": {
|
||||
"count": 500,
|
||||
"bytes": 514030
|
||||
},
|
||||
"restart_seconds": 7.557,
|
||||
"resumed_resources": 10,
|
||||
"replayed_pending_room_messages": 10,
|
||||
"membership_groupchat_deliveries_after_resume": 10,
|
||||
"after_resume": {
|
||||
"rss_kib": 20896,
|
||||
"cpu_seconds": 7.09,
|
||||
"fd_count": 22
|
||||
},
|
||||
"slots_after_resume": {
|
||||
"count": 500,
|
||||
"bytes": 671070
|
||||
},
|
||||
"total_seconds": 128.38,
|
||||
"peak_rss_kib": 41852
|
||||
}
|
||||
@@ -0,0 +1,62 @@
|
||||
{
|
||||
"clients": 500,
|
||||
"accounts": 100,
|
||||
"rooms": 10,
|
||||
"message_rounds": 3,
|
||||
"concurrency": 20,
|
||||
"binary": "/tmp/xmpp-durable-muc-load-wgcesopv/rust-xmpp",
|
||||
"binary_sha256": "130431025c4c7c2f72e28b77a31c83867f857ef8bb56d1f71b10e4b19d7caaf5",
|
||||
"before": {
|
||||
"rss_kib": 9468,
|
||||
"cpu_seconds": 0.0,
|
||||
"fd_count": 12
|
||||
},
|
||||
"connect_seconds": 8.677,
|
||||
"connect_latency": {
|
||||
"samples": 500,
|
||||
"median_ms": 337.216,
|
||||
"p95_ms": 475.311,
|
||||
"max_ms": 632.696
|
||||
},
|
||||
"phase": "completed",
|
||||
"status": "pass",
|
||||
"elapsed_seconds": 151.334,
|
||||
"join_seconds": 104.226,
|
||||
"joined": {
|
||||
"rss_kib": 41468,
|
||||
"cpu_seconds": 39.74,
|
||||
"fd_count": 513
|
||||
},
|
||||
"join_presence_deliveries": 25000,
|
||||
"slots_joined": {
|
||||
"count": 500,
|
||||
"bytes": 309790
|
||||
},
|
||||
"message_seconds": 17.857,
|
||||
"room_messages_sent": 30,
|
||||
"room_messages_delivered": 1500,
|
||||
"before_kill": {
|
||||
"rss_kib": 41464,
|
||||
"cpu_seconds": 48.58,
|
||||
"fd_count": 512
|
||||
},
|
||||
"slots_before_kill": {
|
||||
"count": 500,
|
||||
"bytes": 514030
|
||||
},
|
||||
"restart_seconds": 10.083,
|
||||
"resumed_resources": 10,
|
||||
"replayed_pending_room_messages": 10,
|
||||
"membership_groupchat_deliveries_after_resume": 10,
|
||||
"after_resume": {
|
||||
"rss_kib": 20660,
|
||||
"cpu_seconds": 9.73,
|
||||
"fd_count": 22
|
||||
},
|
||||
"slots_after_resume": {
|
||||
"count": 500,
|
||||
"bytes": 671070
|
||||
},
|
||||
"total_seconds": 151.334,
|
||||
"peak_rss_kib": 41476
|
||||
}
|
||||
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"status": "fail",
|
||||
"cases": [],
|
||||
"binary_sha256": "a5f5b3a552be44fb8ab8b6525dc228bc2dc97f81aecfbc559e12cc29b3e8c00b",
|
||||
"failure": ".rredo recovery skipped journal sync and falsely reported successful handoff"
|
||||
}
|
||||
@@ -0,0 +1,6 @@
|
||||
{
|
||||
"status": "fail",
|
||||
"cases": [],
|
||||
"binary_sha256": "a5f5b3a552be44fb8ab8b6525dc228bc2dc97f81aecfbc559e12cc29b3e8c00b",
|
||||
"failure": ".wal recovery skipped journal sync and falsely reported successful handoff"
|
||||
}
|
||||
@@ -0,0 +1,95 @@
|
||||
{
|
||||
"binary_source": "/workspace/scratch/362577cdf2e0/recovered/rust-xmpp/target/release/rust-xmpp",
|
||||
"binary_sha256": "8ceb8de3a2d02bc4399b8ae817dff7e3c3fe9bff83dc0c6669b7f19210a3847a",
|
||||
"scope": "real loopback TCP/STARTTLS/SASL2 Bind2; isolated personal MAM fixture; no production latency claim",
|
||||
"timing": "raw ordered samples; auth timer starts before authenticate and ends at parsed success; full timer includes TCP/STARTTLS/features; metadata IQ timer runs from send to parsed result",
|
||||
"result": {
|
||||
"status": "pass",
|
||||
"seeded_personal_archive_records": 10000,
|
||||
"archive_bytes": 2138890,
|
||||
"archive_expires_after": "never",
|
||||
"index_present_before_start": false,
|
||||
"reconnect_samples": 2,
|
||||
"expected_metadata": {
|
||||
"start": {
|
||||
"id": "seed-0",
|
||||
"timestamp": "2026-01-01T00:00:00Z"
|
||||
},
|
||||
"end": {
|
||||
"id": "seed-9999",
|
||||
"timestamp": "2026-01-01T02:46:39Z"
|
||||
}
|
||||
},
|
||||
"observed_metadata": [
|
||||
{
|
||||
"start": {
|
||||
"id": "seed-0",
|
||||
"timestamp": "2026-01-01T00:00:00Z"
|
||||
},
|
||||
"end": {
|
||||
"id": "seed-9999",
|
||||
"timestamp": "2026-01-01T02:46:39Z"
|
||||
}
|
||||
},
|
||||
{
|
||||
"start": {
|
||||
"id": "seed-0",
|
||||
"timestamp": "2026-01-01T00:00:00Z"
|
||||
},
|
||||
"end": {
|
||||
"id": "seed-9999",
|
||||
"timestamp": "2026-01-01T02:46:39Z"
|
||||
}
|
||||
}
|
||||
],
|
||||
"mam_metadata_iq": {
|
||||
"samples_ms": [
|
||||
1.066574000105902,
|
||||
0.6786549997741531
|
||||
],
|
||||
"median_ms": 0.8726144999400276,
|
||||
"p95_ms": 1.066574000105902,
|
||||
"max_ms": 1.066574000105902
|
||||
},
|
||||
"observed_iq_metadata": [
|
||||
{
|
||||
"start": {
|
||||
"id": "seed-0",
|
||||
"timestamp": "2026-01-01T00:00:00Z"
|
||||
},
|
||||
"end": {
|
||||
"id": "seed-9999",
|
||||
"timestamp": "2026-01-01T02:46:39Z"
|
||||
}
|
||||
},
|
||||
{
|
||||
"start": {
|
||||
"id": "seed-0",
|
||||
"timestamp": "2026-01-01T00:00:00Z"
|
||||
},
|
||||
"end": {
|
||||
"id": "seed-9999",
|
||||
"timestamp": "2026-01-01T02:46:39Z"
|
||||
}
|
||||
}
|
||||
],
|
||||
"sasl2_bind2_success": {
|
||||
"samples_ms": [
|
||||
5.694160000075499,
|
||||
4.350195999904827
|
||||
],
|
||||
"median_ms": 5.022177999990163,
|
||||
"p95_ms": 5.694160000075499,
|
||||
"max_ms": 5.694160000075499
|
||||
},
|
||||
"tcp_starttls_to_bind2_success": {
|
||||
"samples_ms": [
|
||||
51.93691700014824,
|
||||
9.144003000074008
|
||||
],
|
||||
"median_ms": 30.540460000111125,
|
||||
"p95_ms": 51.93691700014824,
|
||||
"max_ms": 51.93691700014824
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,418 @@
|
||||
{
|
||||
"binary_sha256": "8ceb8de3a2d02bc4399b8ae817dff7e3c3fe9bff83dc0c6669b7f19210a3847a",
|
||||
"scope": "loopback STARTTLS; fixture only; forced CSI dwell explicitly separated",
|
||||
"cases": [
|
||||
{
|
||||
"active": {
|
||||
"delivery": {
|
||||
"samples": 1,
|
||||
"median_ms": 42.84763399982694,
|
||||
"p95_ms": 42.84763399982694,
|
||||
"max_ms": 42.84763399982694,
|
||||
"samples_ms": [
|
||||
42.84763399982694
|
||||
]
|
||||
},
|
||||
"sender_sm_handled": {
|
||||
"samples": 1,
|
||||
"median_ms": 41.45799199977773,
|
||||
"p95_ms": 41.45799199977773,
|
||||
"max_ms": 41.45799199977773,
|
||||
"samples_ms": [
|
||||
41.45799199977773
|
||||
]
|
||||
},
|
||||
"client_receipt_end_to_end": {
|
||||
"samples": 1,
|
||||
"median_ms": 51.320845000191184,
|
||||
"p95_ms": 51.320845000191184,
|
||||
"max_ms": 51.320845000191184,
|
||||
"samples_ms": [
|
||||
51.320845000191184
|
||||
]
|
||||
},
|
||||
"receipt_return_leg": {
|
||||
"samples": 1,
|
||||
"median_ms": 8.362257000044337,
|
||||
"p95_ms": 8.362257000044337,
|
||||
"max_ms": 8.362257000044337,
|
||||
"samples_ms": [
|
||||
8.362257000044337
|
||||
]
|
||||
}
|
||||
},
|
||||
"inactive": {
|
||||
"delivery": {
|
||||
"samples": 1,
|
||||
"median_ms": 7.461921999947663,
|
||||
"p95_ms": 7.461921999947663,
|
||||
"max_ms": 7.461921999947663,
|
||||
"samples_ms": [
|
||||
7.461921999947663
|
||||
]
|
||||
},
|
||||
"sender_sm_handled": {
|
||||
"samples": 1,
|
||||
"median_ms": 6.771670000034646,
|
||||
"p95_ms": 6.771670000034646,
|
||||
"max_ms": 6.771670000034646,
|
||||
"samples_ms": [
|
||||
6.771670000034646
|
||||
]
|
||||
},
|
||||
"client_receipt_end_to_end": {
|
||||
"samples": 1,
|
||||
"median_ms": 12.700967000000674,
|
||||
"p95_ms": 12.700967000000674,
|
||||
"max_ms": 12.700967000000674,
|
||||
"samples_ms": [
|
||||
12.700967000000674
|
||||
]
|
||||
},
|
||||
"receipt_return_leg": {
|
||||
"samples": 1,
|
||||
"median_ms": 5.201973000112048,
|
||||
"p95_ms": 5.201973000112048,
|
||||
"max_ms": 5.201973000112048,
|
||||
"samples_ms": [
|
||||
5.201973000112048
|
||||
]
|
||||
}
|
||||
},
|
||||
"receipt_only-active": {
|
||||
"delivery_ms": 23.464398000214715,
|
||||
"sender_sm_handled_ms": 23.388774000068224,
|
||||
"durably_csi_buffered": false,
|
||||
"forced_dwell_seconds": 0,
|
||||
"after_activation_ms": null
|
||||
},
|
||||
"receipt_only-inactive": {
|
||||
"delivery_ms": 11.619346000316,
|
||||
"sender_sm_handled_ms": 11.170642000251974,
|
||||
"durably_csi_buffered": false,
|
||||
"forced_dwell_seconds": 0,
|
||||
"after_activation_ms": null
|
||||
},
|
||||
"omemo_legacy_only-active": {
|
||||
"delivery_ms": 3.3123490002253675,
|
||||
"sender_sm_handled_ms": 3.2682549999663024,
|
||||
"durably_csi_buffered": false,
|
||||
"forced_dwell_seconds": 0,
|
||||
"after_activation_ms": null
|
||||
},
|
||||
"omemo_legacy_only-inactive": {
|
||||
"delivery_ms": 5.355494000014005,
|
||||
"sender_sm_handled_ms": 4.585807000239583,
|
||||
"durably_csi_buffered": false,
|
||||
"forced_dwell_seconds": 0,
|
||||
"after_activation_ms": null
|
||||
},
|
||||
"omemo_2_only-active": {
|
||||
"delivery_ms": 6.6929370000252675,
|
||||
"sender_sm_handled_ms": 5.942287999914697,
|
||||
"durably_csi_buffered": false,
|
||||
"forced_dwell_seconds": 0,
|
||||
"after_activation_ms": null
|
||||
},
|
||||
"omemo_2_only-inactive": {
|
||||
"delivery_ms": 7.484531000045536,
|
||||
"sender_sm_handled_ms": 7.103434999862657,
|
||||
"durably_csi_buffered": false,
|
||||
"forced_dwell_seconds": 0,
|
||||
"after_activation_ms": null
|
||||
},
|
||||
"omemo_with_eme-active": {
|
||||
"delivery_ms": 8.187372000065807,
|
||||
"sender_sm_handled_ms": 7.642809000117268,
|
||||
"durably_csi_buffered": false,
|
||||
"forced_dwell_seconds": 0,
|
||||
"after_activation_ms": null
|
||||
},
|
||||
"omemo_with_eme-inactive": {
|
||||
"delivery_ms": 8.972958999947878,
|
||||
"sender_sm_handled_ms": 8.508876000178134,
|
||||
"durably_csi_buffered": false,
|
||||
"forced_dwell_seconds": 0,
|
||||
"after_activation_ms": null
|
||||
},
|
||||
"marker_received-active": {
|
||||
"delivery_ms": 7.674009000311344,
|
||||
"sender_sm_handled_ms": 7.3800410000330885,
|
||||
"durably_csi_buffered": false,
|
||||
"forced_dwell_seconds": 0,
|
||||
"after_activation_ms": null
|
||||
},
|
||||
"marker_received-inactive": {
|
||||
"delivery_ms": 5.613406000065879,
|
||||
"sender_sm_handled_ms": 5.646823999995831,
|
||||
"durably_csi_buffered": false,
|
||||
"forced_dwell_seconds": 0,
|
||||
"after_activation_ms": null
|
||||
},
|
||||
"marker_displayed-active": {
|
||||
"delivery_ms": 5.180439000014303,
|
||||
"sender_sm_handled_ms": 4.853748000186897,
|
||||
"durably_csi_buffered": false,
|
||||
"forced_dwell_seconds": 0,
|
||||
"after_activation_ms": null
|
||||
},
|
||||
"marker_displayed-inactive": {
|
||||
"delivery_ms": 8.705914999609377,
|
||||
"sender_sm_handled_ms": 8.28981999984535,
|
||||
"durably_csi_buffered": false,
|
||||
"forced_dwell_seconds": 0,
|
||||
"after_activation_ms": null
|
||||
},
|
||||
"marker_acknowledged-active": {
|
||||
"delivery_ms": 6.910242000230937,
|
||||
"sender_sm_handled_ms": 6.799871000112034,
|
||||
"durably_csi_buffered": false,
|
||||
"forced_dwell_seconds": 0,
|
||||
"after_activation_ms": null
|
||||
},
|
||||
"marker_acknowledged-inactive": {
|
||||
"delivery_ms": 8.707511000011436,
|
||||
"sender_sm_handled_ms": 8.398837999720854,
|
||||
"durably_csi_buffered": false,
|
||||
"forced_dwell_seconds": 0,
|
||||
"after_activation_ms": null
|
||||
},
|
||||
"low_priority_headline-active": {
|
||||
"delivery_ms": 4.267847999926744,
|
||||
"sender_sm_handled_ms": 4.236566000145103,
|
||||
"durably_csi_buffered": false,
|
||||
"forced_dwell_seconds": 0,
|
||||
"after_activation_ms": null
|
||||
},
|
||||
"low_priority_headline-inactive": {
|
||||
"delivery_ms": 108.03854200003116,
|
||||
"sender_sm_handled_ms": 2.853955999853497,
|
||||
"durably_csi_buffered": true,
|
||||
"forced_dwell_seconds": 0.1,
|
||||
"after_activation_ms": 3.106009000021004
|
||||
},
|
||||
"sm_backlog": {
|
||||
"seeded_messages": 30,
|
||||
"durable_pending_before_sample": 30,
|
||||
"delivery_ms": 8.070619000136503,
|
||||
"sender_sm_handled_ms": 7.959626000229036
|
||||
},
|
||||
"status": "pass",
|
||||
"final_personal_mam_records_per_user": [
|
||||
1045,
|
||||
1045
|
||||
],
|
||||
"csi_policy": "default",
|
||||
"baseline_expect_held": false,
|
||||
"roster_contacts_per_user": 0,
|
||||
"seeded_personal_mam_records_per_user": 1000,
|
||||
"archive_policy": "always"
|
||||
},
|
||||
{
|
||||
"active": {
|
||||
"delivery": {
|
||||
"samples": 1,
|
||||
"median_ms": 9.351153999887174,
|
||||
"p95_ms": 9.351153999887174,
|
||||
"max_ms": 9.351153999887174,
|
||||
"samples_ms": [
|
||||
9.351153999887174
|
||||
]
|
||||
},
|
||||
"sender_sm_handled": {
|
||||
"samples": 1,
|
||||
"median_ms": 9.169040999950084,
|
||||
"p95_ms": 9.169040999950084,
|
||||
"max_ms": 9.169040999950084,
|
||||
"samples_ms": [
|
||||
9.169040999950084
|
||||
]
|
||||
},
|
||||
"client_receipt_end_to_end": {
|
||||
"samples": 1,
|
||||
"median_ms": 13.487032000284671,
|
||||
"p95_ms": 13.487032000284671,
|
||||
"max_ms": 13.487032000284671,
|
||||
"samples_ms": [
|
||||
13.487032000284671
|
||||
]
|
||||
},
|
||||
"receipt_return_leg": {
|
||||
"samples": 1,
|
||||
"median_ms": 4.101098000319325,
|
||||
"p95_ms": 4.101098000319325,
|
||||
"max_ms": 4.101098000319325,
|
||||
"samples_ms": [
|
||||
4.101098000319325
|
||||
]
|
||||
}
|
||||
},
|
||||
"inactive": {
|
||||
"delivery": {
|
||||
"samples": 1,
|
||||
"median_ms": 5.198949999794422,
|
||||
"p95_ms": 5.198949999794422,
|
||||
"max_ms": 5.198949999794422,
|
||||
"samples_ms": [
|
||||
5.198949999794422
|
||||
]
|
||||
},
|
||||
"sender_sm_handled": {
|
||||
"samples": 1,
|
||||
"median_ms": 5.164645999684581,
|
||||
"p95_ms": 5.164645999684581,
|
||||
"max_ms": 5.164645999684581,
|
||||
"samples_ms": [
|
||||
5.164645999684581
|
||||
]
|
||||
},
|
||||
"client_receipt_end_to_end": {
|
||||
"samples": 1,
|
||||
"median_ms": 13.508577999800764,
|
||||
"p95_ms": 13.508577999800764,
|
||||
"max_ms": 13.508577999800764,
|
||||
"samples_ms": [
|
||||
13.508577999800764
|
||||
]
|
||||
},
|
||||
"receipt_return_leg": {
|
||||
"samples": 1,
|
||||
"median_ms": 8.226426999954128,
|
||||
"p95_ms": 8.226426999954128,
|
||||
"max_ms": 8.226426999954128,
|
||||
"samples_ms": [
|
||||
8.226426999954128
|
||||
]
|
||||
}
|
||||
},
|
||||
"receipt_only-active": {
|
||||
"delivery_ms": 8.37234799973885,
|
||||
"sender_sm_handled_ms": 7.964516999891202,
|
||||
"durably_csi_buffered": false,
|
||||
"forced_dwell_seconds": 0,
|
||||
"after_activation_ms": null
|
||||
},
|
||||
"receipt_only-inactive": {
|
||||
"delivery_ms": 109.20343799989496,
|
||||
"sender_sm_handled_ms": 5.452117000004364,
|
||||
"durably_csi_buffered": true,
|
||||
"forced_dwell_seconds": 0.1,
|
||||
"after_activation_ms": 2.879265000046871
|
||||
},
|
||||
"omemo_legacy_only-active": {
|
||||
"delivery_ms": 4.961447999903612,
|
||||
"sender_sm_handled_ms": 4.207620999750361,
|
||||
"durably_csi_buffered": false,
|
||||
"forced_dwell_seconds": 0,
|
||||
"after_activation_ms": null
|
||||
},
|
||||
"omemo_legacy_only-inactive": {
|
||||
"delivery_ms": 107.77542400001039,
|
||||
"sender_sm_handled_ms": 4.299592999814195,
|
||||
"durably_csi_buffered": true,
|
||||
"forced_dwell_seconds": 0.1,
|
||||
"after_activation_ms": 1.5704200000072888
|
||||
},
|
||||
"omemo_2_only-active": {
|
||||
"delivery_ms": 3.2654749998073385,
|
||||
"sender_sm_handled_ms": 3.3216599999832397,
|
||||
"durably_csi_buffered": false,
|
||||
"forced_dwell_seconds": 0,
|
||||
"after_activation_ms": null
|
||||
},
|
||||
"omemo_2_only-inactive": {
|
||||
"delivery_ms": 109.47085399993739,
|
||||
"sender_sm_handled_ms": 5.282463000185089,
|
||||
"durably_csi_buffered": true,
|
||||
"forced_dwell_seconds": 0.1,
|
||||
"after_activation_ms": 2.073672999813425
|
||||
},
|
||||
"omemo_with_eme-active": {
|
||||
"delivery_ms": 6.262568000238389,
|
||||
"sender_sm_handled_ms": 6.371549000050436,
|
||||
"durably_csi_buffered": false,
|
||||
"forced_dwell_seconds": 0,
|
||||
"after_activation_ms": null
|
||||
},
|
||||
"omemo_with_eme-inactive": {
|
||||
"delivery_ms": 5.960880999737128,
|
||||
"sender_sm_handled_ms": 5.667837000146392,
|
||||
"durably_csi_buffered": false,
|
||||
"forced_dwell_seconds": 0,
|
||||
"after_activation_ms": null
|
||||
},
|
||||
"marker_received-active": {
|
||||
"delivery_ms": 7.435425000039686,
|
||||
"sender_sm_handled_ms": 7.0634520002386125,
|
||||
"durably_csi_buffered": false,
|
||||
"forced_dwell_seconds": 0,
|
||||
"after_activation_ms": null
|
||||
},
|
||||
"marker_received-inactive": {
|
||||
"delivery_ms": 110.35115000004225,
|
||||
"sender_sm_handled_ms": 5.86408199978905,
|
||||
"durably_csi_buffered": true,
|
||||
"forced_dwell_seconds": 0.1,
|
||||
"after_activation_ms": 2.434945000004518
|
||||
},
|
||||
"marker_displayed-active": {
|
||||
"delivery_ms": 3.6287610000727,
|
||||
"sender_sm_handled_ms": 3.696032999869203,
|
||||
"durably_csi_buffered": false,
|
||||
"forced_dwell_seconds": 0,
|
||||
"after_activation_ms": null
|
||||
},
|
||||
"marker_displayed-inactive": {
|
||||
"delivery_ms": 112.26732199975231,
|
||||
"sender_sm_handled_ms": 8.923948999836284,
|
||||
"durably_csi_buffered": true,
|
||||
"forced_dwell_seconds": 0.1,
|
||||
"after_activation_ms": 1.43316000003324
|
||||
},
|
||||
"marker_acknowledged-active": {
|
||||
"delivery_ms": 4.8418340002172044,
|
||||
"sender_sm_handled_ms": 4.916578000120353,
|
||||
"durably_csi_buffered": false,
|
||||
"forced_dwell_seconds": 0,
|
||||
"after_activation_ms": null
|
||||
},
|
||||
"marker_acknowledged-inactive": {
|
||||
"delivery_ms": 110.98447399990619,
|
||||
"sender_sm_handled_ms": 5.764318000274216,
|
||||
"durably_csi_buffered": true,
|
||||
"forced_dwell_seconds": 0.1,
|
||||
"after_activation_ms": 3.3109219998550543
|
||||
},
|
||||
"low_priority_headline-active": {
|
||||
"delivery_ms": 4.907228000320174,
|
||||
"sender_sm_handled_ms": 4.593587000272237,
|
||||
"durably_csi_buffered": false,
|
||||
"forced_dwell_seconds": 0,
|
||||
"after_activation_ms": null
|
||||
},
|
||||
"low_priority_headline-inactive": {
|
||||
"delivery_ms": 106.62631800005329,
|
||||
"sender_sm_handled_ms": 2.990170999964903,
|
||||
"durably_csi_buffered": true,
|
||||
"forced_dwell_seconds": 0.1,
|
||||
"after_activation_ms": 1.6857880000316072
|
||||
},
|
||||
"sm_backlog": {
|
||||
"seeded_messages": 30,
|
||||
"durable_pending_before_sample": 30,
|
||||
"delivery_ms": 8.221272999890061,
|
||||
"sender_sm_handled_ms": 8.186153000224294
|
||||
},
|
||||
"status": "pass",
|
||||
"final_personal_mam_records_per_user": [
|
||||
1045,
|
||||
1045
|
||||
],
|
||||
"csi_policy": "native",
|
||||
"baseline_expect_held": false,
|
||||
"roster_contacts_per_user": 0,
|
||||
"seeded_personal_mam_records_per_user": 1000,
|
||||
"archive_policy": "always"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
{
|
||||
"status": "pass",
|
||||
"recipient_state": "hibernating",
|
||||
"durable_pending_at_gateway_stream": {
|
||||
"push-first": true,
|
||||
"push-second": true
|
||||
},
|
||||
"observation_seconds": 2,
|
||||
"elapsed_seconds": 2.013597706999917,
|
||||
"gateway_post_tls_stream_observed": true,
|
||||
"gateway_post_tls_stream_at_bound": true,
|
||||
"blocked_stage": "post-tls-features",
|
||||
"durable_pending": {
|
||||
"push-first": true,
|
||||
"push-second": true
|
||||
},
|
||||
"sender_sm_ack_h": [
|
||||
4
|
||||
],
|
||||
"sender_inputs": 4,
|
||||
"sender_handled_before": 2,
|
||||
"binary_version": "rust-xmpp 0.8.3 \u2014 experimental XMPP runtime",
|
||||
"binary_sha256": "8ceb8de3a2d02bc4399b8ae817dff7e3c3fe9bff83dc0c6669b7f19210a3847a",
|
||||
"scope": "loopback TLS fixture; stalled external XMPP push service; no APNs/FCM"
|
||||
}
|
||||
@@ -0,0 +1,25 @@
|
||||
{
|
||||
"status": "pass",
|
||||
"recipient_state": "offline",
|
||||
"durable_pending_at_gateway_stream": {
|
||||
"push-first": true,
|
||||
"push-second": true
|
||||
},
|
||||
"observation_seconds": 2,
|
||||
"elapsed_seconds": 2.1132061280000016,
|
||||
"gateway_post_tls_stream_observed": true,
|
||||
"gateway_post_tls_stream_at_bound": true,
|
||||
"blocked_stage": "post-tls-features",
|
||||
"durable_pending": {
|
||||
"push-first": true,
|
||||
"push-second": true
|
||||
},
|
||||
"sender_sm_ack_h": [
|
||||
4
|
||||
],
|
||||
"sender_inputs": 4,
|
||||
"sender_handled_before": 2,
|
||||
"binary_version": "rust-xmpp 0.8.3 \u2014 experimental XMPP runtime",
|
||||
"binary_sha256": "8ceb8de3a2d02bc4399b8ae817dff7e3c3fe9bff83dc0c6669b7f19210a3847a",
|
||||
"scope": "loopback TLS fixture; stalled external XMPP push service; no APNs/FCM"
|
||||
}
|
||||
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"status": "pass",
|
||||
"cases": [
|
||||
{
|
||||
"suffix": ".wal",
|
||||
"status": "pass",
|
||||
"checkpoint_rejected_sync_failure": true,
|
||||
"journals_preserved_on_failure": 4,
|
||||
"serve_rejected_before_listeners": true,
|
||||
"clean_retry_preserved_sessions": 4,
|
||||
"clean_retry_preserved_archives": 2
|
||||
},
|
||||
{
|
||||
"suffix": ".rredo",
|
||||
"status": "pass",
|
||||
"checkpoint_rejected_sync_failure": true,
|
||||
"journals_preserved_on_failure": 2,
|
||||
"serve_rejected_before_listeners": false,
|
||||
"clean_retry_preserved_sessions": 4,
|
||||
"clean_retry_preserved_archives": 2
|
||||
}
|
||||
],
|
||||
"binary_sha256": "8ceb8de3a2d02bc4399b8ae817dff7e3c3fe9bff83dc0c6669b7f19210a3847a"
|
||||
}
|
||||
@@ -0,0 +1,2 @@
|
||||
Compiling rust-xmpp v0.8.3 (/workspace/scratch/362577cdf2e0/recovered/rust-xmpp)
|
||||
Finished `release` profile [optimized] target(s) in 1m 43s
|
||||
@@ -0,0 +1,99 @@
|
||||
{
|
||||
"binary_sha256": "8ceb8de3a2d02bc4399b8ae817dff7e3c3fe9bff83dc0c6669b7f19210a3847a",
|
||||
"scope": "loopback STARTTLS; four SM resources/two accounts; MAM + pending avatar metadata PEP; fixture-only fsync/fdatasync including data directories; real sync retained; operation windows include background SM heartbeat",
|
||||
"cases": [
|
||||
{
|
||||
"samples": [
|
||||
{
|
||||
"phase": "settled",
|
||||
"round": 0,
|
||||
"delivery_ms": {
|
||||
"direct": 2042.56345,
|
||||
"sent": 2239.283932,
|
||||
"received": 2442.564925
|
||||
},
|
||||
"sender_sm_handled_ms": 2441.985416,
|
||||
"publication_plus_delivery_ms": 2442.641984,
|
||||
"sync": {
|
||||
"total": 12,
|
||||
"by_store_kind_op": {
|
||||
"mam:directory:fsync": 2,
|
||||
"mam:file:fsync": 6,
|
||||
"sm:file:fdatasync": 4
|
||||
}
|
||||
},
|
||||
"pending_per_resource": [
|
||||
1,
|
||||
2,
|
||||
2,
|
||||
2
|
||||
]
|
||||
},
|
||||
{
|
||||
"phase": "avatar_pending",
|
||||
"round": 0,
|
||||
"delivery_ms": {
|
||||
"direct": 2676.283903,
|
||||
"sent": 2476.200907,
|
||||
"received": 3095.924978
|
||||
},
|
||||
"sender_sm_handled_ms": 3094.949323,
|
||||
"publication_plus_delivery_ms": 3096.115223,
|
||||
"sync": {
|
||||
"total": 15,
|
||||
"by_store_kind_op": {
|
||||
"mam:file:fsync": 2,
|
||||
"pep:directory:fsync": 2,
|
||||
"pep:file:fsync": 2,
|
||||
"sm:file:fdatasync": 9
|
||||
}
|
||||
},
|
||||
"pending_per_resource": [
|
||||
3,
|
||||
2,
|
||||
3,
|
||||
2
|
||||
]
|
||||
}
|
||||
],
|
||||
"mam_records_per_account": [
|
||||
102,
|
||||
102
|
||||
],
|
||||
"status": "pass",
|
||||
"delay_ms_per_sync": 200,
|
||||
"server_message_processing_slow_events": [
|
||||
{
|
||||
"duration_ms": 2440,
|
||||
"route": "local",
|
||||
"pre_routing_ms": 0,
|
||||
"sender_mam_ms": 813,
|
||||
"recipient_mam_ms": 816,
|
||||
"delivery_ms": 202,
|
||||
"carbons_ms": 404,
|
||||
"offline_push_ms": 0,
|
||||
"handled_ms": 202,
|
||||
"target_count": 1,
|
||||
"carbon_count": 2
|
||||
},
|
||||
{
|
||||
"duration_ms": 1259,
|
||||
"route": "local",
|
||||
"pre_routing_ms": 0,
|
||||
"sender_mam_ms": 204,
|
||||
"recipient_mam_ms": 208,
|
||||
"delivery_ms": 223,
|
||||
"carbons_ms": 410,
|
||||
"offline_push_ms": 0,
|
||||
"handled_ms": 211,
|
||||
"target_count": 1,
|
||||
"carbon_count": 2
|
||||
}
|
||||
],
|
||||
"server_message_processing_slow_ms": [
|
||||
2440,
|
||||
1259
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,74 @@
|
||||
{
|
||||
"binary_sha256": "130431025c4c7c2f72e28b77a31c83867f857ef8bb56d1f71b10e4b19d7caaf5",
|
||||
"scope": "loopback STARTTLS; four SM resources/two accounts; MAM + pending avatar PEP; fixture-only fsync/fdatasync including data directories; real sync retained",
|
||||
"cases": [
|
||||
{
|
||||
"samples": [
|
||||
{
|
||||
"phase": "settled",
|
||||
"round": 0,
|
||||
"delivery_ms": {
|
||||
"direct": 2836.63993,
|
||||
"sent": 3238.840305,
|
||||
"received": 3643.707041
|
||||
},
|
||||
"sender_sm_handled_ms": 3643.297905,
|
||||
"publication_plus_delivery_ms": 3643.772507,
|
||||
"sync": {
|
||||
"total": 24,
|
||||
"by_store_kind_op": {
|
||||
"mam:directory:fsync": 4,
|
||||
"mam:file:fsync": 4,
|
||||
"sm:directory:fsync": 8,
|
||||
"sm:file:fsync": 8
|
||||
}
|
||||
},
|
||||
"pending_per_resource": [
|
||||
1,
|
||||
2,
|
||||
2,
|
||||
2
|
||||
]
|
||||
},
|
||||
{
|
||||
"phase": "avatar_pending",
|
||||
"round": 0,
|
||||
"delivery_ms": {
|
||||
"direct": 5782.018326,
|
||||
"sent": 5384.85421,
|
||||
"received": 6596.874338
|
||||
},
|
||||
"sender_sm_handled_ms": 6595.924675,
|
||||
"publication_plus_delivery_ms": 6597.492339,
|
||||
"sync": {
|
||||
"total": 30,
|
||||
"by_store_kind_op": {
|
||||
"mam:directory:fsync": 4,
|
||||
"mam:file:fsync": 4,
|
||||
"pep:directory:fsync": 2,
|
||||
"pep:file:fsync": 2,
|
||||
"sm:directory:fsync": 9,
|
||||
"sm:file:fsync": 9
|
||||
}
|
||||
},
|
||||
"pending_per_resource": [
|
||||
3,
|
||||
2,
|
||||
3,
|
||||
2
|
||||
]
|
||||
}
|
||||
],
|
||||
"mam_records_per_account": [
|
||||
102,
|
||||
102
|
||||
],
|
||||
"status": "pass",
|
||||
"delay_ms_per_sync": 200,
|
||||
"server_message_processing_slow_ms": [
|
||||
3245,
|
||||
3235
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,255 @@
|
||||
{
|
||||
"binary_sha256": "8ceb8de3a2d02bc4399b8ae817dff7e3c3fe9bff83dc0c6669b7f19210a3847a",
|
||||
"scope": "loopback STARTTLS; four SM resources/two accounts; MAM + pending avatar metadata PEP; fixture-only fsync/fdatasync including data directories; real sync retained; operation windows include background SM heartbeat",
|
||||
"cases": [
|
||||
{
|
||||
"samples": [
|
||||
{
|
||||
"phase": "settled",
|
||||
"round": 0,
|
||||
"delivery_ms": {
|
||||
"direct": 1083.729337,
|
||||
"sent": 1183.535171,
|
||||
"received": 1301.736052
|
||||
},
|
||||
"sender_sm_handled_ms": 1300.213796,
|
||||
"publication_plus_delivery_ms": 1309.067665,
|
||||
"sync": {
|
||||
"total": 12,
|
||||
"by_store_kind_op": {
|
||||
"mam:directory:fsync": 2,
|
||||
"mam:file:fsync": 6,
|
||||
"sm:file:fdatasync": 4
|
||||
}
|
||||
},
|
||||
"pending_per_resource": [
|
||||
1,
|
||||
2,
|
||||
2,
|
||||
2
|
||||
]
|
||||
},
|
||||
{
|
||||
"phase": "settled",
|
||||
"round": 1,
|
||||
"delivery_ms": {
|
||||
"direct": 416.413847,
|
||||
"sent": 521.445561,
|
||||
"received": 622.692061
|
||||
},
|
||||
"sender_sm_handled_ms": 622.557259,
|
||||
"publication_plus_delivery_ms": 627.341892,
|
||||
"sync": {
|
||||
"total": 6,
|
||||
"by_store_kind_op": {
|
||||
"mam:file:fsync": 2,
|
||||
"sm:file:fdatasync": 4
|
||||
}
|
||||
},
|
||||
"pending_per_resource": [
|
||||
1,
|
||||
2,
|
||||
2,
|
||||
2
|
||||
]
|
||||
},
|
||||
{
|
||||
"phase": "settled",
|
||||
"round": 2,
|
||||
"delivery_ms": {
|
||||
"direct": 419.683064,
|
||||
"sent": 551.764241,
|
||||
"received": 644.751777
|
||||
},
|
||||
"sender_sm_handled_ms": 644.082891,
|
||||
"publication_plus_delivery_ms": 644.882621,
|
||||
"sync": {
|
||||
"total": 6,
|
||||
"by_store_kind_op": {
|
||||
"mam:file:fsync": 2,
|
||||
"sm:file:fdatasync": 4
|
||||
}
|
||||
},
|
||||
"pending_per_resource": [
|
||||
1,
|
||||
2,
|
||||
2,
|
||||
2
|
||||
]
|
||||
},
|
||||
{
|
||||
"phase": "avatar_pending",
|
||||
"round": 0,
|
||||
"delivery_ms": {
|
||||
"direct": 1376.989787,
|
||||
"sent": 1273.220821,
|
||||
"received": 1478.990457
|
||||
},
|
||||
"sender_sm_handled_ms": 1477.870242,
|
||||
"publication_plus_delivery_ms": 1479.132408,
|
||||
"sync": {
|
||||
"total": 14,
|
||||
"by_store_kind_op": {
|
||||
"mam:file:fsync": 2,
|
||||
"pep:directory:fsync": 2,
|
||||
"pep:file:fsync": 2,
|
||||
"sm:file:fdatasync": 8
|
||||
}
|
||||
},
|
||||
"pending_per_resource": [
|
||||
3,
|
||||
2,
|
||||
3,
|
||||
2
|
||||
]
|
||||
},
|
||||
{
|
||||
"phase": "avatar_pending",
|
||||
"round": 1,
|
||||
"delivery_ms": {
|
||||
"direct": 1523.59144,
|
||||
"sent": 1288.285577,
|
||||
"received": 1626.311997
|
||||
},
|
||||
"sender_sm_handled_ms": 1626.005979,
|
||||
"publication_plus_delivery_ms": 1626.574287,
|
||||
"sync": {
|
||||
"total": 14,
|
||||
"by_store_kind_op": {
|
||||
"mam:file:fsync": 2,
|
||||
"pep:directory:fsync": 2,
|
||||
"pep:file:fsync": 2,
|
||||
"sm:file:fdatasync": 8
|
||||
}
|
||||
},
|
||||
"pending_per_resource": [
|
||||
3,
|
||||
2,
|
||||
3,
|
||||
2
|
||||
]
|
||||
},
|
||||
{
|
||||
"phase": "avatar_pending",
|
||||
"round": 2,
|
||||
"delivery_ms": {
|
||||
"direct": 1378.231158,
|
||||
"sent": 1276.210346,
|
||||
"received": 1582.305858
|
||||
},
|
||||
"sender_sm_handled_ms": 1581.975429,
|
||||
"publication_plus_delivery_ms": 1582.463448,
|
||||
"sync": {
|
||||
"total": 15,
|
||||
"by_store_kind_op": {
|
||||
"mam:file:fsync": 2,
|
||||
"pep:directory:fsync": 2,
|
||||
"pep:file:fsync": 2,
|
||||
"sm:file:fdatasync": 9
|
||||
}
|
||||
},
|
||||
"pending_per_resource": [
|
||||
3,
|
||||
2,
|
||||
3,
|
||||
2
|
||||
]
|
||||
}
|
||||
],
|
||||
"mam_records_per_account": [
|
||||
106,
|
||||
106
|
||||
],
|
||||
"status": "pass",
|
||||
"delay_ms_per_sync": 100,
|
||||
"server_message_processing_slow_events": [
|
||||
{
|
||||
"duration_ms": 1298,
|
||||
"route": "local",
|
||||
"pre_routing_ms": 0,
|
||||
"sender_mam_ms": 421,
|
||||
"recipient_mam_ms": 416,
|
||||
"delivery_ms": 109,
|
||||
"carbons_ms": 232,
|
||||
"offline_push_ms": 0,
|
||||
"handled_ms": 117,
|
||||
"target_count": 1,
|
||||
"carbon_count": 2
|
||||
},
|
||||
{
|
||||
"duration_ms": 621,
|
||||
"route": "local",
|
||||
"pre_routing_ms": 0,
|
||||
"sender_mam_ms": 101,
|
||||
"recipient_mam_ms": 104,
|
||||
"delivery_ms": 104,
|
||||
"carbons_ms": 206,
|
||||
"offline_push_ms": 0,
|
||||
"handled_ms": 103,
|
||||
"target_count": 1,
|
||||
"carbon_count": 2
|
||||
},
|
||||
{
|
||||
"duration_ms": 638,
|
||||
"route": "local",
|
||||
"pre_routing_ms": 0,
|
||||
"sender_mam_ms": 101,
|
||||
"recipient_mam_ms": 102,
|
||||
"delivery_ms": 104,
|
||||
"carbons_ms": 219,
|
||||
"offline_push_ms": 1,
|
||||
"handled_ms": 108,
|
||||
"target_count": 1,
|
||||
"carbon_count": 2
|
||||
},
|
||||
{
|
||||
"duration_ms": 625,
|
||||
"route": "local",
|
||||
"pre_routing_ms": 1,
|
||||
"sender_mam_ms": 104,
|
||||
"recipient_mam_ms": 102,
|
||||
"delivery_ms": 106,
|
||||
"carbons_ms": 208,
|
||||
"offline_push_ms": 0,
|
||||
"handled_ms": 101,
|
||||
"target_count": 1,
|
||||
"carbon_count": 2
|
||||
},
|
||||
{
|
||||
"duration_ms": 756,
|
||||
"route": "local",
|
||||
"pre_routing_ms": 1,
|
||||
"sender_mam_ms": 105,
|
||||
"recipient_mam_ms": 103,
|
||||
"delivery_ms": 102,
|
||||
"carbons_ms": 340,
|
||||
"offline_push_ms": 0,
|
||||
"handled_ms": 102,
|
||||
"target_count": 1,
|
||||
"carbon_count": 2
|
||||
},
|
||||
{
|
||||
"duration_ms": 617,
|
||||
"route": "local",
|
||||
"pre_routing_ms": 1,
|
||||
"sender_mam_ms": 102,
|
||||
"recipient_mam_ms": 104,
|
||||
"delivery_ms": 101,
|
||||
"carbons_ms": 204,
|
||||
"offline_push_ms": 0,
|
||||
"handled_ms": 101,
|
||||
"target_count": 1,
|
||||
"carbon_count": 2
|
||||
}
|
||||
],
|
||||
"server_message_processing_slow_ms": [
|
||||
1298,
|
||||
621,
|
||||
638,
|
||||
625,
|
||||
756,
|
||||
617
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,350 @@
|
||||
{
|
||||
"binary_sha256": "130431025c4c7c2f72e28b77a31c83867f857ef8bb56d1f71b10e4b19d7caaf5",
|
||||
"scope": "loopback STARTTLS; four SM resources/two accounts; MAM + pending avatar PEP; fixture-only fsync/fdatasync including data directories; real sync retained",
|
||||
"cases": [
|
||||
{
|
||||
"samples": [
|
||||
{
|
||||
"phase": "settled",
|
||||
"round": 0,
|
||||
"delivery_ms": {
|
||||
"direct": 13.745165,
|
||||
"sent": 16.132819,
|
||||
"received": 18.746494
|
||||
},
|
||||
"sender_sm_handled_ms": 18.699594,
|
||||
"publication_plus_delivery_ms": 18.823329,
|
||||
"sync": {
|
||||
"total": 16,
|
||||
"by_store_kind_op": {
|
||||
"mam:directory:fsync": 4,
|
||||
"mam:file:fsync": 4,
|
||||
"sm:directory:fsync": 4,
|
||||
"sm:file:fsync": 4
|
||||
}
|
||||
},
|
||||
"pending_per_resource": [
|
||||
1,
|
||||
2,
|
||||
2,
|
||||
2
|
||||
]
|
||||
},
|
||||
{
|
||||
"phase": "settled",
|
||||
"round": 1,
|
||||
"delivery_ms": {
|
||||
"direct": 15.636949,
|
||||
"sent": 18.138746,
|
||||
"received": 20.807559
|
||||
},
|
||||
"sender_sm_handled_ms": 20.705943,
|
||||
"publication_plus_delivery_ms": 20.843968,
|
||||
"sync": {
|
||||
"total": 16,
|
||||
"by_store_kind_op": {
|
||||
"mam:directory:fsync": 4,
|
||||
"mam:file:fsync": 4,
|
||||
"sm:directory:fsync": 4,
|
||||
"sm:file:fsync": 4
|
||||
}
|
||||
},
|
||||
"pending_per_resource": [
|
||||
1,
|
||||
2,
|
||||
2,
|
||||
2
|
||||
]
|
||||
},
|
||||
{
|
||||
"phase": "settled",
|
||||
"round": 2,
|
||||
"delivery_ms": {
|
||||
"direct": 21.358818,
|
||||
"sent": 24.788203,
|
||||
"received": 26.125405
|
||||
},
|
||||
"sender_sm_handled_ms": 26.090729,
|
||||
"publication_plus_delivery_ms": 26.187115,
|
||||
"sync": {
|
||||
"total": 16,
|
||||
"by_store_kind_op": {
|
||||
"mam:directory:fsync": 4,
|
||||
"mam:file:fsync": 4,
|
||||
"sm:directory:fsync": 4,
|
||||
"sm:file:fsync": 4
|
||||
}
|
||||
},
|
||||
"pending_per_resource": [
|
||||
1,
|
||||
2,
|
||||
2,
|
||||
2
|
||||
]
|
||||
},
|
||||
{
|
||||
"phase": "avatar_pending",
|
||||
"round": 0,
|
||||
"delivery_ms": {
|
||||
"direct": 32.67965,
|
||||
"sent": 29.361139,
|
||||
"received": 35.870217
|
||||
},
|
||||
"sender_sm_handled_ms": 35.819252,
|
||||
"publication_plus_delivery_ms": 35.991777,
|
||||
"sync": {
|
||||
"total": 28,
|
||||
"by_store_kind_op": {
|
||||
"mam:directory:fsync": 4,
|
||||
"mam:file:fsync": 4,
|
||||
"pep:directory:fsync": 2,
|
||||
"pep:file:fsync": 2,
|
||||
"sm:directory:fsync": 8,
|
||||
"sm:file:fsync": 8
|
||||
}
|
||||
},
|
||||
"pending_per_resource": [
|
||||
3,
|
||||
2,
|
||||
3,
|
||||
2
|
||||
]
|
||||
},
|
||||
{
|
||||
"phase": "avatar_pending",
|
||||
"round": 1,
|
||||
"delivery_ms": {
|
||||
"direct": 30.47626,
|
||||
"sent": 28.128456,
|
||||
"received": 32.654735
|
||||
},
|
||||
"sender_sm_handled_ms": 32.540092,
|
||||
"publication_plus_delivery_ms": 32.738534,
|
||||
"sync": {
|
||||
"total": 28,
|
||||
"by_store_kind_op": {
|
||||
"mam:directory:fsync": 4,
|
||||
"mam:file:fsync": 4,
|
||||
"pep:directory:fsync": 2,
|
||||
"pep:file:fsync": 2,
|
||||
"sm:directory:fsync": 8,
|
||||
"sm:file:fsync": 8
|
||||
}
|
||||
},
|
||||
"pending_per_resource": [
|
||||
3,
|
||||
2,
|
||||
3,
|
||||
2
|
||||
]
|
||||
},
|
||||
{
|
||||
"phase": "avatar_pending",
|
||||
"round": 2,
|
||||
"delivery_ms": {
|
||||
"direct": 30.492529,
|
||||
"sent": 28.027238,
|
||||
"received": 33.697165
|
||||
},
|
||||
"sender_sm_handled_ms": 33.576139,
|
||||
"publication_plus_delivery_ms": 33.786933,
|
||||
"sync": {
|
||||
"total": 28,
|
||||
"by_store_kind_op": {
|
||||
"mam:directory:fsync": 4,
|
||||
"mam:file:fsync": 4,
|
||||
"pep:directory:fsync": 2,
|
||||
"pep:file:fsync": 2,
|
||||
"sm:directory:fsync": 8,
|
||||
"sm:file:fsync": 8
|
||||
}
|
||||
},
|
||||
"pending_per_resource": [
|
||||
3,
|
||||
2,
|
||||
3,
|
||||
2
|
||||
]
|
||||
}
|
||||
],
|
||||
"mam_records_per_account": [
|
||||
106,
|
||||
106
|
||||
],
|
||||
"status": "pass",
|
||||
"delay_ms_per_sync": 0
|
||||
},
|
||||
{
|
||||
"samples": [
|
||||
{
|
||||
"phase": "settled",
|
||||
"round": 0,
|
||||
"delivery_ms": {
|
||||
"direct": 1226.407308,
|
||||
"sent": 1431.404759,
|
||||
"received": 1837.301696
|
||||
},
|
||||
"sender_sm_handled_ms": 1633.73631,
|
||||
"publication_plus_delivery_ms": 1837.391548,
|
||||
"sync": {
|
||||
"total": 18,
|
||||
"by_store_kind_op": {
|
||||
"mam:directory:fsync": 4,
|
||||
"mam:file:fsync": 4,
|
||||
"sm:directory:fsync": 5,
|
||||
"sm:file:fsync": 5
|
||||
}
|
||||
},
|
||||
"pending_per_resource": [
|
||||
1,
|
||||
2,
|
||||
2,
|
||||
2
|
||||
]
|
||||
},
|
||||
{
|
||||
"phase": "settled",
|
||||
"round": 1,
|
||||
"delivery_ms": {
|
||||
"direct": 1234.414585,
|
||||
"sent": 1436.607311,
|
||||
"received": 1642.84668
|
||||
},
|
||||
"sender_sm_handled_ms": 1641.598264,
|
||||
"publication_plus_delivery_ms": 1642.913856,
|
||||
"sync": {
|
||||
"total": 16,
|
||||
"by_store_kind_op": {
|
||||
"mam:directory:fsync": 4,
|
||||
"mam:file:fsync": 4,
|
||||
"sm:directory:fsync": 4,
|
||||
"sm:file:fsync": 4
|
||||
}
|
||||
},
|
||||
"pending_per_resource": [
|
||||
1,
|
||||
2,
|
||||
2,
|
||||
2
|
||||
]
|
||||
},
|
||||
{
|
||||
"phase": "settled",
|
||||
"round": 2,
|
||||
"delivery_ms": {
|
||||
"direct": 1232.468024,
|
||||
"sent": 1441.824206,
|
||||
"received": 1872.009731
|
||||
},
|
||||
"sender_sm_handled_ms": 1652.92941,
|
||||
"publication_plus_delivery_ms": 1872.108448,
|
||||
"sync": {
|
||||
"total": 18,
|
||||
"by_store_kind_op": {
|
||||
"mam:directory:fsync": 4,
|
||||
"mam:file:fsync": 4,
|
||||
"sm:directory:fsync": 5,
|
||||
"sm:file:fsync": 5
|
||||
}
|
||||
},
|
||||
"pending_per_resource": [
|
||||
1,
|
||||
2,
|
||||
2,
|
||||
2
|
||||
]
|
||||
},
|
||||
{
|
||||
"phase": "avatar_pending",
|
||||
"round": 0,
|
||||
"delivery_ms": {
|
||||
"direct": 3209.381557,
|
||||
"sent": 3419.876568,
|
||||
"received": 3614.69826
|
||||
},
|
||||
"sender_sm_handled_ms": 3613.565949,
|
||||
"publication_plus_delivery_ms": 3615.484237,
|
||||
"sync": {
|
||||
"total": 28,
|
||||
"by_store_kind_op": {
|
||||
"mam:directory:fsync": 4,
|
||||
"mam:file:fsync": 4,
|
||||
"pep:directory:fsync": 2,
|
||||
"pep:file:fsync": 2,
|
||||
"sm:directory:fsync": 8,
|
||||
"sm:file:fsync": 8
|
||||
}
|
||||
},
|
||||
"pending_per_resource": [
|
||||
3,
|
||||
2,
|
||||
3,
|
||||
2
|
||||
]
|
||||
},
|
||||
{
|
||||
"phase": "avatar_pending",
|
||||
"round": 1,
|
||||
"delivery_ms": {
|
||||
"direct": 2665.684412,
|
||||
"sent": 2461.378505,
|
||||
"received": 2870.784696
|
||||
},
|
||||
"sender_sm_handled_ms": 2870.455915,
|
||||
"publication_plus_delivery_ms": 2871.000834,
|
||||
"sync": {
|
||||
"total": 28,
|
||||
"by_store_kind_op": {
|
||||
"mam:directory:fsync": 4,
|
||||
"mam:file:fsync": 4,
|
||||
"pep:directory:fsync": 2,
|
||||
"pep:file:fsync": 2,
|
||||
"sm:directory:fsync": 8,
|
||||
"sm:file:fsync": 8
|
||||
}
|
||||
},
|
||||
"pending_per_resource": [
|
||||
3,
|
||||
2,
|
||||
3,
|
||||
2
|
||||
]
|
||||
},
|
||||
{
|
||||
"phase": "avatar_pending",
|
||||
"round": 2,
|
||||
"delivery_ms": {
|
||||
"direct": 2683.292251,
|
||||
"sent": 2475.225974,
|
||||
"received": 3636.635597
|
||||
},
|
||||
"sender_sm_handled_ms": 3195.209226,
|
||||
"publication_plus_delivery_ms": 3640.372412,
|
||||
"sync": {
|
||||
"total": 30,
|
||||
"by_store_kind_op": {
|
||||
"mam:directory:fsync": 4,
|
||||
"mam:file:fsync": 4,
|
||||
"pep:directory:fsync": 2,
|
||||
"pep:file:fsync": 2,
|
||||
"sm:directory:fsync": 9,
|
||||
"sm:file:fsync": 9
|
||||
}
|
||||
},
|
||||
"pending_per_resource": [
|
||||
3,
|
||||
2,
|
||||
3,
|
||||
2
|
||||
]
|
||||
}
|
||||
],
|
||||
"mam_records_per_account": [
|
||||
106,
|
||||
106
|
||||
],
|
||||
"status": "pass",
|
||||
"delay_ms_per_sync": 100
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,111 @@
|
||||
Compiling rust-xmpp v0.8.3 (/workspace/scratch/362577cdf2e0/recovered/rust-xmpp)
|
||||
Finished `test` profile [unoptimized] target(s) in 29.67s
|
||||
Running unittests src/lib.rs (target/debug/deps/rust_xmpp-792bf902c86621ec)
|
||||
|
||||
running 50 tests
|
||||
test csi::checkpoint_tests::failed_queue_checkpoints_restore_append_and_drain_without_copying_queue ... ok
|
||||
test federation::tests::external_must_be_in_the_advertised_mechanisms ... ok
|
||||
test federation::tests::starttls_must_be_an_immediate_stream_feature ... ok
|
||||
test csi::checkpoint_tests::only_queue_changes_require_a_checkpoint ... ok
|
||||
test federation::tests::federation_domains_are_dns_names_and_case_insensitive ... ok
|
||||
test federation::tests::stream_restarts_compare_canonical_domains ... ok
|
||||
test http_config::tests::literal_pchars_and_percent_encoding_are_supported ... ok
|
||||
test http_config::tests::invalid_paths_types_urls_and_conflicts_are_errors ... ok
|
||||
test http_config::tests::external_prefix_and_standard_port ... ok
|
||||
test http_config::tests::routes_hosts_and_default_without_overriding_known_hosts ... ok
|
||||
test http_proxy::tests::config_validation_and_network_boundaries ... ok
|
||||
test http_proxy::tests::empty_native_lua_table_disables_proxy_trust ... ok
|
||||
test http_config::tests::mounts_inherit_per_app_and_expand_virtual_host ... ok
|
||||
test http_proxy::tests::forwarded_tls_termination_cannot_upgrade_a_plaintext_chain_hop ... ok
|
||||
test http_proxy::tests::malformed_forwarded_and_unknown_identifiers_are_ignored ... ok
|
||||
test http_proxy::tests::untrusted_socket_ignores_spoofed_headers ... ok
|
||||
test http_proxy::tests::legacy_chain_uses_rightmost_untrusted_and_tls_is_preserved ... ok
|
||||
test http_proxy::tests::forwarded_mode_stops_at_first_untrusted_hop ... ok
|
||||
test http_transport::tests::bosh_processing_capacity_is_bounded ... ok
|
||||
test http_transport::tests::configured_mount_host_origin_and_secure_dispatch ... ok
|
||||
test http_transport::tests::independent_framing ... ok
|
||||
test http_transport::tests::websocket_preserves_inherited_namespaces_and_spaced_declarations ... ok
|
||||
test logging::tests::console_aliases_share_one_sink_with_combined_levels ... ok
|
||||
test sasl::tests::reject_malformed_and_downgrade ... ok
|
||||
test archive::redo_rewrite_tests::crash_gap_after_redo_retirement_before_native_rename_keeps_old_snapshot ... ok
|
||||
test archive::redo_rewrite_tests::crash_gap_after_native_rename_has_no_old_redo_to_resurrect_deletions ... ok
|
||||
test sasl::tests::rfc5802_sha1_vector ... ok
|
||||
test server::authorization_tests::http_interfaces_follow_secure_override_wildcards_and_global_fallback ... ok
|
||||
test server::authorization_tests::pep_diagnostics_count_only_first_item_devices_without_logging_ids ... ok
|
||||
test server::authorization_tests::pep_diagnostics_exclude_payloads_and_identifiers ... ok
|
||||
test http_service::tests::upload_claim_refreshes_completion_state_after_an_earlier_snapshot ... ok
|
||||
test sessions::push_timeout_tests::first_important_push_shortens_extended_hibernation_once ... ok
|
||||
test sessions::tests::acknowledgement_wraps ... ok
|
||||
test sessions::tests::graceful_disconnect_is_not_resumable ... ok
|
||||
test sessions::tests::resume_replays_only_unacked_and_checks_owner ... ok
|
||||
test sessions::tests::traffic_before_sm_enable_is_not_counted ... ok
|
||||
test upload_migration::tests::local_spool_checks_size_and_hash_and_preserves_source ... ok
|
||||
test sasl::tests::sha256_independent_python_vector ... ok
|
||||
test wire::tests::arbitrary_prefix_and_inherited_namespaces ... ok
|
||||
test wire::tests::cdata_and_comments_frame_without_counting_brackets ... ok
|
||||
test wire::tests::declarations_rejected ... ok
|
||||
test wire::tests::fragmented_and_pipelined ... ok
|
||||
test wire::tests::invalid_namespace_and_declarations_rejected ... ok
|
||||
test upload_migration::tests::pending_and_corrupt_journals_block_startup ... ok
|
||||
test http_service::tests::upload_internal_mount_is_independent_of_advertised_proxy_prefix ... ok
|
||||
test http_service::tests::abandoned_slots_release_quota_without_pruning_existing_or_active_files ... ok
|
||||
test sasl::tests::plus_channel_binding_and_ssdp_are_authenticated ... ok
|
||||
test server::authorization_tests::upload_requires_registered_local_account_or_explicit_remote_access ... ok
|
||||
test sasl::tests::wrong_proof_nonce_binding_and_duplicates ... ok
|
||||
test admin::tests::real_admin_xml_protocol_over_duplex ... ok
|
||||
|
||||
test result: ok. 50 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.18s
|
||||
|
||||
Running tests/archive_edges.rs (target/debug/deps/archive_edges-da556b3567110d0b)
|
||||
|
||||
running 6 tests
|
||||
test missing_empty_whitespace_and_single_record ... ok
|
||||
test endpoint_read_recovers_partial_append_before_seeking ... ok
|
||||
test multiline_endpoints_match_native_scanner_with_blank_lines ... ok
|
||||
test oversized_prefix_and_malformed_endpoints_fail ... ok
|
||||
test metadata_skips_large_malformed_middle_without_creating_cache ... ok
|
||||
test tail_padding_respects_record_bound ... ok
|
||||
|
||||
test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.26s
|
||||
|
||||
Running tests/archive_redo.rs (target/debug/deps/archive_redo-d041d27370ba29bd)
|
||||
|
||||
running 13 tests
|
||||
test atomic_native_replace_without_handoff_cannot_discard_committed_frames ... ok
|
||||
test explicit_native_atomic_handoff_and_missing_baseline_fail_closed ... ok
|
||||
|
||||
running 1 test
|
||||
test durable_handoff_allows_native_in_place_append_and_next_rust_append ... ok
|
||||
test legacy_rollback_marker_is_migrated_before_redo_append ... ok
|
||||
test committed_redo_recovers_missing_and_partial_native_without_duplicates ... ok
|
||||
test redo_process_child ... ok
|
||||
test killed_writer_retains_accepted_records_after_native_data_loss ... ok
|
||||
test native_baseline_in_place_edits_are_rejected ... ok
|
||||
test complete_checksum_corruption_fails_without_modifying_native_or_journal ... ok
|
||||
test copied_archive_replays_committed_redo_despite_new_native_inode ... ok
|
||||
test incomplete_final_frame_and_uncommitted_native_suffix_are_discarded ... ok
|
||||
test byte_limit_rotates_before_frame_count_limit ... ok
|
||||
test bounded_rotation_recovers_suffix_and_rewrites_never_resurrect_deleted_records ... ok
|
||||
|
||||
test result: ok. 13 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.91s
|
||||
|
||||
Running tests/sm_journal.rs (target/debug/deps/sm_journal-79fad6209c2b84bb)
|
||||
|
||||
running 14 tests
|
||||
test offline_checkpoint_refuses_live_store_and_preserves_redo ... ok
|
||||
test complete_corrupt_frame_fails_closed_even_with_valid_export ... ok
|
||||
test journal_symlink_is_rejected_without_following_target ... ok
|
||||
test export_failure_after_commit_advances_memory_and_fences_later_writes ... ok
|
||||
test durable_tombstone_overrides_resurrected_stale_json ... ok
|
||||
test offline_checkpoint_durably_applies_expiration_and_tombstones ... ok
|
||||
test legacy_json_migrates_and_baseline_survives_partial_first_append ... ok
|
||||
test offline_checkpoint_repairs_exports_removes_redo_and_allows_legacy_restart ... ok
|
||||
test partial_tail_is_discarded_and_subsequent_commits_survive_restart ... ok
|
||||
test redo_repairs_missing_or_torn_export_before_json_validation ... ok
|
||||
test superseded_frames_still_require_valid_checksums_json_and_baseline_flags ... ok
|
||||
test incomplete_baseline_or_magic_never_falls_back_to_json ... ok
|
||||
test retired_journals_are_collected_in_bounded_batches ... ok
|
||||
test rotation_preserves_latest_snapshot_and_bounds_redo_size ... ok
|
||||
|
||||
test result: ok. 14 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 5.97s
|
||||
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,169 @@
|
||||
{
|
||||
"binary_sha256": "8ceb8de3a2d02bc4399b8ae817dff7e3c3fe9bff83dc0c6669b7f19210a3847a",
|
||||
"scope": "loopback STARTTLS; four SM resources/two accounts; MAM + pending avatar metadata PEP; fixture-only fsync/fdatasync including data directories; real sync retained; operation windows include background SM heartbeat",
|
||||
"cases": [
|
||||
{
|
||||
"samples": [
|
||||
{
|
||||
"phase": "settled",
|
||||
"round": 0,
|
||||
"delivery_ms": {
|
||||
"direct": 18.615095,
|
||||
"sent": 19.940848,
|
||||
"received": 26.953392
|
||||
},
|
||||
"sender_sm_handled_ms": 26.88493,
|
||||
"publication_plus_delivery_ms": 27.065981,
|
||||
"sync": {
|
||||
"total": 12,
|
||||
"by_store_kind_op": {
|
||||
"mam:directory:fsync": 2,
|
||||
"mam:file:fsync": 6,
|
||||
"sm:file:fdatasync": 4
|
||||
}
|
||||
},
|
||||
"pending_per_resource": [
|
||||
1,
|
||||
2,
|
||||
2,
|
||||
2
|
||||
]
|
||||
},
|
||||
{
|
||||
"phase": "settled",
|
||||
"round": 1,
|
||||
"delivery_ms": {
|
||||
"direct": 6.003256,
|
||||
"sent": 7.461811,
|
||||
"received": 10.135311
|
||||
},
|
||||
"sender_sm_handled_ms": 9.993133,
|
||||
"publication_plus_delivery_ms": 10.173245,
|
||||
"sync": {
|
||||
"total": 6,
|
||||
"by_store_kind_op": {
|
||||
"mam:file:fsync": 2,
|
||||
"sm:file:fdatasync": 4
|
||||
}
|
||||
},
|
||||
"pending_per_resource": [
|
||||
1,
|
||||
2,
|
||||
2,
|
||||
2
|
||||
]
|
||||
},
|
||||
{
|
||||
"phase": "settled",
|
||||
"round": 2,
|
||||
"delivery_ms": {
|
||||
"direct": 7.871244,
|
||||
"sent": 13.331419,
|
||||
"received": 15.325955
|
||||
},
|
||||
"sender_sm_handled_ms": 15.079213,
|
||||
"publication_plus_delivery_ms": 15.363148,
|
||||
"sync": {
|
||||
"total": 6,
|
||||
"by_store_kind_op": {
|
||||
"mam:file:fsync": 2,
|
||||
"sm:file:fdatasync": 4
|
||||
}
|
||||
},
|
||||
"pending_per_resource": [
|
||||
1,
|
||||
2,
|
||||
2,
|
||||
2
|
||||
]
|
||||
},
|
||||
{
|
||||
"phase": "avatar_pending",
|
||||
"round": 0,
|
||||
"delivery_ms": {
|
||||
"direct": 38.754553,
|
||||
"sent": 25.865646,
|
||||
"received": 40.65703
|
||||
},
|
||||
"sender_sm_handled_ms": 40.40549,
|
||||
"publication_plus_delivery_ms": 40.769686,
|
||||
"sync": {
|
||||
"total": 14,
|
||||
"by_store_kind_op": {
|
||||
"mam:file:fsync": 2,
|
||||
"pep:directory:fsync": 2,
|
||||
"pep:file:fsync": 2,
|
||||
"sm:file:fdatasync": 8
|
||||
}
|
||||
},
|
||||
"pending_per_resource": [
|
||||
3,
|
||||
2,
|
||||
3,
|
||||
2
|
||||
]
|
||||
},
|
||||
{
|
||||
"phase": "avatar_pending",
|
||||
"round": 1,
|
||||
"delivery_ms": {
|
||||
"direct": 20.942607,
|
||||
"sent": 20.204899,
|
||||
"received": 22.05411
|
||||
},
|
||||
"sender_sm_handled_ms": 21.856674,
|
||||
"publication_plus_delivery_ms": 23.679041,
|
||||
"sync": {
|
||||
"total": 14,
|
||||
"by_store_kind_op": {
|
||||
"mam:file:fsync": 2,
|
||||
"pep:directory:fsync": 2,
|
||||
"pep:file:fsync": 2,
|
||||
"sm:file:fdatasync": 8
|
||||
}
|
||||
},
|
||||
"pending_per_resource": [
|
||||
3,
|
||||
2,
|
||||
3,
|
||||
2
|
||||
]
|
||||
},
|
||||
{
|
||||
"phase": "avatar_pending",
|
||||
"round": 2,
|
||||
"delivery_ms": {
|
||||
"direct": 17.980156,
|
||||
"sent": 18.759152,
|
||||
"received": 21.369599
|
||||
},
|
||||
"sender_sm_handled_ms": 21.332607,
|
||||
"publication_plus_delivery_ms": 21.518131,
|
||||
"sync": {
|
||||
"total": 14,
|
||||
"by_store_kind_op": {
|
||||
"mam:file:fsync": 2,
|
||||
"pep:directory:fsync": 2,
|
||||
"pep:file:fsync": 2,
|
||||
"sm:file:fdatasync": 8
|
||||
}
|
||||
},
|
||||
"pending_per_resource": [
|
||||
3,
|
||||
2,
|
||||
3,
|
||||
2
|
||||
]
|
||||
}
|
||||
],
|
||||
"mam_records_per_account": [
|
||||
106,
|
||||
106
|
||||
],
|
||||
"status": "pass",
|
||||
"delay_ms_per_sync": 0,
|
||||
"server_message_processing_slow_events": [],
|
||||
"server_message_processing_slow_ms": []
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,180 @@
|
||||
{
|
||||
"binary_sha256": "130431025c4c7c2f72e28b77a31c83867f857ef8bb56d1f71b10e4b19d7caaf5",
|
||||
"scope": "loopback STARTTLS; four SM resources/two accounts; MAM + pending avatar metadata PEP; fixture-only fsync/fdatasync including data directories; real sync retained; operation windows include background SM heartbeat",
|
||||
"cases": [
|
||||
{
|
||||
"samples": [
|
||||
{
|
||||
"phase": "settled",
|
||||
"round": 0,
|
||||
"delivery_ms": {
|
||||
"direct": 26.330895,
|
||||
"sent": 29.575091,
|
||||
"received": 33.552895
|
||||
},
|
||||
"sender_sm_handled_ms": 33.407559,
|
||||
"publication_plus_delivery_ms": 33.592558,
|
||||
"sync": {
|
||||
"total": 16,
|
||||
"by_store_kind_op": {
|
||||
"mam:directory:fsync": 4,
|
||||
"mam:file:fsync": 4,
|
||||
"sm:directory:fsync": 4,
|
||||
"sm:file:fsync": 4
|
||||
}
|
||||
},
|
||||
"pending_per_resource": [
|
||||
1,
|
||||
2,
|
||||
2,
|
||||
2
|
||||
]
|
||||
},
|
||||
{
|
||||
"phase": "settled",
|
||||
"round": 1,
|
||||
"delivery_ms": {
|
||||
"direct": 25.112248,
|
||||
"sent": 31.688103,
|
||||
"received": 39.725842
|
||||
},
|
||||
"sender_sm_handled_ms": 39.683719,
|
||||
"publication_plus_delivery_ms": 39.867826,
|
||||
"sync": {
|
||||
"total": 16,
|
||||
"by_store_kind_op": {
|
||||
"mam:directory:fsync": 4,
|
||||
"mam:file:fsync": 4,
|
||||
"sm:directory:fsync": 4,
|
||||
"sm:file:fsync": 4
|
||||
}
|
||||
},
|
||||
"pending_per_resource": [
|
||||
1,
|
||||
2,
|
||||
2,
|
||||
2
|
||||
]
|
||||
},
|
||||
{
|
||||
"phase": "settled",
|
||||
"round": 2,
|
||||
"delivery_ms": {
|
||||
"direct": 11.894093,
|
||||
"sent": 11.977544,
|
||||
"received": 13.638711
|
||||
},
|
||||
"sender_sm_handled_ms": 13.371964,
|
||||
"publication_plus_delivery_ms": 13.720665,
|
||||
"sync": {
|
||||
"total": 16,
|
||||
"by_store_kind_op": {
|
||||
"mam:directory:fsync": 4,
|
||||
"mam:file:fsync": 4,
|
||||
"sm:directory:fsync": 4,
|
||||
"sm:file:fsync": 4
|
||||
}
|
||||
},
|
||||
"pending_per_resource": [
|
||||
1,
|
||||
2,
|
||||
2,
|
||||
2
|
||||
]
|
||||
},
|
||||
{
|
||||
"phase": "avatar_pending",
|
||||
"round": 0,
|
||||
"delivery_ms": {
|
||||
"direct": 42.768972,
|
||||
"sent": 32.889933,
|
||||
"received": 48.207819
|
||||
},
|
||||
"sender_sm_handled_ms": 47.576405,
|
||||
"publication_plus_delivery_ms": 51.552818,
|
||||
"sync": {
|
||||
"total": 28,
|
||||
"by_store_kind_op": {
|
||||
"mam:directory:fsync": 4,
|
||||
"mam:file:fsync": 4,
|
||||
"pep:directory:fsync": 2,
|
||||
"pep:file:fsync": 2,
|
||||
"sm:directory:fsync": 8,
|
||||
"sm:file:fsync": 8
|
||||
}
|
||||
},
|
||||
"pending_per_resource": [
|
||||
3,
|
||||
2,
|
||||
3,
|
||||
2
|
||||
]
|
||||
},
|
||||
{
|
||||
"phase": "avatar_pending",
|
||||
"round": 1,
|
||||
"delivery_ms": {
|
||||
"direct": 28.176101,
|
||||
"sent": 26.386914,
|
||||
"received": 29.822606
|
||||
},
|
||||
"sender_sm_handled_ms": 29.714999,
|
||||
"publication_plus_delivery_ms": 30.011235,
|
||||
"sync": {
|
||||
"total": 28,
|
||||
"by_store_kind_op": {
|
||||
"mam:directory:fsync": 4,
|
||||
"mam:file:fsync": 4,
|
||||
"pep:directory:fsync": 2,
|
||||
"pep:file:fsync": 2,
|
||||
"sm:directory:fsync": 8,
|
||||
"sm:file:fsync": 8
|
||||
}
|
||||
},
|
||||
"pending_per_resource": [
|
||||
3,
|
||||
2,
|
||||
3,
|
||||
2
|
||||
]
|
||||
},
|
||||
{
|
||||
"phase": "avatar_pending",
|
||||
"round": 2,
|
||||
"delivery_ms": {
|
||||
"direct": 29.714366,
|
||||
"sent": 26.01394,
|
||||
"received": 32.367508
|
||||
},
|
||||
"sender_sm_handled_ms": 32.252144,
|
||||
"publication_plus_delivery_ms": 33.076522,
|
||||
"sync": {
|
||||
"total": 28,
|
||||
"by_store_kind_op": {
|
||||
"mam:directory:fsync": 4,
|
||||
"mam:file:fsync": 4,
|
||||
"pep:directory:fsync": 2,
|
||||
"pep:file:fsync": 2,
|
||||
"sm:directory:fsync": 8,
|
||||
"sm:file:fsync": 8
|
||||
}
|
||||
},
|
||||
"pending_per_resource": [
|
||||
3,
|
||||
2,
|
||||
3,
|
||||
2
|
||||
]
|
||||
}
|
||||
],
|
||||
"mam_records_per_account": [
|
||||
106,
|
||||
106
|
||||
],
|
||||
"status": "pass",
|
||||
"delay_ms_per_sync": 0,
|
||||
"server_message_processing_slow_events": [],
|
||||
"server_message_processing_slow_ms": []
|
||||
}
|
||||
]
|
||||
}
|
||||
File diff suppressed because it is too large.
Load diff
@@ -0,0 +1,209 @@
|
||||
{
|
||||
"version": "0.8.3",
|
||||
"date_utc": "2026-10-21",
|
||||
"release_binary_sha256": "8ceb8de3a2d02bc4399b8ae817dff7e3c3fe9bff83dc0c6669b7f19210a3847a",
|
||||
"ordinary_tests": {
|
||||
"passed": 334,
|
||||
"failed": 0,
|
||||
"ignored": 11
|
||||
},
|
||||
"fmt": "passed",
|
||||
"clippy": "passed",
|
||||
"release": "passed",
|
||||
"full_verify": "passed in one final frozen-source run",
|
||||
"native_module_tests": 8,
|
||||
"native_federation_tests": 1,
|
||||
"extra_large_archive_tests": 1,
|
||||
"ignored_not_run": [
|
||||
"AF_UNIX admin_socket"
|
||||
],
|
||||
"load": {
|
||||
"general_tls_clients": 500,
|
||||
"general_accounts": 100,
|
||||
"muc_sm": {
|
||||
"clients": 500,
|
||||
"accounts": 100,
|
||||
"rooms": 10,
|
||||
"message_rounds": 3,
|
||||
"concurrency": 20,
|
||||
"binary": "/tmp/xmpp-durable-muc-load-ssixqqk9/rust-xmpp",
|
||||
"binary_sha256": "8ceb8de3a2d02bc4399b8ae817dff7e3c3fe9bff83dc0c6669b7f19210a3847a",
|
||||
"before": {
|
||||
"rss_kib": 9516,
|
||||
"cpu_seconds": 0.01,
|
||||
"fd_count": 12
|
||||
},
|
||||
"connect_seconds": 12.048,
|
||||
"connect_latency": {
|
||||
"samples": 500,
|
||||
"median_ms": 398.044,
|
||||
"p95_ms": 1086.236,
|
||||
"max_ms": 1403.7
|
||||
},
|
||||
"phase": "completed",
|
||||
"status": "pass",
|
||||
"elapsed_seconds": 128.38,
|
||||
"join_seconds": 94.493,
|
||||
"joined": {
|
||||
"rss_kib": 41828,
|
||||
"cpu_seconds": 55.61,
|
||||
"fd_count": 513
|
||||
},
|
||||
"join_presence_deliveries": 25000,
|
||||
"slots_joined": {
|
||||
"count": 500,
|
||||
"bytes": 309790
|
||||
},
|
||||
"message_seconds": 9.585,
|
||||
"room_messages_sent": 30,
|
||||
"room_messages_delivered": 1500,
|
||||
"before_kill": {
|
||||
"rss_kib": 41740,
|
||||
"cpu_seconds": 62.6,
|
||||
"fd_count": 512
|
||||
},
|
||||
"slots_before_kill": {
|
||||
"count": 500,
|
||||
"bytes": 514030
|
||||
},
|
||||
"restart_seconds": 7.557,
|
||||
"resumed_resources": 10,
|
||||
"replayed_pending_room_messages": 10,
|
||||
"membership_groupchat_deliveries_after_resume": 10,
|
||||
"after_resume": {
|
||||
"rss_kib": 20896,
|
||||
"cpu_seconds": 7.09,
|
||||
"fd_count": 22
|
||||
},
|
||||
"slots_after_resume": {
|
||||
"count": 500,
|
||||
"bytes": 671070
|
||||
},
|
||||
"total_seconds": 128.38,
|
||||
"peak_rss_kib": 41852
|
||||
}
|
||||
},
|
||||
"personal_delivery_comparisons": [
|
||||
{
|
||||
"delay_ms_per_sync": 0,
|
||||
"phase": "settled",
|
||||
"samples_per_binary": 3,
|
||||
"body_before_ms": 25.112248,
|
||||
"body_after_ms": 7.871244,
|
||||
"sender_ack_before_ms": 33.407559,
|
||||
"sender_ack_after_ms": 15.079213
|
||||
},
|
||||
{
|
||||
"delay_ms_per_sync": 0,
|
||||
"phase": "avatar_pending",
|
||||
"samples_per_binary": 3,
|
||||
"body_before_ms": 29.714366,
|
||||
"body_after_ms": 20.942607,
|
||||
"sender_ack_before_ms": 32.252144,
|
||||
"sender_ack_after_ms": 21.856674
|
||||
},
|
||||
{
|
||||
"delay_ms_per_sync": 100,
|
||||
"phase": "settled",
|
||||
"samples_per_binary": 3,
|
||||
"body_before_ms": 1232.468024,
|
||||
"body_after_ms": 419.683064,
|
||||
"sender_ack_before_ms": 1641.598264,
|
||||
"sender_ack_after_ms": 644.082891
|
||||
},
|
||||
{
|
||||
"delay_ms_per_sync": 100,
|
||||
"phase": "avatar_pending",
|
||||
"samples_per_binary": 3,
|
||||
"body_before_ms": 2683.292251,
|
||||
"body_after_ms": 1378.231158,
|
||||
"sender_ack_before_ms": 3195.209226,
|
||||
"sender_ack_after_ms": 1581.975429
|
||||
},
|
||||
{
|
||||
"delay_ms_per_sync": 200,
|
||||
"phase": "settled",
|
||||
"samples_per_binary": 1,
|
||||
"body_before_ms": 2836.63993,
|
||||
"body_after_ms": 2042.56345,
|
||||
"sender_ack_before_ms": 3643.297905,
|
||||
"sender_ack_after_ms": 2441.985416
|
||||
},
|
||||
{
|
||||
"delay_ms_per_sync": 200,
|
||||
"phase": "avatar_pending",
|
||||
"samples_per_binary": 1,
|
||||
"body_before_ms": 5782.018326,
|
||||
"body_after_ms": 2676.283903,
|
||||
"sender_ack_before_ms": 6595.924675,
|
||||
"sender_ack_after_ms": 3094.949323
|
||||
}
|
||||
],
|
||||
"durability_crash": {
|
||||
"status": "pass",
|
||||
"delay_ms_per_sync": 100,
|
||||
"acknowledged_messages": 3,
|
||||
"mam_records_per_account": [
|
||||
3,
|
||||
3
|
||||
],
|
||||
"resumed_resources": 4,
|
||||
"replayed_personal_deliveries": 9,
|
||||
"offline_checkpoint": {
|
||||
"exported_sessions": 4,
|
||||
"native_archives": 2,
|
||||
"journals_removed": true,
|
||||
"second_invocation_idempotent": true
|
||||
},
|
||||
"pending_before_reauthentication": [
|
||||
2,
|
||||
3,
|
||||
4,
|
||||
3
|
||||
],
|
||||
"sync_between_publication_and_sigkill": {
|
||||
"total": 32,
|
||||
"by_store_kind_op": {
|
||||
"mam:directory:fsync": 2,
|
||||
"mam:file:fsync": 10,
|
||||
"pep:directory:fsync": 2,
|
||||
"pep:file:fsync": 2,
|
||||
"sm:file:fdatasync": 16
|
||||
}
|
||||
},
|
||||
"binary_sha256": "8ceb8de3a2d02bc4399b8ae817dff7e3c3fe9bff83dc0c6669b7f19210a3847a",
|
||||
"scope": "loopback STARTTLS; real delayed fsync/fdatasync; sender SM acknowledgement then SIGKILL; authoritative queues and MAM checked before reauthentication; at-least-once replay"
|
||||
},
|
||||
"recovery_sync_faults": {
|
||||
"status": "pass",
|
||||
"cases": [
|
||||
{
|
||||
"suffix": ".wal",
|
||||
"status": "pass",
|
||||
"checkpoint_rejected_sync_failure": true,
|
||||
"journals_preserved_on_failure": 4,
|
||||
"serve_rejected_before_listeners": true,
|
||||
"clean_retry_preserved_sessions": 4,
|
||||
"clean_retry_preserved_archives": 2
|
||||
},
|
||||
{
|
||||
"suffix": ".rredo",
|
||||
"status": "pass",
|
||||
"checkpoint_rejected_sync_failure": true,
|
||||
"journals_preserved_on_failure": 2,
|
||||
"serve_rejected_before_listeners": false,
|
||||
"clean_retry_preserved_sessions": 4,
|
||||
"clean_retry_preserved_archives": 2
|
||||
}
|
||||
],
|
||||
"binary_sha256": "8ceb8de3a2d02bc4399b8ae817dff7e3c3fe9bff83dc0c6669b7f19210a3847a"
|
||||
},
|
||||
"limitations": [
|
||||
"Loopback fixture, not production client or disk qualification",
|
||||
"0ms pair ran sequentially without full verify; 100/200ms final runs had background final verify",
|
||||
"One200ms sample; startup/migration/rotation and background heartbeat affect sync counts",
|
||||
"SM global IO mutex and full snapshots remain; no atomic transaction spanning MAM/SM/network",
|
||||
"Offline storage-checkpoint required before legacy/native writer handoff",
|
||||
"10GiB transfers, real MinIO/AWS, Docker/Gitea/GitHub run were not repeated in this release"
|
||||
]
|
||||
}
|
||||
+182
-1667
File diff suppressed because it is too large.
Load diff
@@ -30,6 +30,9 @@ python3 tests/preauth_network.py target/release/rust-xmpp
|
||||
python3 tests/muc_restart_network.py target/release/rust-xmpp
|
||||
python3 tests/muc_crash_network.py target/release/rust-xmpp
|
||||
python3 tests/sm_restart_network.py target/release/rust-xmpp
|
||||
python3 tests/slow_storage_latency_network.py --binary target/release/rust-xmpp --delay-ms 0 25 --rounds 1 --output /tmp/slow-storage-latency.json
|
||||
python3 tests/slow_storage_crash_network.py --binary target/release/rust-xmpp --delay-ms 25 --messages 3 --output /tmp/slow-storage-crash.json
|
||||
python3 tests/journal_recovery_sync_network.py --binary target/release/rust-xmpp --output /tmp/journal-recovery-sync.json
|
||||
python3 tests/push_latency_network.py --binary target/release/rust-xmpp --recipient hibernating --output /tmp/push-latency.json
|
||||
python3 tests/push_latency_network.py --binary target/release/rust-xmpp --recipient offline --output /tmp/push-offline-latency.json
|
||||
python3 tests/personal_message_latency_network.py --binary target/release/rust-xmpp --rounds 1 --history-records 1000 --roster-sizes 0 --backlog 30 --csi-dwell 0.1 --output /tmp/personal-message-latency.json
|
||||
|
||||
+380
-20
@@ -174,7 +174,103 @@ pub fn scan_store<F: FnMut(Entry) -> Result<()>>(
|
||||
let _guard = lock.lock().map_err(|_| anyhow!("archive lock poisoned"))?;
|
||||
scan_unlocked(root, host, user, store, visit)
|
||||
}
|
||||
fn recover_append(p: &Path) -> Result<()> {
|
||||
// The native file remains directly readable by Prosody. While this sidecar exists,
|
||||
// its complete frames define the committed suffix: do not edit a live native file
|
||||
// in place or replace it until checkpoint_all has durably retired the journal.
|
||||
const REDO_MAGIC: &[u8; 8] = b"XARRED01";
|
||||
const REDO_HEADER: usize = 96;
|
||||
const REDO_LIMIT: u64 = 1024 * 1024;
|
||||
const REDO_FRAMES: usize = 128;
|
||||
#[derive(Clone)]
|
||||
struct RedoState {
|
||||
length: u64,
|
||||
bytes: u64,
|
||||
frames: usize,
|
||||
}
|
||||
#[derive(Clone, PartialEq, Eq)]
|
||||
struct FileStamp {
|
||||
len: u64,
|
||||
modified: Option<std::time::SystemTime>,
|
||||
identity: (u64, u64),
|
||||
#[cfg(unix)]
|
||||
changed: (i64, i64),
|
||||
}
|
||||
fn identity(meta: &std::fs::Metadata) -> (u64, u64) {
|
||||
#[cfg(unix)]
|
||||
{
|
||||
use std::os::unix::fs::MetadataExt;
|
||||
(meta.dev(), meta.ino())
|
||||
}
|
||||
#[cfg(not(unix))]
|
||||
{
|
||||
let _ = meta;
|
||||
(0, 0)
|
||||
}
|
||||
}
|
||||
fn stamp(p: &Path) -> Result<FileStamp> {
|
||||
let m = std::fs::metadata(p)?;
|
||||
Ok(FileStamp {
|
||||
len: m.len(),
|
||||
modified: m.modified().ok(),
|
||||
identity: identity(&m),
|
||||
#[cfg(unix)]
|
||||
changed: {
|
||||
use std::os::unix::fs::MetadataExt;
|
||||
(m.ctime(), m.ctime_nsec())
|
||||
},
|
||||
})
|
||||
}
|
||||
type RedoCache = std::collections::HashMap<std::path::PathBuf, (FileStamp, FileStamp, RedoState)>;
|
||||
static REDO_CACHE: OnceLock<Mutex<RedoCache>> = OnceLock::new();
|
||||
fn remember_redo(p: &Path, state: &RedoState) -> Result<()> {
|
||||
let native = stamp(p)?;
|
||||
let journal = stamp(&p.with_extension("rredo"))?;
|
||||
let mut cache = REDO_CACHE
|
||||
.get_or_init(|| Mutex::new(RedoCache::new()))
|
||||
.lock()
|
||||
.map_err(|_| anyhow!("archive redo cache poisoned"))?;
|
||||
if cache.len() >= 1024 {
|
||||
cache.clear();
|
||||
}
|
||||
cache.insert(p.to_path_buf(), (native, journal, state.clone()));
|
||||
Ok(())
|
||||
}
|
||||
fn digest(data: &[u8]) -> [u8; 32] {
|
||||
use sha2::{Digest, Sha256};
|
||||
Sha256::digest(data).into()
|
||||
}
|
||||
fn baseline_anchor(file: &mut std::fs::File, length: u64) -> Result<[u8; 32]> {
|
||||
let count = length.min(4096);
|
||||
file.seek(SeekFrom::Start(length - count))?;
|
||||
let mut bytes = vec![0; count as usize];
|
||||
file.read_exact(&mut bytes)?;
|
||||
Ok(digest(&bytes))
|
||||
}
|
||||
/// Native fsync precedes journal replacement. Either old or new header can
|
||||
/// survive a crash, and both refer to an already durable native baseline.
|
||||
fn checkpoint_redo(p: &Path) -> Result<RedoState> {
|
||||
let mut file = std::fs::OpenOptions::new().read(true).write(true).open(p)?;
|
||||
file.sync_all()?;
|
||||
let meta = file.metadata()?;
|
||||
let length = meta.len();
|
||||
let (dev, ino) = identity(&meta);
|
||||
let mut header = Vec::with_capacity(REDO_HEADER);
|
||||
header.extend_from_slice(REDO_MAGIC);
|
||||
for n in [length, dev, ino] {
|
||||
header.extend_from_slice(&n.to_le_bytes());
|
||||
}
|
||||
header.extend_from_slice(&baseline_anchor(&mut file, length)?);
|
||||
header.extend_from_slice(&digest(&header));
|
||||
storage::atomic_write(&p.with_extension("rredo"), &header)?;
|
||||
let state = RedoState {
|
||||
length,
|
||||
bytes: REDO_HEADER as u64,
|
||||
frames: 0,
|
||||
};
|
||||
remember_redo(p, &state)?;
|
||||
Ok(state)
|
||||
}
|
||||
fn recover_legacy_append(p: &Path) -> Result<()> {
|
||||
let journal = p.with_extension("rappend");
|
||||
match std::fs::read(&journal) {
|
||||
Ok(data) => {
|
||||
@@ -195,6 +291,194 @@ fn recover_append(p: &Path) -> Result<()> {
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
fn recover_redo(p: &Path) -> Result<Option<RedoState>> {
|
||||
recover_legacy_append(p)?;
|
||||
let journal = p.with_extension("rredo");
|
||||
let jm = match stamp(&journal) {
|
||||
Ok(m) => m,
|
||||
Err(e)
|
||||
if e.downcast_ref::<std::io::Error>()
|
||||
.is_some_and(|e| e.kind() == std::io::ErrorKind::NotFound) =>
|
||||
{
|
||||
return Ok(None);
|
||||
}
|
||||
Err(e) => return Err(e),
|
||||
};
|
||||
let nm = stamp(p).ok();
|
||||
if let Some(native) = &nm {
|
||||
let cache = REDO_CACHE
|
||||
.get_or_init(|| Mutex::new(RedoCache::new()))
|
||||
.lock()
|
||||
.map_err(|_| anyhow!("archive redo cache poisoned"))?;
|
||||
if let Some((n, j, state)) = cache.get(p)
|
||||
&& n == native
|
||||
&& j == &jm
|
||||
{
|
||||
return Ok(Some(state.clone()));
|
||||
}
|
||||
}
|
||||
// One oversized record is allowed, but the journal never grows without bound.
|
||||
anyhow::ensure!(
|
||||
jm.len <= REDO_LIMIT + MAX_RECORD_BYTES as u64 + REDO_HEADER as u64 + 72,
|
||||
"archive redo journal exceeds bound"
|
||||
);
|
||||
let data = std::fs::read(&journal)?;
|
||||
anyhow::ensure!(
|
||||
data.len() >= REDO_HEADER && &data[..8] == REDO_MAGIC,
|
||||
"invalid archive redo header"
|
||||
);
|
||||
anyhow::ensure!(
|
||||
digest(&data[..64]) == data[64..96],
|
||||
"archive redo header checksum mismatch"
|
||||
);
|
||||
let number = |offset: usize| u64::from_le_bytes(data[offset..offset + 8].try_into().unwrap());
|
||||
let base = number(8);
|
||||
// Inodes change when an archive is copied/restored. The journal remains
|
||||
// authoritative across that change; identity is only a cache stamp.
|
||||
anyhow::ensure!(nm.is_some() || base == 0, "missing archive redo baseline");
|
||||
let mut opts = std::fs::OpenOptions::new();
|
||||
opts.read(true).write(true).create(true).truncate(false);
|
||||
#[cfg(unix)]
|
||||
{
|
||||
use std::os::unix::fs::OpenOptionsExt;
|
||||
opts.mode(0o600);
|
||||
}
|
||||
let mut file = opts.open(p)?;
|
||||
anyhow::ensure!(
|
||||
file.metadata()?.len() >= base,
|
||||
"archive shorter than redo baseline"
|
||||
);
|
||||
anyhow::ensure!(
|
||||
baseline_anchor(&mut file, base)? == data[32..64],
|
||||
"archive redo baseline changed in place"
|
||||
);
|
||||
let mut pos = REDO_HEADER;
|
||||
let mut frames = Vec::new();
|
||||
let mut length = base;
|
||||
while pos < data.len() {
|
||||
if data.len() - pos < 72 {
|
||||
break;
|
||||
}
|
||||
anyhow::ensure!(
|
||||
digest(&data[pos..pos + 8]) == data[pos + 8..pos + 40],
|
||||
"archive redo frame length checksum mismatch"
|
||||
);
|
||||
let size = number(pos);
|
||||
anyhow::ensure!(
|
||||
size > 0 && size <= MAX_RECORD_BYTES as u64,
|
||||
"invalid archive redo frame length"
|
||||
);
|
||||
let end = pos + 72 + size as usize;
|
||||
if end > data.len() {
|
||||
break;
|
||||
}
|
||||
let payload = &data[pos + 72..end];
|
||||
anyhow::ensure!(
|
||||
digest(payload) == data[pos + 40..pos + 72],
|
||||
"archive redo frame checksum mismatch"
|
||||
);
|
||||
frames.push((length, payload));
|
||||
anyhow::ensure!(
|
||||
frames.len() <= REDO_FRAMES,
|
||||
"archive redo frame count exceeds bound"
|
||||
);
|
||||
length = length
|
||||
.checked_add(size)
|
||||
.ok_or_else(|| anyhow!("archive redo offset overflow"))?;
|
||||
pos = end;
|
||||
}
|
||||
// Native indexes are rebuildable. A crash may have resurrected an unlink
|
||||
// from the append path; invalidate it again before exposing recovered data.
|
||||
match std::fs::remove_file(p.with_extension("lidx")) {
|
||||
Ok(()) => std::fs::File::open(p.parent().unwrap())?.sync_all()?,
|
||||
Err(e) if e.kind() == std::io::ErrorKind::NotFound => (),
|
||||
Err(e) => return Err(e.into()),
|
||||
}
|
||||
// A complete frame can survive a process crash in page cache even if its
|
||||
// writer never completed sync. Durably adopt all validated frames before
|
||||
// exposing them or allowing later recovery to truncate the native file.
|
||||
let redo = std::fs::OpenOptions::new().write(true).open(&journal)?;
|
||||
if pos < data.len() {
|
||||
redo.set_len(pos as u64)?;
|
||||
}
|
||||
redo.sync_all()?;
|
||||
// Validate the complete journal before changing native data. Replaying is
|
||||
// idempotent, including after a second crash midway through recovery.
|
||||
let mut changed = file.metadata()?.len() != length;
|
||||
for (offset, payload) in &frames {
|
||||
file.seek(SeekFrom::Start(*offset))?;
|
||||
let mut native = vec![0; payload.len()];
|
||||
let equal = file.read_exact(&mut native).is_ok() && native.as_slice() == *payload;
|
||||
if !equal {
|
||||
file.seek(SeekFrom::Start(*offset))?;
|
||||
file.write_all(payload)?;
|
||||
changed = true;
|
||||
}
|
||||
}
|
||||
if changed {
|
||||
file.set_len(length)?;
|
||||
file.sync_all()?;
|
||||
}
|
||||
let state = RedoState {
|
||||
length,
|
||||
bytes: pos as u64,
|
||||
frames: frames.len(),
|
||||
};
|
||||
if nm.is_none() {
|
||||
return checkpoint_redo(p).map(Some);
|
||||
}
|
||||
remember_redo(p, &state)?;
|
||||
Ok(Some(state))
|
||||
}
|
||||
fn recover_append(p: &Path) -> Result<()> {
|
||||
recover_redo(p)?;
|
||||
Ok(())
|
||||
}
|
||||
/// Durable handoff to a native writer. Call after stopping archive users and
|
||||
/// before letting Prosody modify `.list` files in place. Abrupt shutdown leaves
|
||||
/// redo recovery active; unjournaled native suffixes are then uncommitted.
|
||||
pub fn checkpoint_all(root: &Path) -> Result<usize> {
|
||||
with_exclusive_access(|| {
|
||||
let mut pending = vec![root.to_path_buf()];
|
||||
let mut count = 0;
|
||||
while let Some(dir) = pending.pop() {
|
||||
let entries = match std::fs::read_dir(&dir) {
|
||||
Ok(entries) => entries,
|
||||
Err(e) if e.kind() == std::io::ErrorKind::NotFound => continue,
|
||||
Err(e) => return Err(e.into()),
|
||||
};
|
||||
for entry in entries {
|
||||
let entry = entry?;
|
||||
let kind = entry.file_type()?;
|
||||
if kind.is_dir() {
|
||||
pending.push(entry.path());
|
||||
} else if kind.is_file() && entry.path().extension().is_some_and(|e| e == "rredo") {
|
||||
let path = entry.path().with_extension("list");
|
||||
prepare_rewrite(&path)?;
|
||||
count += 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(count)
|
||||
})
|
||||
}
|
||||
/// Retire redo before an atomic snapshot replacement. The old native snapshot
|
||||
/// is synced first, so a crash after unlink and before rename retains every
|
||||
/// committed record. After rename there is no old redo to resurrect deletions.
|
||||
fn prepare_rewrite(p: &Path) -> Result<()> {
|
||||
if recover_redo(p)?.is_none() {
|
||||
return Ok(());
|
||||
}
|
||||
std::fs::File::open(p)?.sync_all()?;
|
||||
std::fs::remove_file(p.with_extension("rredo"))?;
|
||||
std::fs::File::open(p.parent().unwrap())?.sync_all()?;
|
||||
REDO_CACHE
|
||||
.get_or_init(|| Mutex::new(RedoCache::new()))
|
||||
.lock()
|
||||
.map_err(|_| anyhow!("archive redo cache poisoned"))?
|
||||
.remove(p);
|
||||
Ok(())
|
||||
}
|
||||
fn scan_unlocked<F: FnMut(Entry) -> Result<()>>(
|
||||
root: &Path,
|
||||
host: &str,
|
||||
@@ -458,7 +742,7 @@ fn save_unlocked(
|
||||
data.push_str(&encode_entry(e)?);
|
||||
}
|
||||
let p = storage::path(root, host, user, store).with_extension("list");
|
||||
recover_append(&p)?;
|
||||
prepare_rewrite(&p)?;
|
||||
match std::fs::remove_file(p.with_extension("lidx")) {
|
||||
Ok(()) => (),
|
||||
Err(e) if e.kind() == std::io::ErrorKind::NotFound => (),
|
||||
@@ -519,37 +803,57 @@ fn append_unlocked(
|
||||
entry: &Entry,
|
||||
) -> Result<()> {
|
||||
let data = encode_entry(entry)?;
|
||||
anyhow::ensure!(
|
||||
data.len() <= MAX_RECORD_BYTES,
|
||||
"archive record exceeds 16 MiB"
|
||||
);
|
||||
let path = storage::path(root, host, user, store).with_extension("list");
|
||||
let parent = path.parent().unwrap();
|
||||
std::fs::create_dir_all(parent)?;
|
||||
recover_append(&path)?;
|
||||
std::fs::create_dir_all(path.parent().unwrap())?;
|
||||
let state = recover_redo(&path)?;
|
||||
let mut opts = std::fs::OpenOptions::new();
|
||||
opts.create(true).append(true);
|
||||
opts.create(true).read(true).write(true).truncate(false);
|
||||
#[cfg(unix)]
|
||||
{
|
||||
use std::os::unix::fs::OpenOptionsExt;
|
||||
opts.mode(0o600);
|
||||
}
|
||||
let mut file = opts.open(&path)?;
|
||||
let length = file.metadata()?.len();
|
||||
let journal = path.with_extension("rappend");
|
||||
storage::atomic_write(&journal, &length.to_le_bytes())?;
|
||||
let mut state = match state {
|
||||
Some(state) => state,
|
||||
None => checkpoint_redo(&path)?,
|
||||
};
|
||||
if state.bytes >= REDO_LIMIT || state.frames >= REDO_FRAMES {
|
||||
state = checkpoint_redo(&path)?;
|
||||
}
|
||||
match std::fs::remove_file(path.with_extension("lidx")) {
|
||||
Ok(()) => (),
|
||||
Err(e) if e.kind() == std::io::ErrorKind::NotFound => (),
|
||||
Err(e) => return Err(e.into()),
|
||||
}
|
||||
let result = (|| -> Result<()> {
|
||||
file.write_all(data.as_bytes())?;
|
||||
file.sync_all()?;
|
||||
std::fs::remove_file(&journal)?;
|
||||
std::fs::File::open(parent)?.sync_all()?;
|
||||
Ok(())
|
||||
})();
|
||||
if result.is_err() {
|
||||
recover_append(&path)?;
|
||||
}
|
||||
result
|
||||
let mut frame = Vec::with_capacity(72 + data.len());
|
||||
frame.extend_from_slice(&(data.len() as u64).to_le_bytes());
|
||||
frame.extend_from_slice(&digest(&(data.len() as u64).to_le_bytes()));
|
||||
frame.extend_from_slice(&digest(data.as_bytes()));
|
||||
frame.extend_from_slice(data.as_bytes());
|
||||
// Invalidate the cache before any write, including error paths. A failed
|
||||
// sync is indeterminate and must be recovered, never blindly rolled back.
|
||||
REDO_CACHE
|
||||
.get_or_init(|| Mutex::new(RedoCache::new()))
|
||||
.lock()
|
||||
.map_err(|_| anyhow!("archive redo cache poisoned"))?
|
||||
.remove(&path);
|
||||
file.seek(SeekFrom::Start(state.length))?;
|
||||
file.write_all(data.as_bytes())?;
|
||||
let mut journal = std::fs::OpenOptions::new()
|
||||
.append(true)
|
||||
.open(path.with_extension("rredo"))?;
|
||||
journal.write_all(&frame)?;
|
||||
journal.sync_all()?;
|
||||
state.length += data.len() as u64;
|
||||
state.bytes += frame.len() as u64;
|
||||
state.frames += 1;
|
||||
remember_redo(&path, &state)?;
|
||||
Ok(())
|
||||
}
|
||||
/// Owner is localpart, peer is bare JID. Caller enforces archive preferences.
|
||||
pub fn archive_message(
|
||||
@@ -614,6 +918,7 @@ fn retain_unlocked<F: FnMut(&Entry) -> bool>(
|
||||
mut retain: F,
|
||||
) -> Result<usize> {
|
||||
let path = storage::path(root, host, user, store).with_extension("list");
|
||||
recover_append(&path)?;
|
||||
if !path.exists() {
|
||||
return Ok(0);
|
||||
}
|
||||
@@ -641,6 +946,7 @@ fn retain_unlocked<F: FnMut(&Entry) -> bool>(
|
||||
return Ok(0);
|
||||
}
|
||||
file.sync_all()?;
|
||||
prepare_rewrite(&path)?;
|
||||
match std::fs::remove_file(path.with_extension("lidx")) {
|
||||
Ok(()) => (),
|
||||
Err(e) if e.kind() == std::io::ErrorKind::NotFound => (),
|
||||
@@ -653,3 +959,57 @@ fn retain_unlocked<F: FnMut(&Entry) -> bool>(
|
||||
let _ = std::fs::remove_file(&temp);
|
||||
result
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
mod redo_rewrite_tests {
|
||||
use super::*;
|
||||
|
||||
fn fixture() -> (tempfile::TempDir, std::path::PathBuf) {
|
||||
let root = tempfile::tempdir().unwrap();
|
||||
append_new(
|
||||
root.path(),
|
||||
"example.org",
|
||||
Some("alice"),
|
||||
"archive",
|
||||
Entry {
|
||||
id: "committed-before-rewrite".into(),
|
||||
when: 42.,
|
||||
peer: String::new(),
|
||||
xml: "<message><body>durable</body></message>".into(),
|
||||
},
|
||||
)
|
||||
.unwrap();
|
||||
let path = storage::path(root.path(), "example.org", Some("alice"), "archive")
|
||||
.with_extension("list");
|
||||
(root, path)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn crash_gap_after_redo_retirement_before_native_rename_keeps_old_snapshot() {
|
||||
let (root, path) = fixture();
|
||||
let expected = std::fs::read(&path).unwrap();
|
||||
prepare_rewrite(&path).unwrap();
|
||||
// Crash boundary: redo unlink and directory sync completed, replacement
|
||||
// has not happened. Restart must retain the complete durable old list.
|
||||
assert!(!path.with_extension("rredo").exists());
|
||||
assert_eq!(std::fs::read(&path).unwrap(), expected);
|
||||
let rows = load(root.path(), "example.org", "alice").unwrap();
|
||||
assert_eq!(rows.len(), 1);
|
||||
assert_eq!(rows[0].id, "committed-before-rewrite");
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn crash_gap_after_native_rename_has_no_old_redo_to_resurrect_deletions() {
|
||||
let (root, path) = fixture();
|
||||
prepare_rewrite(&path).unwrap();
|
||||
storage::atomic_write(&path, b"").unwrap();
|
||||
// Crash boundary: the atomic replacement committed. Recovery sees the
|
||||
// new snapshot directly and cannot replay the removed old record.
|
||||
assert!(!path.with_extension("rredo").exists());
|
||||
assert!(
|
||||
load(root.path(), "example.org", "alice")
|
||||
.unwrap()
|
||||
.is_empty()
|
||||
);
|
||||
}
|
||||
}
|
||||
+20
-1
@@ -12,6 +12,25 @@ async fn main() {
|
||||
async fn run() -> Result<()> {
|
||||
let args: Vec<_> = env::args().skip(1).collect();
|
||||
match args.first().map(String::as_str) {
|
||||
Some("storage-checkpoint") if args.len() == 2 => {
|
||||
let (c, domain) = rust_xmpp::server::settings_config(Path::new(&args[1]))?;
|
||||
let data = Path::new(
|
||||
c.option(&domain, "data_path")
|
||||
.and_then(|v| v.as_str())
|
||||
.context("data_path missing")?,
|
||||
);
|
||||
let _data_guard = rust_xmpp::data_lock::DataDirectoryGuard::acquire(data)?;
|
||||
let archives = rust_xmpp::archive::checkpoint_all(data)?;
|
||||
let sm_path = data
|
||||
.join(rust_xmpp::storage::encode_name(&domain))
|
||||
.join("rust_smacks");
|
||||
let sessions = if sm_path.try_exists()? {
|
||||
rust_xmpp::session_store::checkpoint_exports(&sm_path)?
|
||||
} else {
|
||||
0
|
||||
};
|
||||
println!("Durable native archive handoff: {archives}; SM JSON exports: {sessions}");
|
||||
}
|
||||
Some("upload-migrate") if args.len() == 5 || args.len() == 6 => {
|
||||
anyhow::ensure!(
|
||||
args[3] == "--component" && (args.len() == 5 || args[5] == "--execute"),
|
||||
@@ -121,7 +140,7 @@ async fn run() -> Result<()> {
|
||||
println!("Account created: {}@{}", args[2], domain);
|
||||
}
|
||||
Some("--help") | None => println!(
|
||||
"rust-xmpp upload-migrate OLD_CONFIG NEW_CONFIG --component HOST [--execute]\nrust-xmpp check-config PATH\nrust-xmpp convert PATH json|yaml|toml OUTPUT\nrust-xmpp --version\n\nExperimental runtime: serve CONFIG --experimental\ncreate-user CONFIG USER --password-stdin\ncheck [CONFIG] (running server healthcheck)\nshell CONFIG COMMAND\n\nSTARTTLS, SCRAM/PLAIN, local messaging, roster, MAM, MUC, PEP/pubsub, stream management, push, federation and HTTP transports are implemented with limitations; see docs/COMPATIBILITY.md."
|
||||
"rust-xmpp upload-migrate OLD_CONFIG NEW_CONFIG --component HOST [--execute]\nrust-xmpp check-config PATH\nrust-xmpp convert PATH json|yaml|toml OUTPUT\nrust-xmpp --version\nrust-xmpp storage-checkpoint CONFIG (offline native handoff/downgrade)\n\nExperimental runtime: serve CONFIG --experimental\ncreate-user CONFIG USER --password-stdin\ncheck [CONFIG] (running server healthcheck)\nshell CONFIG COMMAND\n\nSTARTTLS, SCRAM/PLAIN, local messaging, roster, MAM, MUC, PEP/pubsub, stream management, push, federation and HTTP transports are implemented with limitations; see docs/COMPATIBILITY.md."
|
||||
),
|
||||
_ => bail!("invalid command; use --help"),
|
||||
}
|
||||
|
||||
+42
-1
@@ -3303,6 +3303,12 @@ async fn authenticated_transport<S: AsyncRead + AsyncWrite + Unpin>(
|
||||
ctx.event(Level::Debug, "stanza-received", &[("kind", kind)]);
|
||||
let handled_session = session.clone();
|
||||
let processing_started = Instant::now();
|
||||
// Fixed, payload-free phase metrics; unvisited phases stay zero for
|
||||
// implicit destinations, blocked messages and MUC handling.
|
||||
let mut message_phases = [Duration::ZERO; 7];
|
||||
let mut personal_message_started = false;
|
||||
let mut target_count = 0;
|
||||
let mut carbon_count = 0;
|
||||
async {
|
||||
if kind == "iq" {
|
||||
let typ = n.attribute("type").unwrap_or("");
|
||||
@@ -3612,7 +3618,10 @@ async fn authenticated_transport<S: AsyncRead + AsyncWrite + Unpin>(
|
||||
.unwrap_or(to),
|
||||
);
|
||||
let typ = n.attribute("type").unwrap_or("normal");
|
||||
personal_message_started = true;
|
||||
message_phases[0] = processing_started.elapsed();
|
||||
if rt.enabled("mam") {
|
||||
let sender_mam_started = Instant::now();
|
||||
xml = crate::mam::strip_owner_stanza_ids(&xml, &rt.domain, &user)?;
|
||||
let now = chrono::Utc::now().timestamp_millis() as f64 / 1000.0;
|
||||
if crate::mam::should_store(n)
|
||||
@@ -3637,7 +3646,9 @@ async fn authenticated_transport<S: AsyncRead + AsyncWrite + Unpin>(
|
||||
} else {
|
||||
sent_xml = xml.clone();
|
||||
}
|
||||
message_phases[1] = sender_mam_started.elapsed();
|
||||
if let Some(u) = &local {
|
||||
let recipient_mam_started = Instant::now();
|
||||
xml = crate::mam::strip_owner_stanza_ids(&xml, &rt.domain, u)?;
|
||||
if u == &user {
|
||||
xml = sent_xml.clone();
|
||||
@@ -3658,14 +3669,17 @@ async fn authenticated_transport<S: AsyncRead + AsyncWrite + Unpin>(
|
||||
)?;
|
||||
xml = stamped;
|
||||
}
|
||||
message_phases[2] = recipient_mam_started.elapsed();
|
||||
}
|
||||
}
|
||||
let delivery_started = Instant::now();
|
||||
let remote_delivered = remote && route_delivery(&rt, to, &xml).await.is_ok();
|
||||
let targets = if local.is_some() {
|
||||
rt.hub.message_targets_for_type(to, typ)
|
||||
} else {
|
||||
vec![]
|
||||
};
|
||||
target_count = targets.len();
|
||||
let mut delivered = usize::from(remote_delivered);
|
||||
let mut target_jids = vec![];
|
||||
for target in targets {
|
||||
@@ -3675,6 +3689,8 @@ async fn authenticated_transport<S: AsyncRead + AsyncWrite + Unpin>(
|
||||
target_jids.push(target.jid.clone());
|
||||
}
|
||||
}
|
||||
message_phases[3] = delivery_started.elapsed();
|
||||
let carbons_started = Instant::now();
|
||||
if rt.enabled("carbons")
|
||||
&& matches!(typ, "chat" | "normal")
|
||||
&& !n.children().any(|n| {
|
||||
@@ -3701,10 +3717,14 @@ async fn authenticated_transport<S: AsyncRead + AsyncWrite + Unpin>(
|
||||
escape(&s.owner),
|
||||
escape(&s.jid)
|
||||
);
|
||||
let _ = s.transmit(carbon);
|
||||
if s.transmit(carbon).is_ok() {
|
||||
carbon_count += 1;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
message_phases[4] = carbons_started.elapsed();
|
||||
let offline_push_started = Instant::now();
|
||||
// A push can wake the client immediately. Commit offline delivery
|
||||
// before publishing its notification, just as for durable SM output.
|
||||
if delivered == 0 && !matches!(typ, "error" | "headline") {
|
||||
@@ -3727,16 +3747,22 @@ async fn authenticated_transport<S: AsyncRead + AsyncWrite + Unpin>(
|
||||
enqueue_push_route(&rt, to, payload);
|
||||
}
|
||||
}
|
||||
message_phases[5] = offline_push_started.elapsed();
|
||||
|
||||
}
|
||||
Ok::<(), anyhow::Error>(())
|
||||
}.await?;
|
||||
if !personal_message_started {
|
||||
message_phases[0] = processing_started.elapsed();
|
||||
}
|
||||
let handled_started = Instant::now();
|
||||
if let Some(s) = handled_session {
|
||||
let mut state = s.lock().unwrap();
|
||||
if !state.closed {
|
||||
state.mark_handled()?;
|
||||
}
|
||||
}
|
||||
message_phases[6] = handled_started.elapsed();
|
||||
if kind == "message" && processing_started.elapsed() >= Duration::from_millis(250) {
|
||||
let route = n.attribute("to").map_or("implicit", |to| {
|
||||
if is_muc_destination(&rt.config, to) {
|
||||
@@ -3756,6 +3782,21 @@ async fn authenticated_transport<S: AsyncRead + AsyncWrite + Unpin>(
|
||||
&processing_started.elapsed().as_millis().to_string(),
|
||||
),
|
||||
("route", route),
|
||||
("pre_routing_ms", &message_phases[0].as_millis().to_string()),
|
||||
("sender_mam_ms", &message_phases[1].as_millis().to_string()),
|
||||
(
|
||||
"recipient_mam_ms",
|
||||
&message_phases[2].as_millis().to_string(),
|
||||
),
|
||||
("delivery_ms", &message_phases[3].as_millis().to_string()),
|
||||
("carbons_ms", &message_phases[4].as_millis().to_string()),
|
||||
(
|
||||
"offline_push_ms",
|
||||
&message_phases[5].as_millis().to_string(),
|
||||
),
|
||||
("handled_ms", &message_phases[6].as_millis().to_string()),
|
||||
("target_count", &target_count.to_string()),
|
||||
("carbon_count", &carbon_count.to_string()),
|
||||
],
|
||||
);
|
||||
}
|
||||
|
||||
+382
-35
@@ -2,10 +2,12 @@
|
||||
//!
|
||||
//! Recovery is deliberately strict: malformed, oversized, or inconsistent committed
|
||||
//! slots fail startup instead of silently losing acknowledged session state. Uncommitted
|
||||
//! UUID temporary files are discarded; the last renamed JSON slot remains authoritative.
|
||||
//! UUID temporary files are discarded; checksummed redo journals are authoritative.
|
||||
//! The readable JSON files are exports and are repaired from redo before loading.
|
||||
use crate::sessions::Session;
|
||||
use anyhow::{Result, ensure};
|
||||
use serde::{Deserialize, Serialize};
|
||||
use sha2::{Digest, Sha256};
|
||||
use std::{
|
||||
collections::BTreeMap,
|
||||
fs::{self, File, OpenOptions},
|
||||
@@ -122,6 +124,9 @@ pub(crate) struct Store {
|
||||
root: PathBuf,
|
||||
_lock: File,
|
||||
slots: Mutex<BTreeMap<String, (String, usize)>>,
|
||||
failed: std::sync::atomic::AtomicBool,
|
||||
retired: Mutex<Vec<String>>,
|
||||
journals: Mutex<BTreeMap<String, u64>>,
|
||||
}
|
||||
fn store(session: &Session) -> Result<Option<Arc<Store>>> {
|
||||
session
|
||||
@@ -180,6 +185,7 @@ impl Store {
|
||||
use std::os::unix::fs::OpenOptionsExt;
|
||||
options.mode(0o600);
|
||||
}
|
||||
regular_or_missing(&path.join("store.lock"))?;
|
||||
let lock = options.open(path.join("store.lock"))?;
|
||||
#[cfg(unix)]
|
||||
{
|
||||
@@ -192,6 +198,9 @@ impl Store {
|
||||
root: path.into(),
|
||||
_lock: lock,
|
||||
slots: Default::default(),
|
||||
failed: std::sync::atomic::AtomicBool::new(false),
|
||||
retired: Default::default(),
|
||||
journals: Default::default(),
|
||||
});
|
||||
let mut recovered = vec![];
|
||||
let mut total = 0usize;
|
||||
@@ -202,6 +211,7 @@ impl Store {
|
||||
let name = entry.file_name();
|
||||
let name = name.to_string_lossy();
|
||||
if let Some(id) = name.strip_suffix(".tmp") {
|
||||
let id = id.strip_suffix(".wal").unwrap_or(id);
|
||||
let owned = safe_id(id).is_ok()
|
||||
&& uuid::Uuid::parse_str(id).is_ok_and(|uuid| uuid.to_string() == id);
|
||||
if owned && entry.file_type()?.is_file() {
|
||||
@@ -210,6 +220,50 @@ impl Store {
|
||||
}
|
||||
}
|
||||
File::open(path)?.sync_all()?;
|
||||
// Redo precedes JSON validation: the export may be stale or torn.
|
||||
for entry in &entries {
|
||||
let name = entry.file_name();
|
||||
let name = name.to_string_lossy();
|
||||
if let Some(id) = name.strip_suffix(".wal") {
|
||||
safe_id(id)?;
|
||||
ensure!(
|
||||
entry.file_type()?.is_file(),
|
||||
"invalid stream management journal"
|
||||
);
|
||||
let snapshot = replay(&entry.path())?;
|
||||
// A process crash can leave a complete, checksummed append
|
||||
// only in page cache. Make its adoption durable before any
|
||||
// recovered slot is exposed to bootstrap or resume consumers.
|
||||
OpenOptions::new()
|
||||
.write(true)
|
||||
.open(entry.path())?
|
||||
.sync_data()?;
|
||||
{
|
||||
let mut journals = store.journals.lock().unwrap();
|
||||
journals.insert(id.to_string(), entry.metadata()?.len());
|
||||
ensure!(
|
||||
journals.len() <= SESSION_LIMIT + 64
|
||||
&& journals.values().sum::<u64>() <= JOURNAL_TOTAL_LIMIT,
|
||||
"stream management aggregate journal limit"
|
||||
);
|
||||
}
|
||||
if let Some(record) = &snapshot {
|
||||
validate_record(record, id)?;
|
||||
}
|
||||
store.export(
|
||||
id,
|
||||
snapshot
|
||||
.as_ref()
|
||||
.map(serde_json::to_vec)
|
||||
.transpose()?
|
||||
.as_deref(),
|
||||
)?;
|
||||
if snapshot.is_none() {
|
||||
store.retired.lock().unwrap().push(id.to_string());
|
||||
}
|
||||
}
|
||||
}
|
||||
let entries = fs::read_dir(path)?.collect::<std::io::Result<Vec<_>>>()?;
|
||||
for entry in entries {
|
||||
let name = entry.file_name();
|
||||
let name = name.to_string_lossy();
|
||||
@@ -237,20 +291,10 @@ impl Store {
|
||||
.take(SLOT_LIMIT as u64 + 1)
|
||||
.read_to_end(&mut bytes)?;
|
||||
let record: Record = serde_json::from_slice(&bytes)?;
|
||||
ensure!(
|
||||
record.version == 1
|
||||
&& record.pending.len() <= 4096
|
||||
&& record.sent.wrapping_sub(record.acked) as usize == record.pending.len()
|
||||
&& crate::protocol::bare(&record.jid) == record.owner
|
||||
&& record.jid.contains('/'),
|
||||
"invalid durable session"
|
||||
);
|
||||
ensure!(
|
||||
name == format!("{}.json", safe_id(&record.resume_id)?),
|
||||
"invalid slot token"
|
||||
);
|
||||
let id = name.strip_suffix(".json").unwrap();
|
||||
validate_record(&record, id)?;
|
||||
if record.deadline_ms <= now_ms() {
|
||||
fs::remove_file(entry.path())?;
|
||||
store.commit(id, None)?;
|
||||
continue;
|
||||
}
|
||||
store
|
||||
@@ -261,6 +305,7 @@ impl Store {
|
||||
recovered.push(record);
|
||||
}
|
||||
File::open(path)?.sync_all()?;
|
||||
store.collect_retired()?;
|
||||
Ok((store, recovered))
|
||||
}
|
||||
fn write(&self, s: &Session, expected: Option<&str>) -> Result<()> {
|
||||
@@ -282,7 +327,8 @@ impl Store {
|
||||
);
|
||||
} else {
|
||||
ensure!(
|
||||
!slots.contains_key(&record.resume_id),
|
||||
!slots.contains_key(&record.resume_id)
|
||||
&& !self.retired.lock().unwrap().contains(&record.resume_id),
|
||||
"duplicate stream token"
|
||||
);
|
||||
}
|
||||
@@ -296,24 +342,7 @@ impl Store {
|
||||
total - previous + bytes.len() <= TOTAL_LIMIT,
|
||||
"stream management storage limit"
|
||||
);
|
||||
let tmp = self.root.join(format!("{name}.tmp"));
|
||||
let mut options = OpenOptions::new();
|
||||
options.write(true).create(true).truncate(true);
|
||||
#[cfg(unix)]
|
||||
{
|
||||
use std::os::unix::fs::OpenOptionsExt;
|
||||
options.mode(0o600);
|
||||
}
|
||||
let mut file = options.open(&tmp)?;
|
||||
#[cfg(unix)]
|
||||
{
|
||||
use std::os::unix::fs::PermissionsExt;
|
||||
file.set_permissions(fs::Permissions::from_mode(0o600))?;
|
||||
}
|
||||
file.write_all(&bytes)?;
|
||||
file.sync_all()?;
|
||||
fs::rename(tmp, self.root.join(format!("{name}.json")))?;
|
||||
File::open(&self.root)?.sync_all()?;
|
||||
self.commit(name, Some(&bytes))?;
|
||||
slots.insert(record.resume_id, (record.connection, bytes.len()));
|
||||
Ok(())
|
||||
}
|
||||
@@ -322,12 +351,189 @@ impl Store {
|
||||
if slots.get(id).is_none_or(|v| v.0 != connection) {
|
||||
return Ok(());
|
||||
}
|
||||
fs::remove_file(self.root.join(format!("{}.json", safe_id(id)?)))?;
|
||||
File::open(&self.root)?.sync_all()?;
|
||||
self.commit(safe_id(id)?, None)?;
|
||||
slots.remove(id);
|
||||
Ok(())
|
||||
}
|
||||
// A journal failure may have reached stable storage. Fence this store rather
|
||||
// than allowing an in-memory rollback to overwrite an uncertain commit.
|
||||
fn commit(&self, id: &str, bytes: Option<&[u8]>) -> Result<()> {
|
||||
use std::sync::atomic::Ordering;
|
||||
ensure!(
|
||||
!self.failed.load(Ordering::SeqCst),
|
||||
"stream management store failed"
|
||||
);
|
||||
let result = self.commit_inner(id, bytes);
|
||||
if result.is_err() {
|
||||
self.failed.store(true, Ordering::SeqCst);
|
||||
}
|
||||
result
|
||||
}
|
||||
fn commit_inner(&self, id: &str, bytes: Option<&[u8]>) -> Result<()> {
|
||||
let path = self.root.join(format!("{id}.wal"));
|
||||
regular_or_missing(&path)?;
|
||||
if !path.try_exists()? {
|
||||
// The baseline embeds the previously committed JSON, so recovery
|
||||
// never depends on an export modified without a subsequent fsync.
|
||||
let old = self.root.join(format!("{id}.json"));
|
||||
let baseline = if old.try_exists()? {
|
||||
regular_or_missing(&old)?;
|
||||
let mut baseline = Vec::new();
|
||||
File::open(old)?
|
||||
.take(SLOT_LIMIT as u64 + 1)
|
||||
.read_to_end(&mut baseline)?;
|
||||
ensure!(baseline.len() <= SLOT_LIMIT, "stream management slot limit");
|
||||
let record: Record = serde_json::from_slice(&baseline)?;
|
||||
validate_record(&record, id)?;
|
||||
Some(baseline)
|
||||
} else {
|
||||
None
|
||||
};
|
||||
let baseline_frame = frame(true, baseline.as_deref())?;
|
||||
if self.journals.lock().unwrap().values().sum::<u64>()
|
||||
+ JOURNAL_MAGIC.len() as u64
|
||||
+ baseline_frame.len() as u64
|
||||
> JOURNAL_TOTAL_LIMIT
|
||||
{
|
||||
self.compact_journals()?;
|
||||
}
|
||||
self.reset_journal(id, baseline.as_deref())?;
|
||||
}
|
||||
// Rotation is amortized over at least several small commits. Sync the
|
||||
// exported snapshot and its rename BEFORE replacing the redo history.
|
||||
let len = fs::metadata(&path)?.len();
|
||||
if len >= JOURNAL_ROTATE {
|
||||
let old = replay(&path)?;
|
||||
let old = old.as_ref().map(serde_json::to_vec).transpose()?;
|
||||
self.checkpoint_journal(id, old.as_deref())?;
|
||||
}
|
||||
let frame = frame(false, bytes)?;
|
||||
if self.journals.lock().unwrap().values().sum::<u64>() + frame.len() as u64
|
||||
> JOURNAL_TOTAL_LIMIT
|
||||
{
|
||||
// An aggregate cap prevents many small slots each growing a full
|
||||
// per-slot rotation budget. Compaction is rare and amortized over
|
||||
// hundreds of MiB of durable appends.
|
||||
self.compact_journals()?;
|
||||
}
|
||||
let mut journal = private_options().append(true).open(&path)?;
|
||||
let new_len = journal.metadata()?.len() + frame.len() as u64;
|
||||
journal.write_all(&frame)?;
|
||||
journal.sync_data()?;
|
||||
self.journals
|
||||
.lock()
|
||||
.unwrap()
|
||||
.insert(id.to_string(), new_len);
|
||||
// This remains readable for diagnostics and wire assertions, but only
|
||||
// the journal is authoritative after a crash.
|
||||
if self.export(id, bytes).is_err() {
|
||||
// The durable commit succeeded. Its caller must advance memory to
|
||||
// the same state rather than rolling it back. Fence future writes;
|
||||
// startup will reconstruct the export from redo.
|
||||
self.failed.store(true, std::sync::atomic::Ordering::SeqCst);
|
||||
return Ok(());
|
||||
}
|
||||
if bytes.is_none() {
|
||||
let mut retired = self.retired.lock().unwrap();
|
||||
retired.push(id.to_string());
|
||||
let collect = retired.len() >= 64;
|
||||
drop(retired);
|
||||
if collect && self.collect_retired().is_err() {
|
||||
self.failed.store(true, std::sync::atomic::Ordering::SeqCst);
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
fn compact_journals(&self) -> Result<()> {
|
||||
let ids: Vec<String> = self.journals.lock().unwrap().keys().cloned().collect();
|
||||
for id in ids {
|
||||
let snapshot = replay(&self.root.join(format!("{id}.wal")))?;
|
||||
let snapshot = snapshot.as_ref().map(serde_json::to_vec).transpose()?;
|
||||
self.checkpoint_journal(&id, snapshot.as_deref())?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
fn collect_retired(&self) -> Result<()> {
|
||||
let mut retired = self.retired.lock().unwrap();
|
||||
if retired.is_empty() {
|
||||
return Ok(());
|
||||
}
|
||||
// Deletions become durable before their redo tombstones disappear.
|
||||
File::open(&self.root)?.sync_all()?;
|
||||
for id in retired.iter() {
|
||||
let path = self.root.join(format!("{id}.wal"));
|
||||
if path.try_exists()? {
|
||||
fs::remove_file(path)?;
|
||||
}
|
||||
self.journals.lock().unwrap().remove(id);
|
||||
}
|
||||
File::open(&self.root)?.sync_all()?;
|
||||
retired.clear();
|
||||
Ok(())
|
||||
}
|
||||
fn checkpoint_journal(&self, id: &str, bytes: Option<&[u8]>) -> Result<()> {
|
||||
self.export(id, bytes)?;
|
||||
if bytes.is_some() {
|
||||
File::open(self.root.join(format!("{id}.json")))?.sync_all()?;
|
||||
}
|
||||
File::open(&self.root)?.sync_all()?;
|
||||
self.reset_journal(id, bytes)
|
||||
}
|
||||
fn reset_journal(&self, id: &str, baseline: Option<&[u8]>) -> Result<()> {
|
||||
let tmp = self.root.join(format!("{id}.wal.tmp"));
|
||||
regular_or_missing(&tmp)?;
|
||||
let mut file = private_options().create(true).truncate(true).open(&tmp)?;
|
||||
private_permissions(&file)?;
|
||||
file.write_all(JOURNAL_MAGIC)?;
|
||||
file.write_all(&frame(true, baseline)?)?;
|
||||
file.sync_all()?;
|
||||
let len = file.metadata()?.len();
|
||||
fs::rename(tmp, self.root.join(format!("{id}.wal")))?;
|
||||
File::open(&self.root)?.sync_all()?;
|
||||
self.journals.lock().unwrap().insert(id.to_string(), len);
|
||||
Ok(())
|
||||
}
|
||||
fn export(&self, id: &str, bytes: Option<&[u8]>) -> Result<()> {
|
||||
let dest = self.root.join(format!("{id}.json"));
|
||||
if let Some(bytes) = bytes {
|
||||
let tmp = self.root.join(format!("{id}.tmp"));
|
||||
regular_or_missing(&tmp)?;
|
||||
let mut file = private_options().create(true).truncate(true).open(&tmp)?;
|
||||
private_permissions(&file)?;
|
||||
file.write_all(bytes)?;
|
||||
fs::rename(tmp, dest)?;
|
||||
} else if dest.try_exists()? {
|
||||
fs::remove_file(dest)?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
/// Materialize the authoritative redo state as durable legacy JSON snapshots.
|
||||
///
|
||||
/// This is an offline handoff operation: it acquires the same exclusive store
|
||||
/// lock as a persistent Hub and fails while a live owner exists. Recovery and
|
||||
/// expiry run first. All current JSON data and directory entries become durable
|
||||
/// before any redo journal is removed. A failed checkpoint may leave redundant
|
||||
/// journals, but never removes the only durable copy of an acknowledged state.
|
||||
/// The caller should additionally hold its server-wide data-directory lock.
|
||||
pub fn checkpoint_exports(path: &Path) -> Result<usize> {
|
||||
let (store, records) = Store::open(path)?;
|
||||
for record in &records {
|
||||
let json = path.join(format!("{}.json", safe_id(&record.resume_id)?));
|
||||
regular_or_missing(&json)?;
|
||||
File::open(json)?.sync_all()?;
|
||||
}
|
||||
File::open(path)?.sync_all()?;
|
||||
let journals: Vec<String> = store.journals.lock().unwrap().keys().cloned().collect();
|
||||
for id in journals {
|
||||
let journal = path.join(format!("{id}.wal"));
|
||||
regular_or_missing(&journal)?;
|
||||
fs::remove_file(journal)?;
|
||||
}
|
||||
File::open(path)?.sync_all()?;
|
||||
Ok(records.len())
|
||||
}
|
||||
|
||||
fn safe_id(id: &str) -> Result<&str> {
|
||||
ensure!(
|
||||
!id.is_empty()
|
||||
@@ -337,3 +543,144 @@ fn safe_id(id: &str) -> Result<&str> {
|
||||
);
|
||||
Ok(id)
|
||||
}
|
||||
|
||||
const JOURNAL_MAGIC: &[u8; 8] = b"XMPPSM01";
|
||||
const JOURNAL_ROTATE: u64 = 16 * 1024 * 1024;
|
||||
const JOURNAL_TOTAL_LIMIT: u64 = (TOTAL_LIMIT * 2) as u64;
|
||||
const FRAME_LIMIT: usize = SLOT_LIMIT + 1024;
|
||||
#[derive(Serialize, Deserialize)]
|
||||
struct JournalEntry<T> {
|
||||
baseline: bool,
|
||||
snapshot: Option<T>,
|
||||
}
|
||||
fn private_options() -> OpenOptions {
|
||||
let mut options = OpenOptions::new();
|
||||
options.write(true);
|
||||
#[cfg(unix)]
|
||||
{
|
||||
use std::os::unix::fs::OpenOptionsExt;
|
||||
options.mode(0o600);
|
||||
}
|
||||
options
|
||||
}
|
||||
fn private_permissions(file: &File) -> Result<()> {
|
||||
#[cfg(unix)]
|
||||
{
|
||||
use std::os::unix::fs::PermissionsExt;
|
||||
file.set_permissions(fs::Permissions::from_mode(0o600))?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
fn frame(baseline: bool, bytes: Option<&[u8]>) -> Result<Vec<u8>> {
|
||||
// The snapshot was already validated and serialized by write() or replay().
|
||||
// Copy its JSON directly into the frame rather than parsing and serializing
|
||||
// the entire pending queue again on every checkpoint.
|
||||
if let Some(bytes) = bytes {
|
||||
ensure!(bytes.len() <= SLOT_LIMIT, "stream management slot limit");
|
||||
}
|
||||
let mut payload = Vec::with_capacity(bytes.map_or(4, |snapshot| snapshot.len()) + 32);
|
||||
payload.extend_from_slice(if baseline {
|
||||
b"{\"baseline\":true,\"snapshot\":"
|
||||
} else {
|
||||
b"{\"baseline\":false,\"snapshot\":"
|
||||
});
|
||||
payload.extend_from_slice(bytes.unwrap_or(b"null"));
|
||||
payload.push(b'}');
|
||||
ensure!(
|
||||
payload.len() <= FRAME_LIMIT,
|
||||
"stream management journal frame limit"
|
||||
);
|
||||
let len = payload.len() as u32;
|
||||
let mut result = Vec::with_capacity(payload.len() + 40);
|
||||
result.extend_from_slice(&len.to_le_bytes());
|
||||
result.extend_from_slice(&(!len).to_le_bytes());
|
||||
result.extend_from_slice(&Sha256::digest(&payload));
|
||||
result.extend_from_slice(&payload);
|
||||
Ok(result)
|
||||
}
|
||||
fn replay(path: &Path) -> Result<Option<Record>> {
|
||||
let mut file = OpenOptions::new().read(true).write(true).open(path)?;
|
||||
private_permissions(&file)?;
|
||||
ensure!(
|
||||
file.metadata()?.len() <= JOURNAL_ROTATE + (FRAME_LIMIT + 40) as u64 + 8,
|
||||
"stream management journal limit"
|
||||
);
|
||||
let mut bytes = Vec::new();
|
||||
(&mut file)
|
||||
.take(JOURNAL_ROTATE + FRAME_LIMIT as u64 + 49)
|
||||
.read_to_end(&mut bytes)?;
|
||||
ensure!(
|
||||
bytes.starts_with(JOURNAL_MAGIC),
|
||||
"invalid stream management journal header"
|
||||
);
|
||||
let mut offset = 8;
|
||||
let mut first = true;
|
||||
let mut latest_payload = None;
|
||||
while offset < bytes.len() {
|
||||
if bytes.len() - offset < 8 {
|
||||
break;
|
||||
}
|
||||
let len = u32::from_le_bytes(bytes[offset..offset + 4].try_into().unwrap());
|
||||
let complement = u32::from_le_bytes(bytes[offset + 4..offset + 8].try_into().unwrap());
|
||||
ensure!(
|
||||
len == !complement && len as usize <= FRAME_LIMIT,
|
||||
"corrupt stream management journal frame"
|
||||
);
|
||||
let end = offset + 40 + len as usize;
|
||||
if end > bytes.len() {
|
||||
break;
|
||||
}
|
||||
let payload = &bytes[offset + 40..end];
|
||||
ensure!(
|
||||
Sha256::digest(payload).as_slice() == &bytes[offset + 8..offset + 40],
|
||||
"stream management journal checksum mismatch"
|
||||
);
|
||||
// Historical snapshots were validated before their original commit.
|
||||
// Check every byte and the full JSON syntax again, without allocating
|
||||
// old pending queues or serializing superseded snapshots during replay.
|
||||
let entry: JournalEntry<serde::de::IgnoredAny> = serde_json::from_slice(payload)?;
|
||||
ensure!(
|
||||
entry.baseline == first,
|
||||
"invalid stream management journal baseline"
|
||||
);
|
||||
latest_payload = Some(payload);
|
||||
first = false;
|
||||
offset = end;
|
||||
}
|
||||
ensure!(!first, "incomplete stream management journal baseline");
|
||||
let latest: JournalEntry<Record> = serde_json::from_slice(latest_payload.unwrap())?;
|
||||
if let Some(record) = &latest.snapshot {
|
||||
let id = path.file_stem().unwrap().to_string_lossy();
|
||||
validate_record(record, &id)?;
|
||||
ensure!(
|
||||
serde_json::to_vec(record)?.len() <= SLOT_LIMIT,
|
||||
"stream management slot limit"
|
||||
);
|
||||
}
|
||||
if offset < bytes.len() {
|
||||
file.set_len(offset as u64)?;
|
||||
file.sync_all()?;
|
||||
}
|
||||
Ok(latest.snapshot)
|
||||
}
|
||||
fn validate_record(record: &Record, id: &str) -> Result<()> {
|
||||
ensure!(
|
||||
record.version == 1
|
||||
&& record.pending.len() <= 4096
|
||||
&& record.sent.wrapping_sub(record.acked) as usize == record.pending.len()
|
||||
&& crate::protocol::bare(&record.jid) == record.owner
|
||||
&& record.jid.contains('/'),
|
||||
"invalid durable session"
|
||||
);
|
||||
ensure!(safe_id(&record.resume_id)? == id, "invalid slot token");
|
||||
Ok(())
|
||||
}
|
||||
|
||||
fn regular_or_missing(path: &Path) -> Result<()> {
|
||||
match fs::symlink_metadata(path) {
|
||||
Ok(metadata) => ensure!(metadata.is_file(), "invalid stream management file"),
|
||||
Err(error) if error.kind() == std::io::ErrorKind::NotFound => {}
|
||||
Err(error) => return Err(error.into()),
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
+1
-1
@@ -427,7 +427,7 @@ pub fn purge_user(root: &Path, host: &str, user: &str) -> Result<()> {
|
||||
if !ty.is_dir() {
|
||||
continue;
|
||||
}
|
||||
for extension in ["dat", "list", "lidx", "rappend"] {
|
||||
for extension in ["dat", "list", "lidx", "rappend", "rredo"] {
|
||||
let p = entry.path().join(format!("{user}.{extension}"));
|
||||
match fs::symlink_metadata(&p) {
|
||||
Ok(m) => {
|
||||
|
||||
@@ -0,0 +1,347 @@
|
||||
//! Crash fixtures model independent loss of unsynced native writes. The redo
|
||||
//! sidecar is the durable commit source until a native checkpoint or handoff.
|
||||
use rust_xmpp::{archive, storage};
|
||||
use std::{
|
||||
fs,
|
||||
io::Write,
|
||||
path::{Path, PathBuf},
|
||||
process::Command,
|
||||
time::{Duration, Instant},
|
||||
};
|
||||
|
||||
fn path(root: &Path) -> PathBuf {
|
||||
storage::path(root, "example.org", Some("alice"), "archive").with_extension("list")
|
||||
}
|
||||
fn entry(id: &str) -> archive::Entry {
|
||||
archive::Entry {
|
||||
id: id.into(),
|
||||
when: 42.,
|
||||
peer: "bob@example.org".into(),
|
||||
xml: "<message xmlns='jabber:client'><body>hello</body></message>".into(),
|
||||
}
|
||||
}
|
||||
fn append(root: &Path, id: &str) {
|
||||
archive::append_new(root, "example.org", Some("alice"), "archive", entry(id)).unwrap();
|
||||
}
|
||||
fn ids(root: &Path) -> Vec<String> {
|
||||
archive::load(root, "example.org", "alice")
|
||||
.unwrap()
|
||||
.into_iter()
|
||||
.map(|e| e.id)
|
||||
.collect()
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn committed_redo_recovers_missing_and_partial_native_without_duplicates() {
|
||||
let t = tempfile::tempdir().unwrap();
|
||||
append(t.path(), "one");
|
||||
append(t.path(), "two");
|
||||
let p = path(t.path());
|
||||
let expected = fs::read(&p).unwrap();
|
||||
for length in [0, 1, expected.len() / 2, expected.len() - 1] {
|
||||
fs::OpenOptions::new()
|
||||
.write(true)
|
||||
.open(&p)
|
||||
.unwrap()
|
||||
.set_len(length as u64)
|
||||
.unwrap();
|
||||
assert_eq!(ids(t.path()), ["one", "two"]);
|
||||
assert_eq!(fs::read(&p).unwrap(), expected);
|
||||
assert_eq!(ids(t.path()), ["one", "two"]);
|
||||
}
|
||||
fs::remove_file(&p).unwrap();
|
||||
assert_eq!(ids(t.path()), ["one", "two"]);
|
||||
assert_eq!(fs::read(&p).unwrap(), expected);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn incomplete_final_frame_and_uncommitted_native_suffix_are_discarded() {
|
||||
for extra in [0, 1, 7, 20, 39, 40, 71, 72, 90] {
|
||||
let t = tempfile::tempdir().unwrap();
|
||||
append(t.path(), "one");
|
||||
let p = path(t.path());
|
||||
let committed = fs::read(&p).unwrap();
|
||||
let journal = p.with_extension("rredo");
|
||||
let committed_journal = fs::metadata(&journal).unwrap().len();
|
||||
append(t.path(), "two");
|
||||
// Construct a crash before the second frame was fully persisted.
|
||||
fs::OpenOptions::new()
|
||||
.write(true)
|
||||
.open(&journal)
|
||||
.unwrap()
|
||||
.set_len(committed_journal + extra)
|
||||
.unwrap();
|
||||
assert_eq!(ids(t.path()), ["one"]);
|
||||
assert_eq!(fs::read(&p).unwrap(), committed);
|
||||
assert_eq!(fs::metadata(&journal).unwrap().len(), committed_journal);
|
||||
append(t.path(), "three");
|
||||
assert_eq!(ids(t.path()), ["one", "three"]);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn complete_checksum_corruption_fails_without_modifying_native_or_journal() {
|
||||
for offset in [8, 64, 96, 104, 136, 168] {
|
||||
let t = tempfile::tempdir().unwrap();
|
||||
append(t.path(), "one");
|
||||
let p = path(t.path());
|
||||
let native = fs::read(&p).unwrap();
|
||||
let journal = p.with_extension("rredo");
|
||||
let mut corrupt = fs::read(&journal).unwrap();
|
||||
corrupt[offset] ^= 1;
|
||||
fs::write(&journal, &corrupt).unwrap();
|
||||
assert!(archive::load(t.path(), "example.org", "alice").is_err());
|
||||
assert_eq!(fs::read(&p).unwrap(), native);
|
||||
assert_eq!(fs::read(&journal).unwrap(), corrupt);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn bounded_rotation_recovers_suffix_and_rewrites_never_resurrect_deleted_records() {
|
||||
let t = tempfile::tempdir().unwrap();
|
||||
for i in 0..300 {
|
||||
append(t.path(), &format!("id{i}"));
|
||||
}
|
||||
let p = path(t.path());
|
||||
let journal = p.with_extension("rredo");
|
||||
let data = fs::read(&journal).unwrap();
|
||||
assert!(data.len() < 1024 * 1024);
|
||||
let baseline = u64::from_le_bytes(data[8..16].try_into().unwrap());
|
||||
assert!(baseline > 0);
|
||||
fs::OpenOptions::new()
|
||||
.write(true)
|
||||
.open(&p)
|
||||
.unwrap()
|
||||
.set_len(baseline)
|
||||
.unwrap();
|
||||
assert_eq!(ids(t.path()).len(), 300);
|
||||
archive::delete_ids(
|
||||
t.path(),
|
||||
"example.org",
|
||||
Some("alice"),
|
||||
"archive",
|
||||
&["id299".into()],
|
||||
)
|
||||
.unwrap();
|
||||
assert_eq!(ids(t.path()).len(), 299);
|
||||
append(t.path(), "after-delete");
|
||||
assert!(!ids(t.path()).contains(&"id299".into()));
|
||||
archive::save_store(
|
||||
t.path(),
|
||||
"example.org",
|
||||
Some("alice"),
|
||||
"archive",
|
||||
&[entry("replacement")],
|
||||
)
|
||||
.unwrap();
|
||||
assert_eq!(ids(t.path()), ["replacement"]);
|
||||
append(t.path(), "after-save");
|
||||
assert_eq!(ids(t.path()), ["replacement", "after-save"]);
|
||||
archive::expire_before(t.path(), "example.org", Some("alice"), "archive", 43.).unwrap();
|
||||
assert!(ids(t.path()).is_empty());
|
||||
append(t.path(), "after-expire");
|
||||
assert_eq!(ids(t.path()), ["after-expire"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn byte_limit_rotates_before_frame_count_limit() {
|
||||
let t = tempfile::tempdir().unwrap();
|
||||
for i in 0..20 {
|
||||
let mut e = entry(&format!("big{i}"));
|
||||
e.xml = format!("<message><body>{}</body></message>", "x".repeat(100_000));
|
||||
archive::append_new(t.path(), "example.org", Some("alice"), "archive", e).unwrap();
|
||||
}
|
||||
let data = fs::read(path(t.path()).with_extension("rredo")).unwrap();
|
||||
assert!(u64::from_le_bytes(data[8..16].try_into().unwrap()) > 0);
|
||||
assert!(data.len() < 1024 * 1024 + 110_000);
|
||||
assert_eq!(ids(t.path()).len(), 20);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn explicit_native_atomic_handoff_and_missing_baseline_fail_closed() {
|
||||
let t = tempfile::tempdir().unwrap();
|
||||
append(t.path(), "old");
|
||||
let p = path(t.path());
|
||||
archive::checkpoint_all(t.path()).unwrap();
|
||||
storage::atomic_write(
|
||||
&p,
|
||||
b"item({name='message';attr={};key='native';when=42});\n",
|
||||
)
|
||||
.unwrap();
|
||||
assert_eq!(ids(t.path()), ["native"]);
|
||||
append(t.path(), "rust");
|
||||
assert_eq!(ids(t.path()), ["native", "rust"]);
|
||||
fs::OpenOptions::new()
|
||||
.append(true)
|
||||
.open(&p)
|
||||
.unwrap()
|
||||
.write_all(b"item(bad uncommitted);")
|
||||
.unwrap();
|
||||
assert_eq!(ids(t.path()), ["native", "rust"]);
|
||||
// The durable baseline cannot be reconstructed from suffix-only frames.
|
||||
fs::remove_file(&p).unwrap();
|
||||
assert!(archive::load(t.path(), "example.org", "alice").is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn native_baseline_in_place_edits_are_rejected() {
|
||||
let t = tempfile::tempdir().unwrap();
|
||||
let p = path(t.path());
|
||||
fs::create_dir_all(p.parent().unwrap()).unwrap();
|
||||
let original = b"item({name='message';attr={};key='native';when=42});\n";
|
||||
fs::write(&p, original).unwrap();
|
||||
append(t.path(), "rust");
|
||||
let mut data = fs::read(&p).unwrap();
|
||||
data[5] ^= 1;
|
||||
fs::write(&p, &data).unwrap();
|
||||
assert!(archive::load(t.path(), "example.org", "alice").is_err());
|
||||
assert_eq!(fs::read(&p).unwrap(), data);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn redo_process_child() {
|
||||
let Some(root) = std::env::var_os("XMPP_ARCHIVE_REDO_CHILD") else {
|
||||
return;
|
||||
};
|
||||
let root = Path::new(&root);
|
||||
append(root, "accepted-before-kill");
|
||||
fs::write(root.join("ready"), b"ready").unwrap();
|
||||
loop {
|
||||
std::thread::park();
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn killed_writer_retains_accepted_records_after_native_data_loss() {
|
||||
let t = tempfile::tempdir().unwrap();
|
||||
let mut child = Command::new(std::env::current_exe().unwrap())
|
||||
.args(["--exact", "redo_process_child", "--nocapture"])
|
||||
.env("XMPP_ARCHIVE_REDO_CHILD", t.path())
|
||||
.spawn()
|
||||
.unwrap();
|
||||
let deadline = Instant::now() + Duration::from_secs(10);
|
||||
while !t.path().join("ready").exists() {
|
||||
if Instant::now() >= deadline {
|
||||
let _ = child.kill();
|
||||
let _ = child.wait();
|
||||
panic!("redo child not ready");
|
||||
}
|
||||
assert!(
|
||||
child.try_wait().unwrap().is_none(),
|
||||
"redo child exited early"
|
||||
);
|
||||
std::thread::sleep(Duration::from_millis(10));
|
||||
}
|
||||
child.kill().unwrap();
|
||||
child.wait().unwrap();
|
||||
// Simulate loss of dirty native pages independently of the synced journal.
|
||||
fs::OpenOptions::new()
|
||||
.write(true)
|
||||
.open(path(t.path()))
|
||||
.unwrap()
|
||||
.set_len(0)
|
||||
.unwrap();
|
||||
assert_eq!(ids(t.path()), ["accepted-before-kill"]);
|
||||
assert_eq!(ids(t.path()), ["accepted-before-kill"]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn durable_handoff_allows_native_in_place_append_and_next_rust_append() {
|
||||
let t = tempfile::tempdir().unwrap();
|
||||
append(t.path(), "rust-before");
|
||||
let p = path(t.path());
|
||||
assert_eq!(archive::checkpoint_all(t.path()).unwrap(), 1);
|
||||
assert!(!p.with_extension("rredo").exists());
|
||||
fs::OpenOptions::new()
|
||||
.append(true)
|
||||
.open(&p)
|
||||
.unwrap()
|
||||
.write_all(b"item({name='message';attr={};key='native';when=42});\n")
|
||||
.unwrap();
|
||||
assert_eq!(ids(t.path()), ["rust-before", "native"]);
|
||||
append(t.path(), "rust-after");
|
||||
assert_eq!(ids(t.path()), ["rust-before", "native", "rust-after"]);
|
||||
assert_eq!(archive::checkpoint_all(t.path()).unwrap(), 1);
|
||||
assert_eq!(archive::checkpoint_all(t.path()).unwrap(), 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn legacy_rollback_marker_is_migrated_before_redo_append() {
|
||||
let t = tempfile::tempdir().unwrap();
|
||||
let p = path(t.path());
|
||||
fs::create_dir_all(p.parent().unwrap()).unwrap();
|
||||
let original = b"item({name='message';attr={};key='legacy';when=42});\n";
|
||||
fs::write(&p, [original.as_slice(), b"item(broken"].concat()).unwrap();
|
||||
fs::write(
|
||||
p.with_extension("rappend"),
|
||||
(original.len() as u64).to_le_bytes(),
|
||||
)
|
||||
.unwrap();
|
||||
append(t.path(), "redo");
|
||||
assert_eq!(ids(t.path()), ["legacy", "redo"]);
|
||||
assert!(!p.with_extension("rappend").exists());
|
||||
assert!(p.with_extension("rredo").exists());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn copied_archive_replays_committed_redo_despite_new_native_inode() {
|
||||
for native_baseline in [false, true] {
|
||||
for lose_entire_suffix in [false, true] {
|
||||
let source = tempfile::tempdir().unwrap();
|
||||
let source_path = path(source.path());
|
||||
if native_baseline {
|
||||
fs::create_dir_all(source_path.parent().unwrap()).unwrap();
|
||||
fs::write(
|
||||
&source_path,
|
||||
b"item({name='message';attr={};key='baseline';when=42});\n",
|
||||
)
|
||||
.unwrap();
|
||||
}
|
||||
append(source.path(), "accepted-one");
|
||||
append(source.path(), "accepted-two");
|
||||
let expected = fs::read(&source_path).unwrap();
|
||||
let redo = fs::read(source_path.with_extension("rredo")).unwrap();
|
||||
let baseline = u64::from_le_bytes(redo[8..16].try_into().unwrap());
|
||||
let restored = tempfile::tempdir().unwrap();
|
||||
let restored_path = path(restored.path());
|
||||
fs::create_dir_all(restored_path.parent().unwrap()).unwrap();
|
||||
fs::copy(&source_path, &restored_path).unwrap();
|
||||
fs::copy(
|
||||
source_path.with_extension("rredo"),
|
||||
restored_path.with_extension("rredo"),
|
||||
)
|
||||
.unwrap();
|
||||
let length = if lose_entire_suffix {
|
||||
baseline
|
||||
} else {
|
||||
expected.len() as u64 - 1
|
||||
};
|
||||
fs::OpenOptions::new()
|
||||
.write(true)
|
||||
.open(&restored_path)
|
||||
.unwrap()
|
||||
.set_len(length)
|
||||
.unwrap();
|
||||
let recovered = ids(restored.path());
|
||||
assert_eq!(recovered.last().unwrap(), "accepted-two");
|
||||
assert_eq!(recovered.len(), if native_baseline { 3 } else { 2 });
|
||||
assert_eq!(fs::read(&restored_path).unwrap(), expected);
|
||||
append(restored.path(), "after-restore");
|
||||
assert_eq!(ids(restored.path()).last().unwrap(), "after-restore");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn atomic_native_replace_without_handoff_cannot_discard_committed_frames() {
|
||||
let t = tempfile::tempdir().unwrap();
|
||||
append(t.path(), "accepted");
|
||||
let p = path(t.path());
|
||||
storage::atomic_write(
|
||||
&p,
|
||||
b"item({name='message';attr={};key='replacement';when=42});\n",
|
||||
)
|
||||
.unwrap();
|
||||
// A replacement without handoff is not authority to discard committed WAL.
|
||||
assert_eq!(ids(t.path()), ["accepted"]);
|
||||
}
|
||||
Vendored
+74
@@ -0,0 +1,74 @@
|
||||
/* Test-only LD_PRELOAD shim: delay, then perform the real durable operation.
|
||||
* Scope is strictly the canonical fixture data root selected by the harness.
|
||||
* No host mount or durability settings are changed. Log contains no file data.
|
||||
* Exceptional XMPP_SYNC_FAIL_SUFFIX=.wal|.rredo injects EIO for matching
|
||||
* fixture regular files. Without that explicit fault mode all syncs are real.
|
||||
*/
|
||||
#define _GNU_SOURCE
|
||||
#include <dlfcn.h>
|
||||
#include <errno.h>
|
||||
#include <fcntl.h>
|
||||
#include <limits.h>
|
||||
#include <stdio.h>
|
||||
#include <stdlib.h>
|
||||
#include <string.h>
|
||||
#include <sys/stat.h>
|
||||
#include <time.h>
|
||||
#include <unistd.h>
|
||||
|
||||
static long long ns(void) {
|
||||
struct timespec t;
|
||||
clock_gettime(CLOCK_MONOTONIC, &t);
|
||||
return (long long)t.tv_sec * 1000000000LL + t.tv_nsec;
|
||||
}
|
||||
static int sync_call(int fd, const char *operation) {
|
||||
int (*real_sync)(int) = dlsym(RTLD_NEXT, operation);
|
||||
if (!real_sync) { errno = ENOSYS; return -1; }
|
||||
const char *prefix = getenv("XMPP_SLOW_SYNC_ROOT");
|
||||
const char *log = getenv("XMPP_SLOW_SYNC_LOG");
|
||||
char link[64], path[PATH_MAX];
|
||||
snprintf(link, sizeof link, "/proc/self/fd/%d", fd);
|
||||
ssize_t length = readlink(link, path, sizeof path - 1);
|
||||
if (!prefix || !log || length < 0) return real_sync(fd);
|
||||
path[length] = '\0';
|
||||
size_t size = strlen(prefix);
|
||||
if (strncmp(path, prefix, size) || (path[size] && path[size] != '/'))
|
||||
return real_sync(fd);
|
||||
struct stat st;
|
||||
if (fstat(fd, &st) || (!S_ISREG(st.st_mode) && !S_ISDIR(st.st_mode)))
|
||||
return real_sync(fd);
|
||||
const char *delay = getenv("XMPP_SLOW_SYNC_MS");
|
||||
long ms = delay ? strtol(delay, NULL, 10) : 100;
|
||||
if (ms < 0 || ms > 10000) ms = 100;
|
||||
long long begin = ns();
|
||||
struct timespec wait = {.tv_sec = ms / 1000, .tv_nsec = (ms % 1000) * 1000000};
|
||||
while (nanosleep(&wait, &wait) && errno == EINTR) {}
|
||||
const char *suffix = getenv("XMPP_SYNC_FAIL_SUFFIX");
|
||||
int injected = 0;
|
||||
if (suffix && (!strcmp(suffix, ".wal") || !strcmp(suffix, ".rredo")) && S_ISREG(st.st_mode)) {
|
||||
size_t suffix_length = strlen(suffix);
|
||||
injected = (size_t)length >= suffix_length &&
|
||||
!strcmp(path + length - suffix_length, suffix);
|
||||
}
|
||||
int result;
|
||||
if (injected) { errno = EIO; result = -1; }
|
||||
else result = real_sync(fd);
|
||||
int saved_errno = errno;
|
||||
long long end = ns();
|
||||
/* One append write per operation avoids partial concurrent log records.
|
||||
* Paths are not logged: only a path-derived store category. */
|
||||
const char *store = strstr(path, "/rust_smacks") ? "sm" :
|
||||
strstr(path, "/archive") ? "mam" :
|
||||
strstr(path, "/pep") ? "pep" : "other";
|
||||
char row[512];
|
||||
int count = snprintf(row, sizeof row,
|
||||
"{\"op\":\"%s\",\"kind\":\"%s\",\"store\":\"%s\",\"begin_ns\":%lld,\"end_ns\":%lld,\"result\":%d,\"injected\":%s}\n",
|
||||
operation, S_ISDIR(st.st_mode) ? "directory" : "file", store, begin, end, result,
|
||||
injected ? "true" : "false");
|
||||
int output = open(log, O_WRONLY | O_CREAT | O_APPEND | O_CLOEXEC, 0600);
|
||||
if (output >= 0) { ssize_t ignored = write(output, row, count); (void)ignored; close(output); }
|
||||
errno = saved_errno;
|
||||
return result;
|
||||
}
|
||||
int fsync(int fd) { return sync_call(fd, "fsync"); }
|
||||
int fdatasync(int fd) { return sync_call(fd, "fdatasync"); }
|
||||
@@ -0,0 +1,199 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Fail recovery journal syncs explicitly; never publish a false durable handoff.
|
||||
|
||||
Uses independent real TLS fixtures for .wal and .rredo so archive native inode
|
||||
identity remains genuine. SIGKILL leaves complete valid journals. The scoped
|
||||
test interposer returns EIO only for the selected suffix beneath fixture data.
|
||||
"""
|
||||
import argparse
|
||||
import asyncio
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import shutil
|
||||
import signal
|
||||
import socket
|
||||
import ssl
|
||||
import subprocess
|
||||
import tempfile
|
||||
import time
|
||||
|
||||
from muc_restart_network import stop
|
||||
from sm_restart_network import durable_slot
|
||||
from slow_storage_latency_network import CARBONS, MeasuredClient, check_shim_scope, iq, sync_rows
|
||||
from slow_storage_crash_network import committed_archive, start
|
||||
|
||||
|
||||
async def seed(root, binary, config, port, log, env):
|
||||
context = ssl.create_default_context(cafile=str(root/'cert.pem'))
|
||||
clients, server = [], None
|
||||
identifier = 'recovery-sync-body'
|
||||
try:
|
||||
server = await asyncio.to_thread(start, binary, config, port, log, env)
|
||||
for number in range(4):
|
||||
client = await MeasuredClient.connect(number, port, context, 2)
|
||||
clients.append(client)
|
||||
client.auto_ack = False
|
||||
await client.enable()
|
||||
await iq(client, f'carbons-{number}', f"<enable xmlns='{CARBONS}'/>")
|
||||
await client.stanza('<presence/>')
|
||||
await client.fence(f'ready-{number}')
|
||||
alice, bob, _, _ = clients
|
||||
await alice.stanza(f"<message to='{bob.jid}' type='chat' id='{identifier}'><body>{identifier}</body></message>")
|
||||
await alice.handled()
|
||||
for client, kind in zip(clients[1:], ('direct', 'sent', 'received')):
|
||||
await client.wait(lambda c=client,k=kind: c.arrivals[(identifier,k)], 'message/carbon before crash')
|
||||
server.send_signal(signal.SIGKILL)
|
||||
assert await asyncio.to_thread(server.wait, 10) == -signal.SIGKILL
|
||||
records = {client.token: durable_slot(root, client.token) for client in clients}
|
||||
assert len(records) == 4
|
||||
for client in clients[1:]:
|
||||
assert sum(identifier in xml for xml in records[client.token]['pending']) == 1
|
||||
archives = {p: committed_archive(p) for p in (root/'data'/'localhost'/'archive').glob('*.list')}
|
||||
assert len(archives) == 2
|
||||
assert all(sum(identifier in row for row in content.splitlines()) == 1 for content in archives.values())
|
||||
return records, archives
|
||||
finally:
|
||||
if server is not None and server.poll() is None:
|
||||
await asyncio.to_thread(stop, server)
|
||||
await asyncio.gather(*(c.close_raw() for c in clients), return_exceptions=True)
|
||||
|
||||
|
||||
def journal_bytes(root, suffix):
|
||||
return {p: p.read_bytes() for p in (root/'data').rglob('*'+suffix)}
|
||||
|
||||
|
||||
def assert_serve_rejects(binary, config, port, log, env):
|
||||
server = subprocess.Popen([str(binary), 'serve', str(config), '--experimental'],
|
||||
env=env, stdout=log, stderr=log)
|
||||
listened = False
|
||||
try:
|
||||
deadline = time.monotonic()+15
|
||||
while server.poll() is None and time.monotonic() < deadline:
|
||||
try:
|
||||
with socket.create_connection(('127.0.0.1', port), timeout=.05):
|
||||
listened = True
|
||||
break
|
||||
except OSError:
|
||||
time.sleep(.01)
|
||||
assert not listened, 'SM recovery sync failure exposed a listener'
|
||||
assert server.wait(timeout=5) != 0, 'SM recovery sync failure reported successful serve'
|
||||
finally:
|
||||
if server.poll() is None:
|
||||
server.kill()
|
||||
server.wait()
|
||||
|
||||
|
||||
def scope_fault(work, shim):
|
||||
root = work/'fault-scope'
|
||||
root.mkdir()
|
||||
code = """import errno,json,os,pathlib,sys
|
||||
root,outside,suffix=pathlib.Path(sys.argv[1]),pathlib.Path(sys.argv[2]),sys.argv[3]
|
||||
statuses=[]
|
||||
for path in (root/('inside'+suffix),root/'unmatched.txt',outside):
|
||||
with path.open('wb') as stream:
|
||||
stream.write(b'fixture');stream.flush()
|
||||
for sync in (os.fsync,os.fdatasync):
|
||||
try:sync(stream.fileno());statuses.append(0)
|
||||
except OSError as error:statuses.append(error.errno)
|
||||
assert statuses==[errno.EIO,errno.EIO,0,0,0,0],statuses
|
||||
"""
|
||||
for suffix in ('.wal', '.rredo'):
|
||||
log = work/('scope-'+suffix[1:]+'.jsonl')
|
||||
env = dict(os.environ, LD_PRELOAD=str(shim), XMPP_SLOW_SYNC_ROOT=str(root),
|
||||
XMPP_SLOW_SYNC_LOG=str(log), XMPP_SLOW_SYNC_MS='0', XMPP_SYNC_FAIL_SUFFIX=suffix)
|
||||
subprocess.run(['python3', '-c', code, str(root), str(work/('outside'+suffix)), suffix],
|
||||
env=env, check=True, capture_output=True)
|
||||
rows = sync_rows(log)
|
||||
assert len(rows) == 4 and sum(row['injected'] for row in rows) == 2
|
||||
|
||||
|
||||
def run_case(work, binary, shim, suffix):
|
||||
root = work/suffix[1:]
|
||||
root.mkdir()
|
||||
subprocess.run(['openssl', 'req', '-x509', '-newkey', 'ec', '-pkeyopt', 'ec_paramgen_curve:P-256',
|
||||
'-nodes', '-days', '1', '-subj', '/CN=localhost', '-addext', 'subjectAltName=DNS:localhost',
|
||||
'-keyout', str(root/'key.pem'), '-out', str(root/'cert.pem')], check=True, capture_output=True)
|
||||
with socket.socket() as probe:
|
||||
probe.bind(('127.0.0.1', 0))
|
||||
port = probe.getsockname()[1]
|
||||
config = root/'config.json'
|
||||
config.write_text(json.dumps({'global': {'data_path': str(root/'data'), 'interfaces': ['127.0.0.1'],
|
||||
'c2s_ports': [port], 's2s_ports': [], 'http_ports': [], 'https_ports': [],
|
||||
'modules_enabled': ['tls', 'saslauth', 'smacks', 'ping', 'mam', 'carbons'],
|
||||
'smacks_hibernation_time': 600, 'archive_expires_after': 'never', 'default_archive_policy': True,
|
||||
'ssl': {'key': str(root/'key.pem'), 'certificate': str(root/'cert.pem')}},
|
||||
'hosts': {'localhost': {'options': {'authentication': 'internal_hashed', 'password_hash': 'SHA-256'}}}}))
|
||||
for number in range(2):
|
||||
subprocess.run([str(binary), 'create-user', str(config), f'load-{number}', '--password-stdin'],
|
||||
input='load-test-password\n', text=True, check=True, capture_output=True)
|
||||
sync_log = root/'sync.jsonl'
|
||||
env = dict(os.environ, LD_PRELOAD=str(shim), XMPP_SLOW_SYNC_ROOT=str(root/'data'),
|
||||
XMPP_SLOW_SYNC_LOG=str(sync_log), XMPP_SLOW_SYNC_MS='0')
|
||||
env.pop('XMPP_SYNC_FAIL_SUFFIX', None)
|
||||
with (root/'server.log').open('w+') as log:
|
||||
records, archives = asyncio.run(seed(root, binary, config, port, log, env))
|
||||
journals = journal_bytes(root, suffix)
|
||||
assert journals, 'network fixture did not leave requested journals'
|
||||
fault_env = dict(env, XMPP_SYNC_FAIL_SUFFIX=suffix)
|
||||
before = len(sync_rows(sync_log))
|
||||
command = [str(binary), 'storage-checkpoint', str(config)]
|
||||
result = subprocess.run(command, env=fault_env, capture_output=True, text=True)
|
||||
assert result.returncode != 0, f'{suffix} recovery skipped journal sync and falsely reported successful handoff'
|
||||
assert 'Durable native archive handoff:' not in result.stdout, 'failed checkpoint advertised handoff success'
|
||||
attempts = [row for row in sync_rows(sync_log)[before:] if row.get('injected')]
|
||||
assert attempts and all(row['result'] == -1 for row in attempts), 'recovery did not attempt faulted journal sync'
|
||||
assert journal_bytes(root, suffix) == journals, 'failed checkpoint removed or changed authoritative journal'
|
||||
if suffix == '.wal':
|
||||
before_serve = len(sync_rows(sync_log))
|
||||
assert_serve_rejects(binary, config, port, log, fault_env)
|
||||
assert any(row.get('injected') and row['store'] == 'sm'
|
||||
for row in sync_rows(sync_log)[before_serve:]), 'serve failed without attempting SM recovery sync'
|
||||
assert journal_bytes(root, suffix) == journals, 'failed serve changed authoritative SM journals'
|
||||
subprocess.run(command, env=env, check=True, capture_output=True, text=True)
|
||||
assert not journal_bytes(root, '.wal') and not journal_bytes(root, '.rredo'), 'successful handoff left journals'
|
||||
for token, record in records.items():
|
||||
paths = list((root/'data').rglob(token+'.json'))
|
||||
assert len(paths) == 1
|
||||
exported = json.loads(paths[0].read_text())
|
||||
assert {k:v for k,v in exported.items() if k != 'deadline_ms'} == {
|
||||
k:v for k,v in record.items() if k != 'deadline_ms'}, 'handoff changed acknowledged SM state'
|
||||
assert all(path.read_text() == content for path, content in archives.items()), 'handoff lost acknowledged MAM body'
|
||||
return {'suffix': suffix, 'status': 'pass', 'checkpoint_rejected_sync_failure': True,
|
||||
'journals_preserved_on_failure': len(journals),
|
||||
'serve_rejected_before_listeners': suffix == '.wal',
|
||||
'clean_retry_preserved_sessions': len(records), 'clean_retry_preserved_archives': len(archives)}
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('--binary', default='target/release/rust-xmpp')
|
||||
parser.add_argument('--output', default='/tmp/journal-recovery-sync.json')
|
||||
parser.add_argument('--suffix', nargs='+', choices=['.wal', '.rredo'], default=['.wal', '.rredo'])
|
||||
args = parser.parse_args()
|
||||
report = {'status': 'running', 'cases': []}
|
||||
try:
|
||||
with tempfile.TemporaryDirectory(prefix='xmpp-journal-recovery-sync-') as directory:
|
||||
work = Path(directory)
|
||||
binary = work/'rust-xmpp'
|
||||
shutil.copy2(Path(args.binary).resolve(), binary)
|
||||
report['binary_sha256'] = hashlib.sha256(binary.read_bytes()).hexdigest()
|
||||
shim = work/'slow_sync.so'
|
||||
subprocess.run(['cc', '-shared', '-fPIC', '-O2', '-Wall', '-Wextra', '-Werror',
|
||||
str(Path(__file__).parent/'fixtures'/'slow_sync.c'), '-o', str(shim), '-ldl'], check=True)
|
||||
check_shim_scope(work, shim)
|
||||
scope_fault(work, shim)
|
||||
for suffix in args.suffix:
|
||||
report['cases'].append(run_case(work, binary, shim, suffix))
|
||||
report['status'] = 'pass'
|
||||
except BaseException as error:
|
||||
report.update(status='fail', failure=str(error))
|
||||
raise
|
||||
finally:
|
||||
Path(args.output).write_text(json.dumps(report, indent=2)+'\n')
|
||||
print(json.dumps(report, indent=2))
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -25,6 +25,8 @@ fn actual_prosody_muc_mam_reads_rust_and_writes_native_archive() {
|
||||
"<message type='groupchat' to='room@rooms.example.org' id='rust-message'><body>Rust archived body</body></message>",
|
||||
);
|
||||
drop(muc);
|
||||
// Transfer archive write ownership before native Prosody appends in place.
|
||||
assert_eq!(rust_xmpp::archive::checkpoint_all(root.path()).unwrap(), 1);
|
||||
let bootstrap = std::fs::read_to_string(concat!(
|
||||
env!("CARGO_MANIFEST_DIR"),
|
||||
"/scripts/prosody_compat/oracle.lua"
|
||||
|
||||
@@ -47,6 +47,8 @@ fn real_prosody_storage_and_auth_roundtrip() {
|
||||
for (i, id) in ["rust-one", "rust-two"].iter().enumerate() {
|
||||
archive::append_new(root,host,Some("alice"),"archive",archive::Entry{id:id.to_string(),when:1767225600.+i as f64,peer:"bob@example.org".into(),xml:"<message xmlns='jabber:client' xml:lang='ru' xmlns:o='urn:oracle' o:flag='yes'><body>Rust body</body></message>".into()}).unwrap();
|
||||
}
|
||||
// Transfer write ownership only after durable native checkpoint.
|
||||
archive::checkpoint_all(root).unwrap();
|
||||
oracle(root, "roundtrip", hash);
|
||||
let v = storage::load(root, host, Some("alice"), "oracle")
|
||||
.unwrap()
|
||||
@@ -97,6 +99,7 @@ fn real_prosody_storage_and_auth_roundtrip() {
|
||||
)
|
||||
.unwrap();
|
||||
assert!(!p.exists());
|
||||
archive::checkpoint_all(root).unwrap();
|
||||
oracle(root, "verify-final", hash);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,287 @@
|
||||
#!/usr/bin/env python3
|
||||
"""SIGKILL after sender SM acknowledgement under real delayed durable IO.
|
||||
|
||||
Four STARTTLS resources/two accounts, MAM, carbons and unacknowledged avatar
|
||||
PEP. Inspect authoritative recipient journals and both archives before any
|
||||
reauthentication; then resume all resources and verify replay and counters.
|
||||
Uses the latency fixture's fsync/fdatasync interposer, retaining real syncs.
|
||||
"""
|
||||
import argparse
|
||||
import asyncio
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import shutil
|
||||
import signal
|
||||
import socket
|
||||
import ssl
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
|
||||
from muc_sm_load_network import SM
|
||||
from muc_restart_network import stop
|
||||
from sm_restart_network import durable_slot
|
||||
from slow_storage_latency_network import (
|
||||
CARBONS, PUBSUB, MeasuredClient, check_shim_scope, counts, iq, sync_rows,
|
||||
)
|
||||
|
||||
|
||||
def start(binary, config, port, log, env):
|
||||
server = subprocess.Popen([str(binary), 'serve', str(config), '--experimental'],
|
||||
env=env, stdout=log, stderr=log)
|
||||
deadline = time.monotonic()+60
|
||||
while time.monotonic() < deadline:
|
||||
assert server.poll() is None, 'server stopped during startup'
|
||||
try:
|
||||
with socket.create_connection(('127.0.0.1', port), timeout=.1):
|
||||
return server
|
||||
except OSError:
|
||||
time.sleep(.05)
|
||||
server.kill()
|
||||
server.wait()
|
||||
raise AssertionError('server startup timeout')
|
||||
|
||||
|
||||
def committed_archive(path):
|
||||
"""Reconstruct native MAM bytes using its authoritative checksummed redo."""
|
||||
native = path.read_bytes()
|
||||
data = path.with_suffix('.rredo').read_bytes()
|
||||
assert len(data) >= 96 and data[:8] == b'XARRED01', 'invalid MAM redo header'
|
||||
assert hashlib.sha256(data[:64]).digest() == data[64:96], 'invalid MAM redo header checksum'
|
||||
baseline = int.from_bytes(data[8:16], 'little')
|
||||
assert baseline <= len(native), 'missing MAM native baseline'
|
||||
committed, offset = native[:baseline], 96
|
||||
while offset < len(data):
|
||||
if len(data)-offset < 72:
|
||||
break
|
||||
size = int.from_bytes(data[offset:offset+8], 'little')
|
||||
assert hashlib.sha256(data[offset:offset+8]).digest() == data[offset+8:offset+40], 'invalid MAM redo length checksum'
|
||||
assert 0 < size <= 16*1024*1024, 'invalid MAM redo frame size'
|
||||
end = offset+72+size
|
||||
if end > len(data):
|
||||
break
|
||||
payload = data[offset+72:end]
|
||||
assert hashlib.sha256(payload).digest() == data[offset+40:offset+72], 'invalid MAM redo frame checksum'
|
||||
committed += payload
|
||||
offset = end
|
||||
# Sender handled acknowledgement must not depend on a native suffix that
|
||||
# lacks a complete, durable redo frame.
|
||||
assert native == committed, 'native MAM suffix is not committed in redo'
|
||||
return committed.decode()
|
||||
|
||||
|
||||
def assert_committed(root, expected, identifiers, avatar):
|
||||
snapshots = [durable_slot(root, old['token']) for old in expected]
|
||||
for index, (record, old) in enumerate(zip(snapshots, expected)):
|
||||
assert record['jid'] == old['jid'], (index, record)
|
||||
assert record['handled'] == old['handled'], (index, record['handled'], old)
|
||||
assert record['acked'] == old['acked'], (index, record['acked'], old)
|
||||
assert record['sent']-record['acked'] == len(record['pending']), (index, record)
|
||||
if index in (1, 2, 3):
|
||||
for identifier in identifiers:
|
||||
assert sum(identifier in xml for xml in record['pending']) == 1, (index, identifier, record)
|
||||
if index in (0, 2):
|
||||
assert any(avatar in xml and 'event' in xml and 'crash-avatar' in xml
|
||||
for xml in record['pending']), ('PEP missing before reauthentication', index, record)
|
||||
paths = sorted((root/'data'/'localhost'/'archive').glob('*.list'))
|
||||
assert len(paths) == 2, paths
|
||||
for path in paths:
|
||||
rows = [row for row in committed_archive(path).splitlines() if row.startswith('item(')]
|
||||
assert len(rows) == len(identifiers), (path.name, len(rows), identifiers)
|
||||
for identifier in identifiers:
|
||||
assert sum(identifier in row for row in rows) == 1, ('acknowledged body missing/duplicated in MAM', path.name, identifier)
|
||||
return snapshots
|
||||
|
||||
|
||||
async def exercise(root, binary, config, port, log, env, sync_log, args):
|
||||
context = ssl.create_default_context(cafile=str(root/'cert.pem'))
|
||||
clients, server = [], None
|
||||
avatar = 'urn:xmpp:avatar:metadata'
|
||||
try:
|
||||
server = await asyncio.to_thread(start, binary, config, port, log, env)
|
||||
for number in range(4):
|
||||
client = await MeasuredClient.connect(number, port, context, 2)
|
||||
clients.append(client)
|
||||
client.auto_ack = False
|
||||
await client.enable()
|
||||
await iq(client, f'carbons-{number}', f"<enable xmlns='{CARBONS}'/>")
|
||||
await client.stanza('<presence/>')
|
||||
await client.fence(f'ready-{number}')
|
||||
alice, bob, alice_other, _ = clients
|
||||
await iq(alice, 'avatar-initial', f"<pubsub xmlns='{PUBSUB}'><publish node='{avatar}'><item id='initial'><metadata xmlns='{avatar}'/></item></publish></pubsub>")
|
||||
for client in (alice, alice_other):
|
||||
await iq(client, f'subscribe-{client.number}', f"<pubsub xmlns='{PUBSUB}'><subscribe node='{avatar}' jid='{client.jid}'/></pubsub>")
|
||||
# Ack every setup stanza, then fence acknowledgement persistence. No
|
||||
# resource sends another ACK until after the crash replay assertions.
|
||||
for client in clients:
|
||||
await client.fence(f'setup-{client.number}')
|
||||
await client.ack()
|
||||
await client.handled()
|
||||
before = len(sync_rows(sync_log))
|
||||
await iq(alice, 'avatar-pending', f"<pubsub xmlns='{PUBSUB}'><publish node='{avatar}'><item id='crash-avatar'><metadata xmlns='{avatar}'><info id='fixture-avatar' bytes='1' type='image/png'/></metadata></item></publish></pubsub>")
|
||||
identifiers = [f'crash-body-{number}' for number in range(args.messages)]
|
||||
for identifier in identifiers:
|
||||
await alice.stanza(f"<message to='{bob.jid}' type='chat' id='{identifier}'><body>{identifier}</body></message>")
|
||||
await alice.handled()
|
||||
expected = [{'token': c.token, 'jid': c.jid, 'acked': c.acked, 'handled': c.inputs}
|
||||
for c in clients]
|
||||
# No receiver fence/delivery wait between sender's final <a/> and kill:
|
||||
# the acknowledged sender operation itself must make the data durable.
|
||||
server.send_signal(signal.SIGKILL)
|
||||
assert await asyncio.to_thread(server.wait, 10) == -signal.SIGKILL
|
||||
snapshots = assert_committed(root, expected, identifiers, avatar)
|
||||
rows = sync_rows(sync_log)[before:]
|
||||
summary = counts(rows)
|
||||
assert rows and all(row['end_ns']-row['begin_ns'] >= args.delay_ms*1000000 for row in rows)
|
||||
assert {'sm', 'mam', 'pep'}.issubset({row['store'] for row in rows}), summary
|
||||
assert any(row['kind'] == 'directory' for row in rows), summary
|
||||
await asyncio.gather(*(c.close_raw() for c in clients), return_exceptions=True)
|
||||
clients.clear()
|
||||
server = await asyncio.to_thread(start, binary, config, port, log, env)
|
||||
# Startup must recover the same queues before a single SASL login.
|
||||
recovered = assert_committed(root, expected, identifiers, avatar)
|
||||
for old, new in zip(snapshots, recovered):
|
||||
assert old['pending'] == new['pending']
|
||||
for number, old in enumerate(expected):
|
||||
client = await MeasuredClient.connect(number, port, context, 2, bind=False)
|
||||
clients.append(client)
|
||||
client.auto_ack = False
|
||||
client.received = client.acked = old['acked']
|
||||
client.inputs, client.token = old['handled'], old['token']
|
||||
await client.send(f"<resume xmlns='{SM}' previd='{client.token}' h='{client.acked}'/>")
|
||||
result = await client.control('resumed')
|
||||
assert result.get('previd') == client.token
|
||||
assert int(result.get('h')) == old['handled'], (number, result.attrib, old)
|
||||
kind = {1: 'direct', 2: 'sent', 3: 'received'}.get(number)
|
||||
if kind:
|
||||
for identifier in identifiers:
|
||||
await client.wait(lambda c=client,i=identifier,k=kind: c.arrivals[(i,k)], 'durable replay '+identifier)
|
||||
assert len(client.arrivals[(identifier,kind)]) == 1, (number, identifier, client.arrivals)
|
||||
await client.fence(f'after-resume-{number}')
|
||||
# The fence follows every replay, so the exact output counter also
|
||||
# detects missing/extra PEP, carbon or IQ replay, beyond body counts.
|
||||
assert client.received == snapshots[number]['sent']+1, (number, client.received, snapshots[number]['sent'])
|
||||
await client.ack()
|
||||
await client.handled()
|
||||
record = durable_slot(root, client.token)
|
||||
assert record['acked'] == client.received and not record['pending'], (number, record)
|
||||
assert not client.errors, client.errors
|
||||
# Native handoff is explicitly offline: release the server's data/store
|
||||
# locks before asking the CLI to durably export and remove journals.
|
||||
await asyncio.to_thread(stop, server)
|
||||
server = None
|
||||
final_slots = {c.token: durable_slot(root, c.token) for c in clients}
|
||||
final_archives = {p: committed_archive(p) for p in
|
||||
(root/'data'/'localhost'/'archive').glob('*.list')}
|
||||
checkpoint = [str(binary), 'storage-checkpoint', str(config)]
|
||||
await asyncio.to_thread(subprocess.run, checkpoint, env=env, check=True, capture_output=True, text=True)
|
||||
assert not list((root/'data').rglob('*.wal')), 'SM WAL survived offline checkpoint'
|
||||
assert not list((root/'data').rglob('*.rredo')), 'MAM redo survived offline checkpoint'
|
||||
exported = {}
|
||||
for token, authoritative in final_slots.items():
|
||||
paths = list((root/'data').rglob(token+'.json'))
|
||||
assert len(paths) == 1, ('checkpoint lost SM export', token)
|
||||
exported[token] = json.loads(paths[0].read_text())
|
||||
previous = dict(authoritative)
|
||||
current = dict(exported[token])
|
||||
previous.pop('deadline_ms')
|
||||
current.pop('deadline_ms')
|
||||
assert current == previous, ('checkpoint changed authoritative SM fields', token, current, previous)
|
||||
assert all(p.read_text() == content for p, content in final_archives.items()), 'checkpoint changed committed MAM records'
|
||||
await asyncio.to_thread(subprocess.run, checkpoint, env=env, check=True, capture_output=True, text=True)
|
||||
assert not list((root/'data').rglob('*.wal')) and not list((root/'data').rglob('*.rredo'))
|
||||
for token, previous in exported.items():
|
||||
paths = list((root/'data').rglob(token+'.json'))
|
||||
assert len(paths) == 1 and json.loads(paths[0].read_text()) == previous, 'second checkpoint changed SM export'
|
||||
assert all(p.read_text() == content for p, content in final_archives.items()), 'second checkpoint changed MAM records'
|
||||
return {'status': 'pass', 'delay_ms_per_sync': args.delay_ms,
|
||||
'acknowledged_messages': len(identifiers), 'mam_records_per_account': [len(identifiers)]*2,
|
||||
'resumed_resources': len(clients), 'replayed_personal_deliveries': 3*len(identifiers),
|
||||
'offline_checkpoint': {'exported_sessions': len(exported), 'native_archives': len(final_archives),
|
||||
'journals_removed': True, 'second_invocation_idempotent': True},
|
||||
'pending_before_reauthentication': [len(record['pending']) for record in snapshots],
|
||||
'sync_between_publication_and_sigkill': summary}
|
||||
finally:
|
||||
if server is not None and server.poll() is None:
|
||||
failing = sys.exc_info()[0] is not None
|
||||
try:
|
||||
await asyncio.to_thread(stop, server)
|
||||
except BaseException:
|
||||
if not failing:
|
||||
raise
|
||||
await asyncio.gather(*(c.close_raw() for c in clients), return_exceptions=True)
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('--binary', default='target/release/rust-xmpp')
|
||||
parser.add_argument('--delay-ms', type=int, default=100)
|
||||
parser.add_argument('--messages', type=int, default=3)
|
||||
parser.add_argument('--output', default='/tmp/slow-storage-crash.json')
|
||||
args = parser.parse_args()
|
||||
assert 0 <= args.delay_ms <= 10000 and args.messages > 0
|
||||
with tempfile.TemporaryDirectory(prefix='xmpp-slow-storage-crash-') as directory:
|
||||
root = Path(directory)
|
||||
binary = root/'rust-xmpp'
|
||||
shutil.copy2(Path(args.binary).resolve(), binary)
|
||||
shim = root/'slow_sync.so'
|
||||
subprocess.run(['cc', '-shared', '-fPIC', '-O2', '-Wall', '-Wextra', '-Werror',
|
||||
str(Path(__file__).parent/'fixtures'/'slow_sync.c'), '-o', str(shim), '-ldl'], check=True)
|
||||
check_shim_scope(root, shim)
|
||||
subprocess.run(['openssl', 'req', '-x509', '-newkey', 'ec', '-pkeyopt', 'ec_paramgen_curve:P-256',
|
||||
'-nodes', '-days', '1', '-subj', '/CN=localhost', '-addext', 'subjectAltName=DNS:localhost',
|
||||
'-keyout', str(root/'key.pem'), '-out', str(root/'cert.pem')], check=True, capture_output=True)
|
||||
with socket.socket() as probe:
|
||||
probe.bind(('127.0.0.1', 0))
|
||||
port = probe.getsockname()[1]
|
||||
config = root/'config.json'
|
||||
config.write_text(json.dumps({'global': {'data_path': str(root/'data'), 'interfaces': ['127.0.0.1'],
|
||||
'c2s_ports': [port], 's2s_ports': [], 'http_ports': [], 'https_ports': [],
|
||||
'smacks_hibernation_time': 600,
|
||||
'modules_enabled': ['tls', 'saslauth', 'smacks', 'csi_simple', 'ping', 'mam', 'pep', 'carbons'],
|
||||
'archive_expires_after': 'never', 'default_archive_policy': True,
|
||||
'ssl': {'key': str(root/'key.pem'), 'certificate': str(root/'cert.pem')}},
|
||||
'hosts': {'localhost': {'options': {'authentication': 'internal_hashed', 'password_hash': 'SHA-256'}}}}))
|
||||
for number in range(2):
|
||||
subprocess.run([str(binary), 'create-user', str(config), f'load-{number}', '--password-stdin'],
|
||||
input='load-test-password\n', text=True, check=True, capture_output=True)
|
||||
sync_log = root/'sync.jsonl'
|
||||
env = dict(os.environ, LD_PRELOAD=str(shim), XMPP_SLOW_SYNC_ROOT=str(root/'data'),
|
||||
XMPP_SLOW_SYNC_LOG=str(sync_log), XMPP_SLOW_SYNC_MS=str(args.delay_ms))
|
||||
with (root/'server.log').open('w+') as log:
|
||||
try:
|
||||
report = asyncio.run(exercise(root, binary, config, port, log, env, sync_log, args))
|
||||
except BaseException as error:
|
||||
# Preserve the first failing run's evidence outside the
|
||||
# TemporaryDirectory before cleanup; never replace its error
|
||||
# with a later shutdown or diagnostics failure.
|
||||
try:
|
||||
log.flush()
|
||||
output = Path(args.output)
|
||||
output.parent.mkdir(parents=True, exist_ok=True)
|
||||
evidence = Path(tempfile.mkdtemp(prefix=output.stem+'-failure-', dir=output.parent))
|
||||
shutil.copy2(root/'server.log', evidence/'server.log')
|
||||
if sync_log.exists():
|
||||
shutil.copy2(sync_log, evidence/'sync.jsonl')
|
||||
Path(args.output).write_text(json.dumps({'status': 'fail',
|
||||
'failure': repr(error), 'delay_ms_per_sync': args.delay_ms,
|
||||
'binary_sha256': hashlib.sha256(binary.read_bytes()).hexdigest(),
|
||||
'evidence': str(evidence)}, indent=2)+'\n')
|
||||
except Exception as diagnostic_error:
|
||||
print(f'Could not preserve failure evidence: {diagnostic_error}', file=sys.stderr)
|
||||
log.seek(0)
|
||||
print(log.read(), file=sys.stderr)
|
||||
raise
|
||||
report['binary_sha256'] = hashlib.sha256(binary.read_bytes()).hexdigest()
|
||||
report['scope'] = 'loopback STARTTLS; real delayed fsync/fdatasync; sender SM acknowledgement then SIGKILL; authoritative queues and MAM checked before reauthentication; at-least-once replay'
|
||||
result = json.dumps(report, indent=2)+'\n'
|
||||
if args.output:
|
||||
Path(args.output).write_text(result)
|
||||
print(result, end='')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,299 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Fixture-only slow durable IO: two accounts, four TLS/SM resources, MAM/PEP.
|
||||
|
||||
Builds only a tiny C LD_PRELOAD interposer; uses an isolated copy of --binary.
|
||||
Every intercepted fsync/fdatasync still executes. Directory syncs under the
|
||||
fixture data root are delayed too and counted separately. No mount changes.
|
||||
Timing includes durable work, with client ACKs withheld during each sample.
|
||||
"""
|
||||
import argparse
|
||||
import asyncio
|
||||
import collections
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import pathlib
|
||||
import re
|
||||
import shutil
|
||||
import socket
|
||||
import ssl
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
import xml.etree.ElementTree as ET
|
||||
|
||||
from muc_sm_load_network import Client, CLIENT, SM
|
||||
from muc_restart_network import stop
|
||||
from message_latency_network import slot
|
||||
|
||||
CARBONS = 'urn:xmpp:carbons:2'
|
||||
PUBSUB = 'http://jabber.org/protocol/pubsub'
|
||||
|
||||
|
||||
PHASE_FIELDS = ('pre_routing_ms', 'sender_mam_ms', 'recipient_mam_ms',
|
||||
'delivery_ms', 'carbons_ms', 'offline_push_ms', 'handled_ms')
|
||||
SLOW_FIELDS = {'duration_ms', 'route', 'target_count', 'carbon_count', *PHASE_FIELDS}
|
||||
|
||||
|
||||
def slow_events(path, delay, rounds, allow_legacy=False):
|
||||
events = []
|
||||
for line in path.read_text().splitlines():
|
||||
if 'event="message-processing-slow"' not in line:
|
||||
continue
|
||||
fields = dict(re.findall(r'(?:^|\s)"([^"\s]+)"="([^"\n]*)"', line))
|
||||
legacy = allow_legacy and set(fields) == {'duration_ms', 'route'}
|
||||
assert legacy or set(fields) == SLOW_FIELDS, ('unexpected slow-event fields', sorted(fields))
|
||||
assert not any(value in line for value in
|
||||
('slow fixture ', 'body-settled-', 'body-avatar_pending-', '<body')), 'payload leaked into slow diagnostics'
|
||||
event = {key: value if key == 'route' else int(value) for key,value in fields.items()}
|
||||
assert event['duration_ms'] >= 250 and event['route'] == 'local', event
|
||||
if not legacy:
|
||||
assert all(event[key] >= 0 for key in PHASE_FIELDS), event
|
||||
assert sum(event[key] for key in PHASE_FIELDS) <= event['duration_ms']+len(PHASE_FIELDS), event
|
||||
assert event['target_count'] == 1 and event['carbon_count'] == 2, event
|
||||
events.append(event)
|
||||
# At >=100ms the retained durable commits alone exceed the 250ms threshold.
|
||||
# Smaller optional delays may legitimately produce no warning.
|
||||
if delay >= 100:
|
||||
assert len(events) == 2*rounds, ('missing/extra slow diagnostics',len(events),2*rounds)
|
||||
return events
|
||||
|
||||
|
||||
class WireReader:
|
||||
def __init__(self, reader, owner):
|
||||
self.reader, self.owner = reader, owner
|
||||
self.parser = ET.XMLPullParser(events=('start', 'end'))
|
||||
self.parser.feed("<root xmlns='jabber:client'>")
|
||||
self.root = next(self.parser.read_events())[1]
|
||||
self.depth = 0
|
||||
|
||||
def __getattr__(self, name):
|
||||
return getattr(self.reader, name)
|
||||
|
||||
async def read(self, size):
|
||||
data = await self.reader.read(size)
|
||||
self.parser.feed(data)
|
||||
for event, node in self.parser.read_events():
|
||||
self.depth += 1 if event == 'start' else -1
|
||||
if event != 'end' or self.depth:
|
||||
continue
|
||||
now = time.monotonic_ns()
|
||||
if node.tag == f'{{{CLIENT}}}message':
|
||||
for message in node.iter(f'{{{CLIENT}}}message'):
|
||||
if message.get('id'):
|
||||
kind = 'direct'
|
||||
for carbon in ('sent', 'received'):
|
||||
if node.find(f'{{{CARBONS}}}{carbon}') is not None:
|
||||
kind = carbon
|
||||
self.owner.arrivals[(message.get('id'), kind)].append(now)
|
||||
if node.tag == f'{{{SM}}}a':
|
||||
self.owner.ack_times[int(node.get('h'))] = now
|
||||
self.root.remove(node)
|
||||
return data
|
||||
|
||||
|
||||
class MeasuredClient(Client):
|
||||
def __init__(self, *args):
|
||||
super().__init__(*args)
|
||||
self.arrivals = collections.defaultdict(list)
|
||||
self.ack_times = {}
|
||||
self.reader = WireReader(self.reader, self)
|
||||
|
||||
|
||||
async def iq(client, identifier, payload):
|
||||
await client.stanza(f"<iq type='set' id='{identifier}'>{payload}</iq>")
|
||||
result = await client.wait(lambda: client.iqs.pop(identifier, None), identifier)
|
||||
assert result.get('type') == 'result', ET.tostring(result)
|
||||
|
||||
|
||||
def sync_rows(path):
|
||||
return [json.loads(row) for row in path.read_text().splitlines()] if path.exists() else []
|
||||
|
||||
|
||||
def counts(rows):
|
||||
result = collections.Counter(f"{row['store']}:{row['kind']}:{row['op']}" for row in rows)
|
||||
assert all(row['result'] == 0 for row in rows), 'real sync failed'
|
||||
return {'total': len(rows), 'by_store_kind_op': dict(sorted(result.items()))}
|
||||
|
||||
|
||||
async def exercise(root, port, args, log):
|
||||
context = ssl.create_default_context(cafile=str(root/'cert.pem'))
|
||||
clients = []
|
||||
report = []
|
||||
try:
|
||||
for number in range(4):
|
||||
client = await MeasuredClient.connect(number, port, context, 2)
|
||||
clients.append(client)
|
||||
client.auto_ack = False
|
||||
await client.enable()
|
||||
await iq(client, f'carbons-{number}', f"<enable xmlns='{CARBONS}'/>")
|
||||
await client.stanza('<presence/>')
|
||||
await client.fence(f'ready-{number}')
|
||||
alice, bob, alice_other, bob_other = clients
|
||||
avatar = 'urn:xmpp:avatar:metadata'
|
||||
# Self-account subscriptions make avatar events reach sibling resources.
|
||||
await iq(alice, 'avatar-initial', f"<pubsub xmlns='{PUBSUB}'><publish node='{avatar}'><item id='initial'><metadata xmlns='{avatar}'/></item></publish></pubsub>")
|
||||
for client in (alice, alice_other):
|
||||
await iq(client, f'subscribe-{client.number}', f"<pubsub xmlns='{PUBSUB}'><subscribe node='{avatar}' jid='{client.jid}'/></pubsub>")
|
||||
for phase in ('settled', 'avatar_pending'):
|
||||
for turn in range(args.rounds):
|
||||
# Fence all previous ACK work before taking operation counts.
|
||||
for client in clients:
|
||||
await client.ack()
|
||||
await client.fence(f'before-{phase}-{turn}-{client.number}')
|
||||
await client.handled()
|
||||
before = len(sync_rows(log))
|
||||
start_ns = time.monotonic_ns()
|
||||
if phase == 'avatar_pending':
|
||||
# Publication remains ahead of the body on this stream; no
|
||||
# client ACK is sent for its IQ result or PEP notifications.
|
||||
await alice.stanza(f"<iq type='set' id='publish-{turn}'><pubsub xmlns='{PUBSUB}'><publish node='{avatar}'><item id='sample-{turn}'><metadata xmlns='{avatar}'><info id='fixture-avatar' bytes='1' type='image/png'/></metadata></item></publish></pubsub></iq>")
|
||||
identifier = f'body-{phase}-{turn}'
|
||||
begun = time.monotonic_ns()
|
||||
await alice.stanza(f"<message to='{bob.jid}' type='chat' id='{identifier}'><body>slow fixture {identifier}</body></message>")
|
||||
await alice.handled()
|
||||
events = []
|
||||
for client, kind in ((bob, 'direct'), (alice_other, 'sent'), (bob_other, 'received')):
|
||||
await client.wait(lambda c=client,k=kind: c.arrivals[(identifier,k)], identifier+' '+kind)
|
||||
arrivals = client.arrivals[(identifier, kind)]
|
||||
assert len(arrivals) == 1, (identifier, kind, arrivals)
|
||||
events.append((kind, (arrivals[0]-begun)/1e6))
|
||||
end_ns = time.monotonic_ns()
|
||||
rows = sync_rows(log)[before:]
|
||||
assert rows, 'interposer recorded no data-root sync operations'
|
||||
assert all(row['end_ns']-row['begin_ns'] >= args.current_delay_ms*1000000
|
||||
for row in rows), 'configured sync delay was not applied'
|
||||
snapshots = [slot(root,c) for c in clients]
|
||||
for index in (1,2,3):
|
||||
assert sum(identifier in xml for xml in snapshots[index]['pending']) == 1, (identifier,index)
|
||||
if phase == 'avatar_pending':
|
||||
assert all(any(avatar in xml and 'event' in xml for xml in snapshots[index]['pending'])
|
||||
for index in (0,2)), 'avatar PEP notifications were not durably pending'
|
||||
report.append({'phase': phase, 'round': turn,
|
||||
'delivery_ms': dict(events),
|
||||
'sender_sm_handled_ms': (alice.ack_times[alice.inputs]-begun)/1e6,
|
||||
'publication_plus_delivery_ms': (end_ns-start_ns)/1e6,
|
||||
'sync': counts(rows),
|
||||
'pending_per_resource': [len(snapshot['pending']) for snapshot in snapshots]})
|
||||
assert all(not client.errors for client in clients), [c.errors for c in clients]
|
||||
# Both ends archive each personal body exactly once, despite carbons.
|
||||
archive_counts = [sum(line.startswith('item(') for line in path.open())
|
||||
for path in sorted((root/'data'/'localhost'/'archive').glob('*.list'))]
|
||||
expected = args.history_records + 2*args.rounds
|
||||
assert archive_counts == [expected, expected], (archive_counts, expected)
|
||||
return {'samples': report, 'mam_records_per_account': archive_counts,
|
||||
'status': 'pass'}
|
||||
finally:
|
||||
await asyncio.gather(*(c.close_raw() for c in clients), return_exceptions=True)
|
||||
|
||||
|
||||
def check_shim_scope(work, shim):
|
||||
"""Prove that both sync APIs preserve success and exclude non-data files."""
|
||||
root = work/'shim-scope-data'
|
||||
root.mkdir()
|
||||
log = work/'shim-scope.jsonl'
|
||||
code = """import os,pathlib,sys
|
||||
root,outside=map(pathlib.Path,sys.argv[1:])
|
||||
for path in (root/'inside',outside):
|
||||
with path.open('wb') as stream:
|
||||
stream.write(b'fixture')
|
||||
stream.flush()
|
||||
os.fsync(stream.fileno())
|
||||
os.fdatasync(stream.fileno())
|
||||
fd=os.open(root,os.O_RDONLY)
|
||||
os.fsync(fd)
|
||||
os.close(fd)
|
||||
"""
|
||||
env = dict(os.environ, LD_PRELOAD=str(shim), XMPP_SLOW_SYNC_ROOT=str(root),
|
||||
XMPP_SLOW_SYNC_LOG=str(log), XMPP_SLOW_SYNC_MS='0')
|
||||
subprocess.run([sys.executable,'-c',code,str(root),str(work/'outside')],
|
||||
env=env,check=True,capture_output=True)
|
||||
rows = sync_rows(log)
|
||||
assert len(rows) == 3, rows
|
||||
assert collections.Counter((r['op'],r['kind']) for r in rows) == {
|
||||
('fsync','file'):1, ('fdatasync','file'):1, ('fsync','directory'):1}
|
||||
assert all(row['result'] == 0 for row in rows)
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('--binary', default='target/release/rust-xmpp')
|
||||
parser.add_argument('--delay-ms', type=int, nargs='+', default=[0, 100])
|
||||
parser.add_argument('--rounds', type=int, default=3)
|
||||
parser.add_argument('--history-records', type=int, default=100)
|
||||
parser.add_argument('--output')
|
||||
parser.add_argument('--allow-legacy-phase-logs', action='store_true',
|
||||
help='baseline binaries only: permit old duration/route diagnostics without phase metrics')
|
||||
args = parser.parse_args()
|
||||
assert args.rounds > 0 and args.history_records >= 0
|
||||
assert all(0 <= delay <= 10000 for delay in args.delay_ms)
|
||||
with tempfile.TemporaryDirectory(prefix='xmpp-slow-storage-') as directory:
|
||||
work = pathlib.Path(directory)
|
||||
binary = work/'rust-xmpp'
|
||||
shutil.copy2(pathlib.Path(args.binary).resolve(), binary)
|
||||
shim = work/'slow_sync.so'
|
||||
subprocess.run(['cc', '-shared', '-fPIC', '-O2', '-Wall', '-Wextra', '-Werror',
|
||||
str(pathlib.Path(__file__).parent/'fixtures'/'slow_sync.c'), '-o', str(shim), '-ldl'], check=True)
|
||||
check_shim_scope(work, shim)
|
||||
report = {'binary_sha256': hashlib.sha256(binary.read_bytes()).hexdigest(),
|
||||
'scope': 'loopback STARTTLS; four SM resources/two accounts; MAM + pending avatar metadata PEP; fixture-only fsync/fdatasync including data directories; real sync retained; operation windows include background SM heartbeat',
|
||||
'cases': []}
|
||||
for delay in args.delay_ms:
|
||||
root = work/f'delay-{delay}'
|
||||
root.mkdir()
|
||||
subprocess.run(['openssl', 'req', '-x509', '-newkey', 'ec', '-pkeyopt', 'ec_paramgen_curve:P-256',
|
||||
'-nodes', '-days', '1', '-subj', '/CN=localhost', '-addext', 'subjectAltName=DNS:localhost',
|
||||
'-keyout', str(root/'key.pem'), '-out', str(root/'cert.pem')], check=True, capture_output=True)
|
||||
with socket.socket() as probe:
|
||||
probe.bind(('127.0.0.1',0))
|
||||
port = probe.getsockname()[1]
|
||||
config = root/'config.json'
|
||||
config.write_text(json.dumps({'global': {'data_path': str(root/'data'), 'interfaces':['127.0.0.1'],
|
||||
'c2s_ports':[port], 's2s_ports':[], 'http_ports':[], 'https_ports':[],
|
||||
'modules_enabled':['tls','saslauth','smacks','csi_simple','ping','mam','pep','carbons'],
|
||||
'archive_expires_after':'never', 'default_archive_policy':True,
|
||||
'ssl':{'key':str(root/'key.pem'),'certificate':str(root/'cert.pem')}},
|
||||
'hosts':{'localhost':{'options':{'authentication':'internal_hashed','password_hash':'SHA-256'}}}}))
|
||||
for number in range(2):
|
||||
subprocess.run([str(binary),'create-user',str(config),f'load-{number}','--password-stdin'],
|
||||
input='load-test-password\n',text=True,check=True,capture_output=True)
|
||||
archive = root/'data'/'localhost'/'archive'/f'load%2d{number}.list'
|
||||
archive.parent.mkdir(parents=True,exist_ok=True)
|
||||
with archive.open('w') as stream:
|
||||
for index in range(args.history_records):
|
||||
stream.write("item({name='message';attr={xmlns='jabber:client';type='chat'};"
|
||||
f"key='seed-{index}';when={time.time()};with='load-{1-number}@localhost';"
|
||||
"{name='body';attr={};'history'}});\n")
|
||||
sync_log = root/'sync.jsonl'
|
||||
env = dict(os.environ, LD_PRELOAD=str(shim), XMPP_SLOW_SYNC_ROOT=str(root/'data'),
|
||||
XMPP_SLOW_SYNC_LOG=str(sync_log), XMPP_SLOW_SYNC_MS=str(delay))
|
||||
with (root/'server.log').open('w+') as log:
|
||||
server = subprocess.Popen([str(binary),'serve',str(config),'--experimental'],env=env,stdout=log,stderr=log)
|
||||
try:
|
||||
deadline = time.monotonic()+60
|
||||
while True:
|
||||
assert server.poll() is None, log.read()
|
||||
try:
|
||||
with socket.create_connection(('127.0.0.1',port),timeout=.1):
|
||||
break
|
||||
except OSError:
|
||||
assert time.monotonic()<deadline, 'server startup timeout'
|
||||
time.sleep(.05)
|
||||
args.current_delay_ms = delay
|
||||
case = asyncio.run(exercise(root,port,args,sync_log))
|
||||
case['delay_ms_per_sync'] = delay
|
||||
events = slow_events(root/'server.log',delay,args.rounds,args.allow_legacy_phase_logs)
|
||||
case['server_message_processing_slow_events'] = events
|
||||
case['server_message_processing_slow_ms'] = [event['duration_ms'] for event in events]
|
||||
report['cases'].append(case)
|
||||
finally:
|
||||
stop(server)
|
||||
if args.output:
|
||||
pathlib.Path(args.output).write_text(json.dumps(report,indent=2)+'\n')
|
||||
print(f'slow storage: delay={delay}ms pass',flush=True)
|
||||
print(json.dumps(report,indent=2))
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,322 @@
|
||||
//! Redo recovery is independent of the best-effort readable JSON export.
|
||||
use rust_xmpp::sessions::Hub;
|
||||
use std::{fs, io::Write, path::Path, time::Duration};
|
||||
use tokio::sync::mpsc;
|
||||
|
||||
fn committed(dir: &Path) -> (String, Vec<u8>) {
|
||||
let hub = Hub::persistent(dir).unwrap();
|
||||
let (tx, _rx) = mpsc::channel(8);
|
||||
let session = hub.bind("a@x/r", "c", tx, Duration::from_secs(60)).unwrap();
|
||||
let mut state = session.lock().unwrap();
|
||||
state.enable(true).unwrap();
|
||||
state.transmit("durable".into()).unwrap();
|
||||
let token = state.resume_id.clone().unwrap();
|
||||
let json = fs::read(dir.join(format!("{token}.json"))).unwrap();
|
||||
drop(state);
|
||||
drop(hub);
|
||||
(token, json)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn redo_repairs_missing_or_torn_export_before_json_validation() {
|
||||
for torn in [false, true] {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let (token, _) = committed(dir.path());
|
||||
let json = dir.path().join(format!("{token}.json"));
|
||||
if torn {
|
||||
fs::write(&json, b"{torn").unwrap();
|
||||
} else {
|
||||
fs::remove_file(&json).unwrap();
|
||||
}
|
||||
let hub = Hub::persistent(dir.path()).unwrap();
|
||||
assert_eq!(
|
||||
hub.get("a@x/r")
|
||||
.unwrap()
|
||||
.lock()
|
||||
.unwrap()
|
||||
.pending
|
||||
.front()
|
||||
.map(String::as_str),
|
||||
Some("durable")
|
||||
);
|
||||
let export: serde_json::Value = serde_json::from_slice(&fs::read(json).unwrap()).unwrap();
|
||||
assert_eq!(export["pending"][0], "durable");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn partial_tail_is_discarded_and_subsequent_commits_survive_restart() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let (token, _) = committed(dir.path());
|
||||
let wal = dir.path().join(format!("{token}.wal"));
|
||||
let good_len = fs::metadata(&wal).unwrap().len();
|
||||
fs::OpenOptions::new()
|
||||
.append(true)
|
||||
.open(&wal)
|
||||
.unwrap()
|
||||
.write_all(&[42, 0, 0])
|
||||
.unwrap();
|
||||
let hub = Hub::persistent(dir.path()).unwrap();
|
||||
assert_eq!(fs::metadata(&wal).unwrap().len(), good_len);
|
||||
hub.get("a@x/r")
|
||||
.unwrap()
|
||||
.lock()
|
||||
.unwrap()
|
||||
.mark_handled()
|
||||
.unwrap();
|
||||
drop(hub);
|
||||
let hub = Hub::persistent(dir.path()).unwrap();
|
||||
assert_eq!(hub.get("a@x/r").unwrap().lock().unwrap().handled, 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn complete_corrupt_frame_fails_closed_even_with_valid_export() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let (token, _) = committed(dir.path());
|
||||
let wal = dir.path().join(format!("{token}.wal"));
|
||||
let mut bytes = fs::read(&wal).unwrap();
|
||||
*bytes.last_mut().unwrap() ^= 1;
|
||||
fs::write(wal, bytes).unwrap();
|
||||
assert!(Hub::persistent(dir.path()).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn durable_tombstone_overrides_resurrected_stale_json() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let (token, old) = committed(dir.path());
|
||||
let hub = Hub::persistent(dir.path()).unwrap();
|
||||
hub.kick_owner("a@x");
|
||||
drop(hub);
|
||||
fs::write(dir.path().join(format!("{token}.json")), old).unwrap();
|
||||
let hub = Hub::persistent(dir.path()).unwrap();
|
||||
assert!(hub.get("a@x/r").is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn legacy_json_migrates_and_baseline_survives_partial_first_append() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let (token, _) = committed(dir.path());
|
||||
let wal = dir.path().join(format!("{token}.wal"));
|
||||
fs::remove_file(&wal).unwrap();
|
||||
let hub = Hub::persistent(dir.path()).unwrap();
|
||||
hub.get("a@x/r")
|
||||
.unwrap()
|
||||
.lock()
|
||||
.unwrap()
|
||||
.mark_handled()
|
||||
.unwrap();
|
||||
drop(hub);
|
||||
let bytes = fs::read(&wal).unwrap();
|
||||
let baseline_len = u32::from_le_bytes(bytes[8..12].try_into().unwrap()) as usize;
|
||||
fs::write(&wal, &bytes[..8 + 40 + baseline_len + 12]).unwrap();
|
||||
let hub = Hub::persistent(dir.path()).unwrap();
|
||||
let session = hub.get("a@x/r").unwrap();
|
||||
let state = session.lock().unwrap();
|
||||
assert_eq!(state.handled, 0);
|
||||
assert_eq!(state.pending.front().map(String::as_str), Some("durable"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn rotation_preserves_latest_snapshot_and_bounds_redo_size() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let hub = Hub::persistent(dir.path()).unwrap();
|
||||
let (tx, _rx) = mpsc::channel(8);
|
||||
let session = hub.bind("a@x/r", "c", tx, Duration::from_secs(60)).unwrap();
|
||||
let token;
|
||||
{
|
||||
let mut state = session.lock().unwrap();
|
||||
state.enable(true).unwrap();
|
||||
token = state.resume_id.clone().unwrap();
|
||||
state.transmit("x".repeat(1024 * 1024)).unwrap();
|
||||
for _ in 0..35 {
|
||||
state.mark_handled().unwrap();
|
||||
}
|
||||
}
|
||||
let wal = dir.path().join(format!("{token}.wal"));
|
||||
assert!(fs::metadata(wal).unwrap().len() < 18 * 1024 * 1024);
|
||||
drop(hub);
|
||||
let hub = Hub::persistent(dir.path()).unwrap();
|
||||
let session = hub.get("a@x/r").unwrap();
|
||||
let state = session.lock().unwrap();
|
||||
assert_eq!(state.handled, 35);
|
||||
assert_eq!(state.pending.front().unwrap().len(), 1024 * 1024);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn incomplete_baseline_or_magic_never_falls_back_to_json() {
|
||||
for keep in [0, 4, 8, 12, 48] {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let (token, _) = committed(dir.path());
|
||||
let wal = dir.path().join(format!("{token}.wal"));
|
||||
let bytes = fs::read(&wal).unwrap();
|
||||
fs::write(wal, &bytes[..keep]).unwrap();
|
||||
assert!(
|
||||
Hub::persistent(dir.path()).is_err(),
|
||||
"truncated baseline at {keep}"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[cfg(unix)]
|
||||
fn journal_symlink_is_rejected_without_following_target() {
|
||||
use std::os::unix::fs::symlink;
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let (token, _) = committed(dir.path());
|
||||
let wal = dir.path().join(format!("{token}.wal"));
|
||||
let other = tempfile::NamedTempFile::new().unwrap();
|
||||
fs::remove_file(&wal).unwrap();
|
||||
symlink(other.path(), wal).unwrap();
|
||||
assert!(Hub::persistent(dir.path()).is_err());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn export_failure_after_commit_advances_memory_and_fences_later_writes() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let hub = Hub::persistent(dir.path()).unwrap();
|
||||
let (tx, _rx) = mpsc::channel(8);
|
||||
let session = hub.bind("a@x/r", "c", tx, Duration::from_secs(60)).unwrap();
|
||||
session.lock().unwrap().enable(true).unwrap();
|
||||
let token = session.lock().unwrap().resume_id.clone().unwrap();
|
||||
let obstruction = dir.path().join(format!("{token}.tmp"));
|
||||
fs::create_dir(&obstruction).unwrap();
|
||||
{
|
||||
let mut state = session.lock().unwrap();
|
||||
state.mark_handled().unwrap();
|
||||
assert_eq!(state.handled, 1);
|
||||
assert!(state.mark_handled().is_err());
|
||||
assert_eq!(state.handled, 1);
|
||||
}
|
||||
drop(hub);
|
||||
fs::remove_dir(obstruction).unwrap();
|
||||
let hub = Hub::persistent(dir.path()).unwrap();
|
||||
assert_eq!(hub.get("a@x/r").unwrap().lock().unwrap().handled, 1);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn retired_journals_are_collected_in_bounded_batches() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let hub = Hub::persistent(dir.path()).unwrap();
|
||||
for index in 0..64 {
|
||||
let (tx, _rx) = mpsc::channel(1);
|
||||
let connection = format!("c{index}");
|
||||
let session = hub
|
||||
.bind("a@x/r", &connection, tx, Duration::from_secs(60))
|
||||
.unwrap();
|
||||
session.lock().unwrap().enable(true).unwrap();
|
||||
hub.disconnect(&connection, true);
|
||||
}
|
||||
assert_eq!(
|
||||
fs::read_dir(dir.path())
|
||||
.unwrap()
|
||||
.filter(|entry| {
|
||||
entry
|
||||
.as_ref()
|
||||
.unwrap()
|
||||
.path()
|
||||
.extension()
|
||||
.is_some_and(|ext| ext == "wal")
|
||||
})
|
||||
.count(),
|
||||
0
|
||||
);
|
||||
drop(hub);
|
||||
assert!(Hub::persistent(dir.path()).unwrap().get("a@x/r").is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn offline_checkpoint_repairs_exports_removes_redo_and_allows_legacy_restart() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let (token, _) = committed(dir.path());
|
||||
let json = dir.path().join(format!("{token}.json"));
|
||||
fs::write(&json, b"{torn").unwrap();
|
||||
assert_eq!(
|
||||
rust_xmpp::session_store::checkpoint_exports(dir.path()).unwrap(),
|
||||
1
|
||||
);
|
||||
assert!(!dir.path().join(format!("{token}.wal")).exists());
|
||||
let export: serde_json::Value = serde_json::from_slice(&fs::read(json).unwrap()).unwrap();
|
||||
assert_eq!(export["pending"][0], "durable");
|
||||
let hub = Hub::persistent(dir.path()).unwrap();
|
||||
let session = hub.get("a@x/r").unwrap();
|
||||
assert_eq!(
|
||||
session.lock().unwrap().pending.front().map(String::as_str),
|
||||
Some("durable")
|
||||
);
|
||||
session.lock().unwrap().mark_handled().unwrap();
|
||||
assert!(dir.path().join(format!("{token}.wal")).exists());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn offline_checkpoint_refuses_live_store_and_preserves_redo() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let hub = Hub::persistent(dir.path()).unwrap();
|
||||
let (tx, _rx) = mpsc::channel(8);
|
||||
let session = hub.bind("a@x/r", "c", tx, Duration::from_secs(60)).unwrap();
|
||||
session.lock().unwrap().enable(true).unwrap();
|
||||
let token = session.lock().unwrap().resume_id.clone().unwrap();
|
||||
assert!(rust_xmpp::session_store::checkpoint_exports(dir.path()).is_err());
|
||||
assert!(dir.path().join(format!("{token}.wal")).exists());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn offline_checkpoint_durably_applies_expiration_and_tombstones() {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let hub = Hub::persistent(dir.path()).unwrap();
|
||||
let (tx, _rx) = mpsc::channel(8);
|
||||
let session = hub.bind("a@x/r", "c", tx, Duration::from_secs(60)).unwrap();
|
||||
let token;
|
||||
{
|
||||
let mut state = session.lock().unwrap();
|
||||
state.enable(true).unwrap();
|
||||
token = state.resume_id.clone().unwrap();
|
||||
state.expires = Some(std::time::Instant::now());
|
||||
state.checkpoint().unwrap();
|
||||
}
|
||||
drop(hub);
|
||||
assert_eq!(
|
||||
rust_xmpp::session_store::checkpoint_exports(dir.path()).unwrap(),
|
||||
0
|
||||
);
|
||||
assert!(!dir.path().join(format!("{token}.json")).exists());
|
||||
assert!(!dir.path().join(format!("{token}.wal")).exists());
|
||||
assert!(Hub::persistent(dir.path()).unwrap().get("a@x/r").is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn superseded_frames_still_require_valid_checksums_json_and_baseline_flags() {
|
||||
use sha2::{Digest, Sha256};
|
||||
for damage in ["checksum", "json", "baseline"] {
|
||||
let dir = tempfile::tempdir().unwrap();
|
||||
let (token, _) = committed(dir.path());
|
||||
let wal = dir.path().join(format!("{token}.wal"));
|
||||
let mut bytes = fs::read(&wal).unwrap();
|
||||
// Damage the baseline, leaving both later committed snapshots intact.
|
||||
let mut len = u32::from_le_bytes(bytes[8..12].try_into().unwrap());
|
||||
let mut payload = 48..48 + len as usize;
|
||||
match damage {
|
||||
"checksum" => bytes[payload.end - 1] ^= 1,
|
||||
"json" => bytes[payload.end - 1] = b'!',
|
||||
"baseline" => {
|
||||
let start = bytes[payload.clone()]
|
||||
.windows(4)
|
||||
.position(|part| part == b"true")
|
||||
.unwrap()
|
||||
+ payload.start;
|
||||
// Preserve valid JSON, but make the first frame a continuation.
|
||||
bytes.splice(start..start + 4, b"false".iter().copied());
|
||||
len += 1;
|
||||
payload.end += 1;
|
||||
bytes[8..12].copy_from_slice(&len.to_le_bytes());
|
||||
bytes[12..16].copy_from_slice(&(!len).to_le_bytes());
|
||||
}
|
||||
_ => unreachable!(),
|
||||
}
|
||||
if damage != "checksum" {
|
||||
let checksum = Sha256::digest(&bytes[payload]);
|
||||
bytes[16..48].copy_from_slice(&checksum);
|
||||
}
|
||||
fs::write(wal, bytes).unwrap();
|
||||
assert!(Hub::persistent(dir.path()).is_err(), "{damage}");
|
||||
}
|
||||
}
|
||||
@@ -5,6 +5,7 @@ Usage: python3 tests/sm_restart_network.py BINARY
|
||||
Pending server stanzas are at-least-once: a received but unacknowledged stanza
|
||||
must replay. This test does not assert exactly-once client stanza processing.
|
||||
"""
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import signal
|
||||
@@ -23,6 +24,36 @@ SM = 'urn:xmpp:sm:3'
|
||||
ROOM = 'durable@rooms.localhost'
|
||||
|
||||
|
||||
def durable_slot(root, token):
|
||||
"""Read the authoritative checksummed WAL, never a potentially stale export."""
|
||||
paths = list((root/'data').rglob(token + '.wal'))
|
||||
assert len(paths) == 1, ('missing or duplicate durable SM journal', paths)
|
||||
data = paths[0].read_bytes()
|
||||
assert data[:8] == b'XMPPSM01', 'invalid durable SM journal header'
|
||||
offset, frames, record = 8, 0, None
|
||||
while offset < len(data):
|
||||
# An abrupt kill can leave an incomplete final append. Recovery uses
|
||||
# the preceding complete frame; an incomplete baseline is never valid.
|
||||
if len(data)-offset < 8:
|
||||
break
|
||||
size = int.from_bytes(data[offset:offset+4], 'little')
|
||||
inverse = int.from_bytes(data[offset+4:offset+8], 'little')
|
||||
assert size ^ inverse == 0xffffffff and size <= 8*1024*1024+1024, 'invalid durable SM frame length'
|
||||
end = offset+40+size
|
||||
if end > len(data):
|
||||
break
|
||||
payload = data[offset+40:end]
|
||||
assert hashlib.sha256(payload).digest() == data[offset+8:offset+40], 'invalid durable SM frame checksum'
|
||||
frame = json.loads(payload)
|
||||
assert frame['baseline'] == (frames == 0), 'invalid durable SM baseline sequence'
|
||||
record = frame['snapshot']
|
||||
frames += 1
|
||||
offset = end
|
||||
assert frames and record is not None, 'missing committed live SM snapshot'
|
||||
assert record['resume_id'] == token, 'durable SM token mismatch'
|
||||
return record
|
||||
|
||||
|
||||
def enable(client):
|
||||
client.send(f"<enable xmlns='{SM}' resume='true'/>")
|
||||
response = client.take(lambda e: e.tag == f'{{{SM}}}enabled')
|
||||
@@ -115,7 +146,7 @@ def run(binary):
|
||||
client = Client(port, cert, user, 'secret', resource)
|
||||
clients.append(client)
|
||||
return client
|
||||
def restart(how):
|
||||
def restart(how, before_start=None):
|
||||
nonlocal server
|
||||
if how == 'kill':
|
||||
server.send_signal(signal.SIGKILL)
|
||||
@@ -125,6 +156,8 @@ def run(binary):
|
||||
for client in clients:
|
||||
client.close()
|
||||
clients.clear()
|
||||
if before_start:
|
||||
before_start()
|
||||
server = start(binary, config, port, log)
|
||||
try:
|
||||
server = start(binary, config, port, log)
|
||||
@@ -139,7 +172,15 @@ def run(binary):
|
||||
alice.send("<message to='bob@localhost'><body>incoming before crash</body></message>")
|
||||
bob.body('incoming before crash')
|
||||
assert handled(alice) == 3
|
||||
restart('kill')
|
||||
def corrupt_export():
|
||||
record = durable_slot(root, token)
|
||||
assert record['handled'] == 3
|
||||
exports = list((root/'data').rglob(token + '.json'))
|
||||
assert len(exports) == 1
|
||||
exports[0].write_text('{torn JSON export')
|
||||
# The fsynced journal must repair a torn readable export and
|
||||
# retain the latest handled counter and pending IQ.
|
||||
restart('kill', corrupt_export)
|
||||
for user, ident, h in [('bob', token, 1), ('alice', 'unknown-token', 1), ('alice', token, 99)]:
|
||||
resume(port, cert, ident, h, clients, user=user, success=False)
|
||||
alice, result = resume(port, cert, token, 1, clients)
|
||||
@@ -203,9 +244,7 @@ def run(binary):
|
||||
assert handled(alice) == 5
|
||||
# Inspect the committed slot before any Bob authentication.
|
||||
# A history replay on rejoining is not offline SM delivery.
|
||||
slots = list((root/'data').rglob(bob_token + '.json'))
|
||||
assert len(slots) == 1, slots
|
||||
record = json.loads(slots[0].read_text())
|
||||
record = durable_slot(root, bob_token)
|
||||
assert record['jid'] == 'bob@localhost/room', record
|
||||
queued = [ET.fromstring(xml) for xml in record['pending']]
|
||||
assert any(packet.get('from') == ROOM + '/Alice'
|
||||
@@ -256,7 +295,7 @@ def run(binary):
|
||||
if sys.exc_info()[0] is not None:
|
||||
log.seek(0)
|
||||
print(log.read(), file=sys.stderr)
|
||||
print('PASS: TLS legacy/SASL2 durable SM after SIGKILL/SIGTERM, counters, ack pruning, at-least-once pending replay, owner/token/h rejection, transport/resource conflict, MUC activation/offline fanout, grace expiry and account deletion/recreation revocation')
|
||||
print('PASS: TLS legacy/SASL2 durable SM after SIGKILL/SIGTERM, WAL repair of torn JSON export, counters, ack pruning, at-least-once pending replay, owner/token/h rejection, transport/resource conflict, MUC activation/offline fanout, grace expiry and account deletion/recreation revocation')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
|
||||
@@ -353,7 +353,7 @@ fn purge_account_exact_files_preserves_others_and_extras() {
|
||||
let root = t.path();
|
||||
for store in ["accounts", "roster", "pep_aabbcc", "custom_store"] {
|
||||
for user in ["alice", "bob"] {
|
||||
for ext in ["dat", "list", "lidx", "rappend"] {
|
||||
for ext in ["dat", "list", "lidx", "rappend", "rredo"] {
|
||||
let p = storage::path(root, "example.test", Some(user), store).with_extension(ext);
|
||||
std::fs::create_dir_all(p.parent().unwrap()).unwrap();
|
||||
std::fs::write(&p, b"fixture").unwrap();
|
||||
@@ -367,7 +367,7 @@ fn purge_account_exact_files_preserves_others_and_extras() {
|
||||
std::fs::write(&hostwide, b"keep").unwrap();
|
||||
a::remove_account(root, "example.test", "alice").unwrap();
|
||||
for store in ["accounts", "roster", "pep_aabbcc", "custom_store"] {
|
||||
for ext in ["dat", "list", "lidx", "rappend"] {
|
||||
for ext in ["dat", "list", "lidx", "rappend", "rredo"] {
|
||||
assert!(
|
||||
!storage::path(root, "example.test", Some("alice"), store)
|
||||
.with_extension(ext)
|
||||
|
||||
Reference in new issue
Block a user