- Уменьшено число последовательных синхронизаций до доставки: ordinary
Test, build and publish / All tests and Linux binary (push) Failing after 30s
Test, build and publish / Container smoke test and Gitea publish (push) Skipped

MAM
  append использует один sync вместо четырёх, SM checkpoint — один
  вместо двух.
  Checksummed redo journals сохраняют commit-before-ACK и
  восстанавливают
  native archive/SM JSON exports после аварии.
- Ограничены рост журналов, ротация и tombstone GC. Старый `.rappend` и
  legacy
  JSON импортируются. Retain/save/purge не возвращают удалённые записи.
- Recovery синхронизирует валидный journal перед публикацией состояния;
  проверка
  checksum не заменяет durability barrier. EIO при adoption блокирует
  startup
  или offline handoff. Новые fault fixtures проверяют сохранность
  журнала и retry.
- Replay SM проверяет checksum/JSON всех frames, но создаёт и валидирует
  полный
  Record только для последнего snapshot, сокращая работу с superseded
  queues.
- `message-processing-slow` содержит времена MAM отправителя/получателя,
  доставки, carbons, offline/push и handled, а также число
  ресурсов/копий.
  Payload и секреты не добавляются в логи.
- Добавлена offline команда `storage-checkpoint CONFIG`: durable экспорт
  и
  передача archive writer/подготовка отката на legacy Rust. Не
  запускайте другого
  writer и не удаляйте journals до успешного завершения команды.
- Новые crash, partial-write, corruption, EIO, recovery, handoff и
  latency
  регрессии включены в GitHub Actions через verify-local.sh; отчёты
  сохраняются.

Измерения, ограничения и эксплуатация:
[docs/DURABLE-MESSAGE-IO.md](docs/DURABLE-MESSAGE-IO.md),
[docs/VALIDATION-0.8.3.md](docs/VALIDATION-0.8.3.md).
This commit is contained in:
wt committed 2026-10-21 13:14:19 +07:00
1 parent e9329ccbb2
commit bff609097b
53 files changed
+8924 -1792

No files matched your search

+3
View File
@@ -85,6 +85,9 @@ jobs:
/tmp/push-latency.json
/tmp/push-offline-latency.json
/tmp/bind2-archive-latency.json
/tmp/slow-storage-latency.json
/tmp/slow-storage-crash.json
/tmp/journal-recovery-sync.json
if-no-files-found: warn
retention-days: 14
+62 -1
View File
@@ -28,7 +28,7 @@
В нём сейчас нет `.git`. Если следующая среда содержит Git checkout, используйте
его состояние и инструкции, не создавайте второй расходящийся источник истины.
Версия Rust source — **0.8.2**. В0.8.0 добавлены durable MUC/SM, затем CI
Версия Rust source — **0.8.3**. В0.8.0 добавлены durable MUC/SM, затем CI
и этот handoff;0.8.1 исправляет задержку сообщений. См. docs/MESSAGE-LATENCY.md.
Перед передачей исходники упаковываются в ZIP; актуальный архив должен содержать
этот файл и `.github/workflows/ci.yml`. В среде ChatGPT сохраняйте новый архив
@@ -237,3 +237,64 @@ bind2_archive_latency_network.py, archive_edges.rs, session_routing_isolation.rs
В передаваемый ZIP включайте AGENTS.md, workflow, vendor reference, исходники,
документацию и scripts; исключайте target, demo data, bin, .git, __pycache__, pyc
и секреты. Сохраняйте executable bits shell scripts и проверяйте содержимое ZIP.
## Durable message IO 0.8.3
Новый пользовательский log 0.8.2 содержит три local message-processing-slow
4792–5549 мс. Таймер начинается внутри server после XML, но старые поля не
доказывают причину IO. Контролируемый slow-sync fixture воспроизводит задержку
1:1 в несколько секунд. MAM 4 sync/append и SM 2 sync/checkpoint заменены
ограниченными checksummed redo journals. Обычная операция теперь требует 1 sync;
создание, recovery и rotation дороже. Точные результаты — VALIDATION-0.8.3.md
и DURABLE-MESSAGE-IO.md. Не приписывайте loopback причину production без phase logs.
SM `.wal` авторитетен, `.json` — readable export. Replay выполняется до JSON parse.
Неполный trailing frame отбрасывается; неполный baseline или полная corruption
блокируют startup. Durable tombstone предшествует delete. GC сначала делает
sync каталога с JSON deletions, затем unlink WAL и sync каталога.
Ordinary commit: append + sync_data, потом JSON export без sync. Ошибка export
после commit не откатывает память; store блокирует дальнейшие записи до recovery.
Precommit/sync error также блокирует store. Не возвращайте 2 sync на каждый
checkpoint и не удаляйте redo без checkpoint. Общий SM mutex пока сохранён.
Slot limit 8 MiB, total 256 MiB; WAL rotation 16 MiB плюс один slot, aggregate
512 MiB. Не допускайте роста каждого из тысяч slots без aggregate cap.
Archive `.rredo`: native baseline и bounded anchor, checksummed record frames.
Rotation после 128 frames/1 MiB плюс одна запись до 16 MiB. Cache ограничен 1024;
изменение metadata вызывает перепроверку. Accepted records восстанавливаются
даже при потере dirty native pages. Unjournaled native suffix не доказывает
external append: truncate до committed end. Legacy `.rappend` ещё восстанавливается.
Save/retain сначала durably фиксируют old native и удаление redo, затем
делают atomic replacement. Смена inode при копировании/backup НЕ отменяет redo: journal
остаётся авторитетным. Любой внешний writer, даже atomic replace, требует handoff. Purge включает rredo.
Recovery повторно инвалидирует `.lidx`: unlink на hot path отдельно не dirsynced.
Anchor проверяет только последние 4 KiB baseline, не всю старую историю.
Перед сменой writer/откатом остановите сервер и новым binary выполните
`rust-xmpp storage-checkpoint CONFIG`. Внешний data-directory lock обязателен.
`archive::checkpoint_all` и `session_store::checkpoint_exports` — только offline.
Сначала sync native/current JSON и каталога, потом remove journal и sync каталога.
Graceful stop сам по себе не означает handoff. Native Prosody не импортирует
Rust SM sessions. Не редактируйте JSON export для expiry fixtures: используйте
короткий настоящий timeout или проверяйте journal recovery. Не удаляйте WAL,
чтобы получить зелёный тест. Native oracle перед Prosody modification явно
вызывает checkpoint_all(root).
Новые tests: archive_redo.rs, sm_journal.rs, slow_storage_latency_network.py,
slow_storage_crash_network.py, journal_recovery_sync_network.py, fixtures/slow_sync.c. Interposer действует только
на изолированный fixture data_path. Обычный режим выполняет настоящий
fsync/fdatasync; отдельный XMPP_SYNC_FAIL_SUFFIX вводит EIO только для fault test.
Crash fixture проверяет journals и оба MAM архива до reauth после sender SM ACK
и SIGKILL. Readable JSON без redo не подтверждает durability. Phase fields —
server.rs и DURABLE-MESSAGE-IO.md.
В восстановленной среде native prefix теперь `/tmp/xmpp-native/usr/usr`, а не
старый `/tmp/xmpp-native/usr`. `/tmp/xmpp083-env.sh` содержит env текущего прогона;
это не portable config. Toolchain переустановлен; tests используют Rust 1.99.1.
Итог 0.8.3: 334 ordinary passed, 11 ignored; 9 native и 1 large-archive ignored
выполнены отдельно. Все network checks и обе нагрузки 500 TLS прошли.
Final binary SHA-256: 8ceb8de3a2d02bc4399b8ae817dff7e3c3fe9bff83dc0c6669b7f19210a3847a.
Подробные samples/границы — docs/VALIDATION-0.8.3.md. Checksum и JSON syntax всех frames
проверяются при SM replay, но Record создаётся только для последнего
snapshot. Recovery WAL sync до публикации обязателен; EIO fixture защищает это.
+25
View File
@@ -0,0 +1,25 @@
# 0.8.3 — durable IO личных сообщений
- Уменьшено число последовательных синхронизаций до доставки: ordinary MAM
append использует один sync вместо четырёх, SM checkpoint — один вместо двух.
Checksummed redo journals сохраняют commit-before-ACK и восстанавливают
native archive/SM JSON exports после аварии.
- Ограничены рост журналов, ротация и tombstone GC. Старый `.rappend` и legacy
JSON импортируются. Retain/save/purge не возвращают удалённые записи.
- Recovery синхронизирует валидный journal перед публикацией состояния; проверка
checksum не заменяет durability barrier. EIO при adoption блокирует startup
или offline handoff. Новые fault fixtures проверяют сохранность журнала и retry.
- Replay SM проверяет checksum/JSON всех frames, но создаёт и валидирует полный
Record только для последнего snapshot, сокращая работу с superseded queues.
- `message-processing-slow` содержит времена MAM отправителя/получателя,
доставки, carbons, offline/push и handled, а также число ресурсов/копий.
Payload и секреты не добавляются в логи.
- Добавлена offline команда `storage-checkpoint CONFIG`: durable экспорт и
передача archive writer/подготовка отката на legacy Rust. Не запускайте другого
writer и не удаляйте journals до успешного завершения команды.
- Новые crash, partial-write, corruption, EIO, recovery, handoff и latency
регрессии включены в GitHub Actions через verify-local.sh; отчёты сохраняются.
Измерения, ограничения и эксплуатация:
[docs/DURABLE-MESSAGE-IO.md](docs/DURABLE-MESSAGE-IO.md),
[docs/VALIDATION-0.8.3.md](docs/VALIDATION-0.8.3.md).
Generated
+1 -1
View File
@@ -1111,7 +1111,7 @@ dependencies = [
[[package]]
name = "rust-xmpp"
version = "0.8.2"
version = "0.8.3"
dependencies = [
"anyhow",
"axum",
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "rust-xmpp"
version = "0.8.2"
version = "0.8.3"
edition = "2024"
description = "Experimental Rust XMPP server with Prosody-compatible storage"
license = "MIT"
+8 -1
View File
@@ -1,4 +1,4 @@
# Rust XMPP 0.8.2
# Rust XMPP 0.8.3
Сервер на Rust, разрабатываемый для замены **Prosody 13.0 nightly 100,
cafd74b9c5ea** с предоставленными модулями. Читает существующий Lua-конфиг,
@@ -251,3 +251,10 @@ recovery Rust служит собственный журнал. Перед от
Push routing больше не блокирует отправителя/SM ACK; CSI defaults немедленно
передают bodyless OMEMO и receipts. Bind2/MAM metadata не сканируют всю историю.
[Причины, тесты и конфигурация](docs/PERSONAL-MESSAGE-LATENCY.md).
## Durable IO личных сообщений в0.8.3
Сокращены последовательные sync MAM/SM через checksummed redo journals,
добавлены замеры этапов медленной обработки и crash/replay проверки.
Перед передачей файлов Prosody/откатом выполните offline `storage-checkpoint`.
[Durability, команда handoff и диагностика](docs/DURABLE-MESSAGE-IO.md).
+6
View File
@@ -166,3 +166,9 @@ ad-hoc destroy/set-affiliation. Общие и host `modules_disabled` приме
Ограничения журнала, native-экспорта и отката:
[MUC-RECOVERY.md](MUC-RECOVERY.md). Итоги проверки:
[VALIDATION-0.8.0.md](VALIDATION-0.8.0.md).
## Durable IO0.8.3
MAM/SM используют authoritative redo sidecars. Перед сменой writer или откатом
на старую версию требуется offline `storage-checkpoint CONFIG`; после аварии
нельзя просто удалить sidecars. Подробности — [DURABLE-MESSAGE-IO.md](DURABLE-MESSAGE-IO.md).
+103
View File
@@ -0,0 +1,103 @@
# Личные сообщения и durable IO в 0.8.3
В присланном журнале сервера 0.8.2 есть три `message-processing-slow` с 4792–5549 мс,
route=local. Этот таймер начинается после приёма XML и заканчивается после
`mark_handled`: ожидание происходит внутри обработчика. Старые события не
содержат разбивки по этапам и сами по себе не доказывают причину медленного IO.
Исходный пользовательский журнал не включён в архив проекта.
## Изменение записи
До 0.8.3 append в MAM требовал 4 последовательных sync: rollback marker file+dir,
native list, удаление marker+dir. Обычное сохранение SM требовало 2: JSON file+dir.
В TLS fixture1:1 с двумя аккаунтами, четырьмя SM-ресурсами и carbons это обычно
16 sync на сообщение. Увеличение задержки каждого настоящего fsync/fdatasync
внутри изолированного data_path воспроизводит задержку в несколько секунд.
Interposer не пропускает sync и не меняет системные настройки/production.
В 0.8.3 принятые данные сначала фиксируются в журнале восстановления с checksum:
- MAM `.rredo`: native `.list` сохраняет формат Prosody; обычный append требует
одного sync журнала. Replay восстанавливает недостающий native suffix и
отбрасывает незавершённый хвост. Полностью записанная повреждённая запись
вызывает ошибку. Ротация —128 frames/1 MiB плюс максимум одна 16 MiB запись.
- SM `.wal`: authoritative полные snapshots и tombstones; readable JSON —
восстанавливаемый экспорт. Обычное сохранение требует одного sync_data.
Ротация —16 MiB плюс максимум один snapshot; aggregate WAL cap 512 MiB,
logical slots по-прежнему 256 MiB/8 MiB на slot/4096 stanzas и 4096 sessions.
Tombstone GC сначала синхронизирует удаление JSON, затем удаляет WAL.
Recovery синхронизирует полностью проверенный журнал перед публикацией
восстановленного состояния: полный frame после SIGKILL может остаться
только в page cache. Checksum подтверждает целостность, но не запись на диск. Ошибка этого sync запрещает startup/handoff.
Создание журналов, восстановление и ротация дополнительно синхронизируют файлы
и каталог. Экономия относится к обычной записи, не каждому возможному запросу.
Отключения durability или периодической записи вместо commit-before-ACK нет.
При неопределённой ошибке SM IO store блокирует дальнейшие изменения до
восстановления. После успешного WAL commit ошибка экспорта не откатывает память.
Queues/archives не образуют общей транзакции с сетью: гарантия replay остаётся
at least once, а не exactly once между всеми хранилищами.
## Передача данных Prosody и откат версии
После SIGKILL/OS crash сначала используйте 0.8.3 для восстановления redo. Не
удаляйте `.wal`/`.rredo` вручную и не редактируйте JSON export как источник истины.
Для смены writer или отката на старую версию остановите сервер и выполните:
```bash
rust-xmpp storage-checkpoint /etc/prosody/prosody.cfg.lua
```
Команда берёт data-directory lock и SM store lock, восстанавливает журналы,
синхронизирует native archives и актуальные JSON, затем durably удаляет redo.
При ошибке передача владения не завершена; разберите ошибку, не запускайте
другой writer. Перед работой с production данными сохраните обычную резервную
копию всего data_path. Native Prosody не понимает resumable SM snapshots Rust;
checkpoint SM обеспечивает совместимый JSON для старого Rust, не переносит
TLS/TCP/SM-сессию в Prosody. Простой graceful stop не заменяет эту явную команду.
Нельзя одновременно изменять `.list` средствами Prosody и Rust. Неjournaled
in-place append при активном redo считается незавершённым и может быть отброшен.
При восстановлении согласованной копии всего data_path журнал остаётся
авторитетным независимо от смены inode файла. Любая внешняя модификация,
включая atomic replacement, требует handoff. Внутренний rewrite сначала
фиксирует старый native snapshot и удаление redo, затем атомарно заменяет native
файл: авария в промежутке сохраняет старый snapshot, после замены — старый/новый.
Редактирование baseline in place без handoff не поддержано. Anchor проверяет
последние 4 KiB baseline, не является checksum всего старого архива.
## Диагностика
`message-processing-slow` по-прежнему имеет threshold 250 мс, duration_ms, route
и connection. Новые поля в integer milliseconds:
|Поле|Измеряемый этап|
|---|---|
|pre_routing_ms|Работа до личного MAM; для MUC/ранних выходов — обработка до handled|
|sender_mam_ms|Политика и архивирование отправителя|
|recipient_mam_ms|Проверка аккаунта, политика и архивирование получателя|
|delivery_ms|Маршрутизация, targets, mutex и transmit|
|carbons_ms|Выбор ресурсов и отправка carbon|
|offline_push_ms|Offline commit и подготовка/enqueue push|
|handled_ms|Session mutex и durable mark_handled|
Неисполненные этапы равны 0. target_count — выбранные локальные ресурсы,
carbon_count — успешные carbon transmit. Это серверные этапы, не время
отображения клиентом. Тексты сообщений, SASL и токены не логируются.
## Проверки и границы
`tests/slow_storage_latency_network.py` — реальный STARTTLS, MAM/carbons/PEP,
контролируемый sync delay и счётчик операций. `tests/slow_storage_crash_network.py`
— sender SM ACK, немедленный SIGKILL, проверка committed journals/обоих архивов
до повторной аутентификации и replay всех ресурсов. Rust regressions проверяют
partial tails, corruption, ротацию, tombstones, migration и offline handoff.
Отдельный `journal_recovery_sync_network.py` вводит scoped EIO для WAL/redo
и проверяет отказ startup/handoff без потери journal и успешный повтор.
Точные измерения и результаты этой сборки — [VALIDATION.md](VALIDATION.md).
Общий mutex SM store, обязательное durable IO, большие pending snapshots,
полные MAM queries/maintenance, PEP storage и federation всё ещё имеют стоимость.
Медленный диск не становится быстрым от сокращения числа sync. Проверка на
loopback не устанавливает причину каждого production warning; новые phase
fields позволяют проверить это на целевой системе.
+4 -1
View File
@@ -80,7 +80,10 @@ Native state без подтверждающих Rust-метаданных са
восстановление. Это не универсальное слияние параллельных записей двух серверов.
Для отката на Prosody остановите Rust и сохраните резервную копию всего
`data_path`. Prosody использует native-экспорт и не читает Rust-журнал. Перед
`data_path`. Начиная с0.8.3 перед сменой writer выполните новым binary
`rust-xmpp storage-checkpoint CONFIG`: он фиксирует MAM redo и SM JSON exports.
Это не перенос Rust SM-сессий в Prosody и не универсальная конверсия MUC journal.
Подробности — [DURABLE-MESSAGE-IO.md](DURABLE-MESSAGE-IO.md). Prosody использует native-экспорт и не читает Rust-журнал. Перед
возвратом с Prosody на Rust согласуйте данные и устраните устаревший
`rust_muc_checkpoint` в остановленном экземпляре: иначе неизменившиеся native
файлы могут быть приняты за экспорт предыдущего поколения Rust. Удаление
+5
View File
@@ -100,3 +100,8 @@ SM Store пока использует общий mutex для durable slot IO;
и внешняя федерация имеют собственную стоимость. Производственная квалификация
клиентов и полная замена Prosody остаются за рамками этих fixtures.
Итоговые тесты/сырые отчёты: VALIDATION-0.8.2.md и validation-logs-0.8.2/.
## Продолжение0.8.3
Новые логи показали seconds-level серверную обработку. Добавлены redo MAM/SM
и разбивка времени на этапы: [DURABLE-MESSAGE-IO.md](DURABLE-MESSAGE-IO.md).
+21 -14
View File
@@ -1,13 +1,14 @@
79f3f5cd46255826869c9151151d3471dd7139c2ea8fb2da4b2b6fbea113f0cf .github/workflows/ci.yml
d9732ba55b93d2907cd1a2a0fa20bc1f46e0f2970de0e9f2bf730048212cfc31 AGENTS.md
f8e6a12039b067ed5af42320bca42fb0555d9a5f438d1652860a5be4534469d1 Cargo.lock
4e8d7ee328f044be284e0cb436f13a468adcac3058bbb1be381f8bff96603e5a Cargo.toml
9984392a77fcbb6d0ee70b536ecdcfc4349101bb300a509e32353aaa1134979e .github/workflows/ci.yml
6593266e57a718762fbca26fd893fcabf1ea842dcc53dd0fd11844b2def2a543 AGENTS.md
b83b20e991af384b349825ca39aad157d8b412d0143cc34c68567ca011bccf58 Cargo.lock
3ecc46532a7fb1a231dbbd5337221d0c0c32ba36574e74655854f1a1d54d4581 Cargo.toml
6be2fbcd1da8a07dda7daf3ddd727b38f363751ebace9c0b87c282f8961cffed Dockerfile
7136ea911bb6b6d352f703f80efeebde921028ea9dc39cd0d2cd55ea465f8c37 examples/config.json
8a65739eb5c24f10b9f2c509f0bd050fb1115706e6a41fdf804b9e0e44f583fd examples/config.toml
ca86274bcf4dc5daebbf69c9f003854afca92954719aff068f810e0a3d615dea examples/config.yaml
bbc2591581d382fef5fb918a480e6c338c7e3e3f2025df5509f51708534631e7 examples/converse-origin.fragment.yaml
66ea9e2b58f6a39b70911b9981cae5e99625483af7a41168e24923464de8ae56 examples/reverse-proxy.fragment.yaml
d13306be0ecda81ce779baa290ffb417380d525928477f16200884b5169cca33 rust-toolchain.toml
c08bd5923906f4d2163a258c8b68dbef39ea6a3e07032f67e5c12d3d9ca81c5e scripts/ci/image-tags.py
8820d262f8018a822b27eb9e5f0d3b9ca0c1272001164ca8a59fcd13911a5426 scripts/ci/prepare-minio.sh
5bcd646c6c9159bbdce3a4eadce4d80d6c7422748e47c6ad04e5769efb4f2920 scripts/ci/prepare-reference.sh
@@ -25,10 +26,10 @@ b1018e1b279c89f6cf14b4017e4f1b14e9165afef2f668cec6ac8bbe0c710d85 scripts/integr
e08ac6136cce27cda2ddbd510da0f25750766ab0150a9f4b8f87c953276d16e9 scripts/prosody_compat/oracle.lua
2c7a2c946885205e9df38b97305073a8ac72f58ac8ff84fc6f9af320c35bbc96 scripts/prosody_compat/verify.sh
1b9ed063daf3e0320819b4fe0135c96dd27e49877044aa2f5093b8ab4fbda41d scripts/smoke.py
c32bc7ed2d5f954a2b3de06db31fb1580378259edfc8ce574aa5cd5719cc2a59 scripts/verify-local.sh
2cd5e17cdf0902ceaaca9b6e8d36ca766e33e66a06784ec97ad99165571eaf91 scripts/verify-local.sh
a2e47f547ab9349f84ee6c4f8e51c147b7110b41a82c17cdb938027a84d82ef3 src/account_modules.rs
5637f816586f0613b83e14d0ed478bd539f325b70ccbfc9fdf893adbe494453d src/admin.rs
7b844cbe377b057555436378ab8bd784f99b85e7c9992cb3c5a044f40326951f src/archive.rs
1b2011fb38482eba69dc245c367ddf496089d0cbde8a000b1ff5e9bbf8c3dea8 src/archive.rs
e15b079a420227ee59e3b7ac305783c51eb59d76a8fb5dd12760547b0f812107 src/auth.rs
83ac35a4c9d20debf006fe80fba8e423c09806fb2526ff384b188ca28d612a5e src/config.rs
175ab07f17ba14c51c76857c4905b0a587d8be68e2be5179456e9c71a523c487 src/csi.rs
@@ -43,7 +44,7 @@ c604a0a764ffac6bae0c9aa15c9ef7022d35cb1f8c9159eb66257114445621c5 src/http_trans
5b27b6c719537af919bd6fe089574eba5e2c7bbe0d32249164914a4bb57a1427 src/lib.rs
a21e1119ea0271b63935e92eb106ab6076f0035e6d3a06c21a352f152b9b37db src/logging.rs
58ce2a9daa36ac26b39c8f263a16f4fa8cc4e57c91bbbf5b24a6aad3f1a326e0 src/lua.rs
a264e2765a31c8c096462551aca229530cad356e39e338370d1a0bdd97860680 src/main.rs
f103c15e11d61b5818e9a9491b8533b904cade7b3654c481cc61d3a7d0bb8173 src/main.rs
68d97d96aa3a370474c144ccfdbf16109c46a6b14123d55d30464237ae29a958 src/mam.rs
f17b06fe5b76381b5cdbfc2b668107f6c2b0b456c05b7a7709bea6a284531ff5 src/muc.rs
c7ba46a140b9dce65cc573176265d6f77049ae657a70ffe59611a1c0f3ef89af src/muc_admin.rs
@@ -56,10 +57,10 @@ f09a6479b76c9c601a169b5f1f9cf1364e82df5ae27e4bab8d1b1428b5cad457 src/preauth.rs
8dbb7a6b43a5f0562a775df42a959ab8badc466135a8b32da86cd79615b5acbf src/pubsub.rs
913cd033f60a68f6c7cf9b3adbaf76db1c5c9bc88a0599196d373b16b64c0a1e src/push.rs
bec477f2520af6a9d7ca54e43eb12158b53e3cf09057653e504dfdeab956885e src/sasl.rs
a445dcab6d0275861b08412a4bd622cb72c3f462ebba5eaedad360e740a6513c src/server.rs
c7450f9058528e3030df559840f87dae591a2b88be8503872956e9b78435d916 src/session_store.rs
1bcc6995e221f316bd4765eeb79c2ad67c9fe9e93d4254e6f6679244fc733e6f src/server.rs
c37f6c478d45b3c0741f0b1bed95f499e67d4d3eb50b78fe606e2b518170e45f src/session_store.rs
1d4e890315886bc4fd0b824e034a5c1833a1f2c1347e8c3ca77254e13a2acab9 src/sessions.rs
71d7ce1036a091d8c27016ea7ce4cffedf904dfca9d1cc4891b7dc66a1e06a4e src/storage.rs
c9fb5d18da148a6366a18afb86eaee26a67d20c7b6664e425e612d44e79a2988 src/storage.rs
462b00edb2c1e02dd34524bafe4f82b5ef902028b44310ba31b085d31f4dc2aa src/tls_binding.rs
a78592aa79d8f01c22af1457d0c3f5922863ccd3e8ebb1aa05e043cc44579b8a src/upload_migration.rs
f9b023565711e0217238b53fb678c5fb14849cdcd2efcd0fce249018274fad69 src/wire.rs
@@ -68,6 +69,7 @@ b4eb7a230570dfde0fd0d301a6ace36559a5eae3e1402e42c76f014f61739c24 tests/_wire_cl
0645fda01bede4362d998ef22aefdd2c02578dd2c2aae3878d0cff87b3679835 tests/android_sasl_network.py
ad225ade463f55bfd0734e67f5306e274cd9aab59e10e91469bf7263547cdf79 tests/archive_edges.rs
366b315521bd4f4448c0bac17bf74ddb77a728ff41fc48e29d46377625e0330f tests/archive_mam.rs
df8530d69beaf36c34e78f0f050930b7e0d6a296c22ca543dfc7533180ba0543 tests/archive_redo.rs
cb6b7bbcd1bc7ee01a4ccf892d4b6074765685c56c4b8bb46e0381b834577b6a tests/auth.rs
011bbdd00c8ff8b648076d133086c395f6784cdcbb0c9f4e1efcbc15c1e79d9c tests/bind2_archive_latency_network.py
b3c3470f0b9b64584abf98c26a4578b0b90089086d00f7b07187aa5f6db6a417 tests/cli.rs
@@ -78,6 +80,7 @@ d4d8d982c6f4ab96a809b598ea9ead7f64fa9d916547f1eb9d797aa71db64fdb tests/data_loc
be18a83fb5884cfdbd2ddc542cb25fd07b70f139cbdce1ae67562568df42a2e2 tests/debug_legacy.rs
63753c2e3524e8e7826bc2d48e2bd78e36e1cc56fde9da19950d5c35e1c38f77 tests/federation_tls.rs
1aaf06ba872cb5dbe0545afffdd8efddd0eae7305de06f3b3b75c19289581865 tests/fixtures/prosody.cfg.lua
33a8a5d9980bb9aa1162a1e3e562a9d889edd3d2f96a7073e14acc8e504bf9c7 tests/fixtures/slow_sync.c
19d570b92075d1355257c3798e4c52e404cc356dfe5dfd0c8fb384e4e7150682 tests/gajim_discovery_network.py
4398bf0639c31f0df328f0b0f1a5df17f2e69d2df84514845bba882146de06bc tests/gajim_features.rs
a346e70af3657fbf56e0bb1c39c427f630de8c349e6858b8933c52f42a21ab46 tests/http_altconnect.rs
@@ -86,6 +89,7 @@ f1ea3eb78d5c716f4281cf17cb41b9fb22e08dbb4cf08ed9c8477c79b7d2382b tests/http_pro
7c599c83eb999cd8776f0ce6811db8f478366b840311f2e513823f70b5d7cd30 tests/http_transport.rs
278db32ba0dec882b0af8f1969da87670ff3d2d749d32320154dabef51cff191 tests/http_upload.rs
da2fc65e931ec74fbcdcd26d04f2cb073e2dc1e354139135166c79580973a477 tests/iq_errors.rs
ec993a6e31d38f82abd13bd46ad99286cf13e8abea63c0385c848037a12b0cd4 tests/journal_recovery_sync_network.py
05160f17f8098466a67c9b069c165dc9d94a84ab03a49896169f110baf8e869a tests/logging.rs
dfc3429e3e755d4c99ec1bd8ee6fb0f1f6aaf86414d5adaca7e9e3b18c7fe3f7 tests/logging_network.py
0141961eadf9246ce287d7ee0dbbbf5243fd6e1332e09ccd2f6a591be419c2c3 tests/message_latency_network.py
@@ -98,7 +102,7 @@ dfc3429e3e755d4c99ec1bd8ee6fb0f1f6aaf86414d5adaca7e9e3b18c7fe3f7 tests/logging_
a8a01faf51bf82bce6cbb2616cdaabb6337047d1c5f17d5f39975c0528a1d5ae tests/muc_flow.rs
b24bd920f096576c2009ff1ccc0366eb8734c5f0b2feb39f04ad7087268a94b9 tests/muc_history.rs
1b34ab721eec69295e1274e44bfd108c0e117dd383624cb6c98d851eb20e7a9a tests/muc_latency_storage.rs
1f3df5f3f39143adc2b2a419fac0cd74b2229337ffa86570261d1f6ac61137a1 tests/muc_native.rs
18f18362f26710043b51168675daee91ae88c071fb44e48210b49686011f8722 tests/muc_native.rs
3df2736ccfd25a6cab290df80ceb7474ea44bb02af6cf768a9216d6b8f7f8630 tests/muc_optional_admin.rs
5dc93407a658b45e504b89e26cfc8f495cd28ad90d0a308b7d228a472435ff9f tests/muc_protocol.rs
ac1caa62a3cd91e74bd7100a5e542c404e97aec61a34e9627900b458c2ee3806 tests/muc_recovery.rs
@@ -124,7 +128,7 @@ b8dd71798f96931f7ab4489cdb1e258556f41411815ef38d3a03e80e4e8fbdfc tests/push_dis
44c8e642f725be2292dfc5282feac8772ac34824927b63058f03f7c98285e5b6 tests/push_latency_network.py
abf5e6a32ab542ccba5861eaaf7aed9a86b8306abfe2c329646d54e43176dc0c tests/reference/COPYING
1ecd0a5c122910cbd066922c95ff0c32e945a07e2c9b0a37a30e5f035b5ecea6 tests/reference/configmanager.lua
f407c6565fa73b6653e7e73204daa65ae1d87111e82c6b96154aea76497ad65d tests/reference_prosody.rs
78fb639a334846c884f2d438407d7fa8491778b695f9e6218a161ebf8acaeced tests/reference_prosody.rs
5ab78e9319d6004d65d35cd4d04dbb99d4ea5ffd8da00473d874bb26000fce35 tests/resource_rebind.py
2e79de5ef8d3be81ac8f715bb96797b6e8e39cdcd61c13d8d873a96b53cb52e4 tests/resource_sessions.rs
cefd1001c9619456402223fa89e449f65599c4397f523860e965ca192a4fd915 tests/sasl2_bind.py
@@ -135,9 +139,12 @@ a7d1519f4f44cc7bfad3d20bcfdf455e92abdd084feed493ffc9e0267a387740 tests/sasl_net
4748fa227743686f78a0fa0364735d2dda5995f8d70a79ad03c1aa1decb42978 tests/sasl_retry_network.py
d8882b8aabd6ff603c81fbbc259eefa68772afb2b3eec2e003ac29efd65d146b tests/sasl_web_network.py
7d69c14ced7884450b9a1ff63cd984cdb9dfe948d6c56631c7593b6fabdabf7f tests/session_routing_isolation.rs
7636b0767fdf24e3d4bde03779ec40309ed2b3301d4e35c6ae5f59aa065e4b2e tests/slow_storage_crash_network.py
7492523bb226b68f72b97b028a7fafeb4fb5375b1efc6fb516400279dd0743d4 tests/slow_storage_latency_network.py
d9eef5c37fcd99404619d61dbf7a2bf7673087924d93e69b08769e716afe297b tests/sm_durable_slots.rs
0b74b2ff47b8ae681b5613bc87e914fdf8e5033156180df06ff2174df2781a5a tests/sm_restart_network.py
72f2d60e47e5295b82d497289b78925a64a1d986c70c277015ae228d211d32bb tests/storage_accounts.rs
b4c9422cdba8fd133572c0fa3a0ae6cda14edbd14db6d984d76f4409eefd1262 tests/sm_journal.rs
8fa5de6a2291a7c35e7bbbc6334d42b4e0836a36bec252d0ef4703e8ee17aecb tests/sm_restart_network.py
f76ad9de4c7e0140b1a4c3da06e849e28bb0562cf9b576ed85a5257f7394506b tests/storage_accounts.rs
5c38afd5f77b5774552d12c461902ffaae21de11735560c6f0088e3dda8afa58 tests/storage_compat.rs
8724991fdf22471952cf1a0fac863abe837069c93d3c8d2fa751af4f6bbc3e38 tests/tls_binding.rs
5bea9b5546d3a6ae4fe78d17acbc3f6e6a1d5e1d13b32f2aa88d0e320892b7c5 tests/tls_plus_network.rs
+79
View File
@@ -0,0 +1,79 @@
# Проверка 0.8.3
2026-10-21, Linux x86_64, Rust 1.99.1. Итоговый единый прогон после заморозки
исходников: [verify.txt](validation-logs-0.8.3/verify.txt).
[Machine-readable results](validation-results-0.8.3.json).
|Проверка|Результат|
|---|---|
|Rust ordinary tests|334 passed, 0 failed, 11 ignored|
|fmt / Clippy all-targets -D warnings / release|Успешно|
|Новые Rust regressions|13 archive redo + 2 rewrite crash boundaries + 14 SM journal, успешно|
|Native reference|8 module/storage + 1 TLS/SASL EXTERNAL federation, успешно|
|Archive >128 MiB|Отдельно выполненный ignored streaming test, успешно|
|Copy/restore и rewrite|Новый inode, потеря suffix, baseline0/nonzero, crash до/после rename — успешно|
|Все network regressions verify-local.sh|Успешно, включая Android, WSS/BOSH, PEP, push, MUC crash/SM resume, upload/migration|
|Slow durable IO|0/100 мс: 3 samples/scenario; 200 мс: 1 sample/scenario; exact MAM/carbons/queues и phase logs проверены|
|SIGKILL сразу после sender SM ACK|100 мс/sync, 3 сообщения, оба MAM архива, 4 resume, 9 replay, до reauth; успешно|
|Recovery sync EIO|WAL/redo: отказ handoff без удаления journal; WAL startup до listeners; retry и данные проверены|
|Offline storage-checkpoint|4 SM exports, 2 archives, durable journal removal, повтор идемпотентен|
|Общая нагрузка|500 TLS, 100 accounts, 5000 ping, 5000 messages, 10000 MAM, failures=[]|
|MUC+SM нагрузка|500 resources, 10 rooms, 25000 presence, 1500 deliveries, SIGKILL и 10 resume; успешно|
## Личная доставка
Медианы для 3 samples, при 200 мс — одна проба; числа в миллисекундах.
|Sync delay / сценарий|Body 0.8.2 → 0.8.3|Sender SM ACK 0.8.2 → 0.8.3|Проб|
|---|---:|---:|---:|
|0 мс / 1:1|25.11 → 7.87|33.41 → 15.08|3|
|0 мс / PEP → 1:1|29.71 → 20.94|32.25 → 21.86|3|
|100 мс / 1:1|1232.47 → 419.68|1641.60 → 644.08|3|
|100 мс / PEP → 1:1|2683.29 → 1378.23|3195.21 → 1581.98|3|
|200 мс / 1:1|2836.64 → 2042.56|3643.30 → 2441.99|1|
|200 мс / PEP → 1:1|5782.02 → 2676.28|6595.92 → 3094.95|1|
0 мс — отдельная последовательная quiet pair, без full verify. Финальные
100/200 мс выполнялись при фоновой работе verify; это не изолированный CPU benchmark.
Первый append включает создание/migration MAM journal. В обычном warm сообщении
6 sync вместо 16; первый — 12. Окна счётчиков могут включать heartbeat, а PEP
сценарий включает предшествующую publication. Реальные fsync/fdatasync сохранены;
interposer замедляет только isolated data_path, не меняет host mount/DNS.
Raw samples, hashes, sync counts и phase fields — validation-logs-0.8.3/*.json.
В production log 0.8.2 серверный handler занимал 4792–5549 мс; старых полей
недостаточно, чтобы доказать конкретный IO stage. Новый таймер разделяет MAM,
доставку, carbons и handled, без payload. Эти loopback результаты не доказывают
время отображения реальным Conversations/Monocles на wthome.ru.
## Нагрузка и сопоставимость
Сопоставимый повтор 0.8.2 в текущей среде: total 151.334 с,
join 104.226 с. Финальный 0.8.3: total 128.380 с,
join 94.493 с, restart 7.557 с,
peak RSS 40.87 MiB. По одной пробе; фоновые условия
различаются, вывод о CPU efficiency не делается. Исторические 45.54 с из другой
среды не являются baseline этого сравнения. Все счётчики доставки и replay
проверены, а не только открытие порта.
## Durability и native handoff
SM WAL и archive redo авторитетны после аварии. Recovery durably adopts полный
валидный journal перед публикацией; EIO regression выявляла отсутствие barrier
в предварительном кандидате a5f5b3a…, теперь проходит на final binary. Отчёты
preliminary-recovery-sync-* сохранены именно как предварительные, не результаты
0.8.2 или final. Изначальный native MUC fixture делал in-place append без передачи
владения: теперь он явно вызывает checkpoint_all перед оригинальным Prosody.
Неjournaled suffix при активном redo намеренно не объявляется external commit.
Copy/restore с новым inode сохраняет авторитет redo; внутренний rewrite
фиксирует старый native и удаление redo перед atomic replacement. Проверены
оба crash boundaries, потеря suffix после копирования и обе baseline длины.
Перед native writer/откатом требуется остановка и `storage-checkpoint CONFIG`.
[Формат, ограничения и эксплуатация](DURABLE-MESSAGE-IO.md).
Release SHA-256: `8ceb8de3a2d02bc4399b8ae817dff7e3c3fe9bff83dc0c6669b7f19210a3847a`.
Один final verify с XMPP_LOAD_CHECK=1 и native env; отдельно archive ignored test
и подробные latency/crash/EIO fixtures. Исторические результаты сохранены.
Оставшийся AF_UNIX ignored test не запускался; 10 GiB upload и реальный MinIO/AWS,
Docker build/run, GitHub/Gitea publish и настоящие APNs/FCM не повторялись.
Общие границы полной совместимости: [COMPATIBILITY.md](COMPATIBILITY.md).
+68 -40
View File
@@ -1,51 +1,79 @@
# Проверка0.8.2
# Проверка 0.8.3
2026-10-21, Linux x86_64, Rust1.99.1. Итоговый единый прогон:
[verify.txt](validation-logs-0.8.2/verify.txt). Machine-readable report:
[validation-results.json](validation-results.json).
2026-10-21, Linux x86_64, Rust 1.99.1. Итоговый единый прогон после заморозки
исходников: [verify.txt](validation-logs-0.8.3/verify.txt).
[Machine-readable results](validation-results-0.8.3.json).
|Проверка|Результат|
|---|---|
|Rust ordinary tests|305passed,0failed,11ignored|
|Rust ordinary tests|334 passed, 0 failed, 11 ignored|
|fmt / Clippy all-targets -D warnings / release|Успешно|
|Новые Rust regressions|6archive endpoints/recovery/limits +2account-lock/duplicateACK — успешно|
|Личная доставка|6fixtures:100000MAM/user, roster0/5000/10000,500SMpending, default/native CSI — успешно|
|Неподвижный push gateway|Настоящий STARTTLS; hibernating+offline: bothmessagesdurable, senderACKh=4 до ответа gateway; offline сохранён до первого postTLSstream|
|Bind2 и MAM metadata|100000records,5reconnects+5IQ, exactIDs/timestamps — успешно|
|Сетевые регрессии|Весь verify-local.sh, включаяprivate WSS/BOSH chat, PEP/push,AndroidSASL,MUCcrash/SMrestart,S3/migration — успешно|
|Native reference|8module/storage +1TLS/SASL EXTERNAL federation — успешно|
|Общая нагрузка|500TLS,100accounts,5000ping,5000messages,10000MAM,failures=[]|
|MUC+SM нагрузка|500resources,10rooms,25000presence,1500messages,SIGKILL+10resume — успешно|
|Новые Rust regressions|13 archive redo + 2 rewrite crash boundaries + 14 SM journal, успешно|
|Native reference|8 module/storage + 1 TLS/SASL EXTERNAL federation, успешно|
|Archive >128 MiB|Отдельно выполненный ignored streaming test, успешно|
|Copy/restore и rewrite|Новый inode, потеря suffix, baseline0/nonzero, crash до/после rename — успешно|
|Все network regressions verify-local.sh|Успешно, включая Android, WSS/BOSH, PEP, push, MUC crash/SM resume, upload/migration|
|Slow durable IO|0/100 мс: 3 samples/scenario; 200 мс: 1 sample/scenario; exact MAM/carbons/queues и phase logs проверены|
|SIGKILL сразу после sender SM ACK|100 мс/sync, 3 сообщения, оба MAM архива, 4 resume, 9 replay, до reauth; успешно|
|Recovery sync EIO|WAL/redo: отказ handoff без удаления journal; WAL startup до listeners; retry и данные проверены|
|Offline storage-checkpoint|4 SM exports, 2 archives, durable journal removal, повтор идемпотентен|
|Общая нагрузка|500 TLS, 100 accounts, 5000 ping, 5000 messages, 10000 MAM, failures=[]|
|MUC+SM нагрузка|500 resources, 10 rooms, 25000 presence, 1500 deliveries, SIGKILL и 10 resume; успешно|
MUC+SM total 45.54с, peak sampled RSS 40.94MiB.
## Личная доставка
Воспроизведение личного push-stall: прежний0.8.1 за2с сохраняет только первое
сообщение и не отвечаетSMACK при зависшем postTLSfeatures gateway. Новый0.8.2
сохраняет оба и отвечаетh=4 в том же2сокне; тестовое окно не является latency SLA.
Offline ordering baseline проверен на0.8.0; его точный hash указан отдельно.
Медианы для 3 samples, при 200 мс — одна проба; числа в миллисекундах.
Bind2 authenticate→success median2742.353→1.976мс; metadataIQ2437.945→0.355мс.
Baseline0.8.0hash74882319… содержит тот же прежний scan path. Все raw samples,
IDs, timestamps и hashes — validation-logs-0.8.2/bind2-*.json.
|Sync delay / сценарий|Body 0.8.2 → 0.8.3|Sender SM ACK 0.8.2 → 0.8.3|Проб|
|---|---:|---:|---:|
|0 мс / 1:1|25.11 → 7.87|33.41 → 15.08|3|
|0 мс / PEP → 1:1|29.71 → 20.94|32.25 → 21.86|3|
|100 мс / 1:1|1232.47 → 419.68|1641.60 → 644.08|3|
|100 мс / PEP → 1:1|2683.29 → 1378.23|3195.21 → 1581.98|3|
|200 мс / 1:1|2836.64 → 2042.56|3643.30 → 2441.99|1|
|200 мс / PEP → 1:1|5782.02 → 2676.28|6595.92 → 3094.95|1|
Default inactive OMEMO доставлен за2.58–4.06мс, receipts7.36–35.56мс в checked
fixtures; explicit[] сохраняет удержание доactive. Намеренный CSI dwell2с
отдельно обозначен в отчёте и не выдаётся за время вычисления/доставки.
Каждый personalarchive содержит ровно100523records/user. Детали и настройка:
[PERSONAL-MESSAGE-LATENCY.md](PERSONAL-MESSAGE-LATENCY.md).
0 мс — отдельная последовательная quiet pair, без full verify. Финальные
100/200 мс выполнялись при фоновой работе verify; это не изолированный CPU benchmark.
Первый append включает создание/migration MAM journal. В обычном warm сообщении
6 sync вместо 16; первый — 12. Окна счётчиков могут включать heartbeat, а PEP
сценарий включает предшествующую publication. Реальные fsync/fdatasync сохранены;
interposer замедляет только isolated data_path, не меняет host mount/DNS.
Raw samples, hashes, sync counts и phase fields — validation-logs-0.8.3/*.json.
Release SHA-256: `130431025c4c7c2f72e28b77a31c83867f857ef8bb56d1f71b10e4b19d7caaf5`. Все новые final reports содержат этот hash.
Команда: `XMPP_LOAD_CHECK=1 bash scripts/verify-local.sh` с native env согласно
scripts/prosody_compat/README.md. Использованы CARGO_INCREMENTAL=0,
CARGO_PROFILE_DEV_DEBUG=0, CARGO_PROFILE_TEST_DEBUG=0. Перед итоговым прогоном
устранены stale package linker artifacts очисткой package build outputs;
исходники не менялись после финальной сборки. Воспроизведение прошло одним
последующим полным прогоном; preliminary compiler errors не являются test failures.
В production log 0.8.2 серверный handler занимал 4792–5549 мс; старых полей
недостаточно, чтобы доказать конкретный IO stage. Новый таймер разделяет MAM,
доставку, carbons и handled, без payload. Эти loopback результаты не доказывают
время отображения реальным Conversations/Monocles на wthome.ru.
Это loopback, не production клиента/диска/сети. Fsync и durable commit до
подтверждения сохранены; boundedpushqueue не обещает durable notification delivery.
SMglobalIOmutex, полные MAMqueries и maintenance остаются отдельной стоимостью.
9nativeignoredtests выполнены; остальные AF_UNIX/largearchiveignoredtests и
10GiBtransfers в этом локальном прогоне не запускались (полный CI entrypoint
по-прежнему включает их). Старые largeupload результаты не приписываются0.8.2.
Общие границы: [COMPATIBILITY.md](COMPATIBILITY.md).
## Нагрузка и сопоставимость
Сопоставимый повтор 0.8.2 в текущей среде: total 151.334 с,
join 104.226 с. Финальный 0.8.3: total 128.380 с,
join 94.493 с, restart 7.557 с,
peak RSS 40.87 MiB. По одной пробе; фоновые условия
различаются, вывод о CPU efficiency не делается. Исторические 45.54 с из другой
среды не являются baseline этого сравнения. Все счётчики доставки и replay
проверены, а не только открытие порта.
## Durability и native handoff
SM WAL и archive redo авторитетны после аварии. Recovery durably adopts полный
валидный journal перед публикацией; EIO regression выявляла отсутствие barrier
в предварительном кандидате a5f5b3a…, теперь проходит на final binary. Отчёты
preliminary-recovery-sync-* сохранены именно как предварительные, не результаты
0.8.2 или final. Изначальный native MUC fixture делал in-place append без передачи
владения: теперь он явно вызывает checkpoint_all перед оригинальным Prosody.
Неjournaled suffix при активном redo намеренно не объявляется external commit.
Copy/restore с новым inode сохраняет авторитет redo; внутренний rewrite
фиксирует старый native и удаление redo перед atomic replacement. Проверены
оба crash boundaries, потеря suffix после копирования и обе baseline длины.
Перед native writer/откатом требуется остановка и `storage-checkpoint CONFIG`.
[Формат, ограничения и эксплуатация](DURABLE-MESSAGE-IO.md).
Release SHA-256: `8ceb8de3a2d02bc4399b8ae817dff7e3c3fe9bff83dc0c6669b7f19210a3847a`.
Один final verify с XMPP_LOAD_CHECK=1 и native env; отдельно archive ignored test
и подробные latency/crash/EIO fixtures. Исторические результаты сохранены.
Оставшийся AF_UNIX ignored test не запускался; 10 GiB upload и реальный MinIO/AWS,
Docker build/run, GitHub/Gitea publish и настоящие APNs/FCM не повторялись.
Общие границы полной совместимости: [COMPATIBILITY.md](COMPATIBILITY.md).
+2
View File
@@ -0,0 +1,2 @@
Checking rust-xmpp v0.8.3 (/workspace/scratch/362577cdf2e0/recovered/rust-xmpp)
Finished `dev` profile [unoptimized] target(s) in 14.04s
@@ -0,0 +1,35 @@
{
"status": "pass",
"delay_ms_per_sync": 100,
"acknowledged_messages": 3,
"mam_records_per_account": [
3,
3
],
"resumed_resources": 4,
"replayed_personal_deliveries": 9,
"offline_checkpoint": {
"exported_sessions": 4,
"native_archives": 2,
"journals_removed": true,
"second_invocation_idempotent": true
},
"pending_before_reauthentication": [
2,
3,
4,
3
],
"sync_between_publication_and_sigkill": {
"total": 32,
"by_store_kind_op": {
"mam:directory:fsync": 2,
"mam:file:fsync": 10,
"pep:directory:fsync": 2,
"pep:file:fsync": 2,
"sm:file:fdatasync": 16
}
},
"binary_sha256": "8ceb8de3a2d02bc4399b8ae817dff7e3c3fe9bff83dc0c6669b7f19210a3847a",
"scope": "loopback STARTTLS; real delayed fsync/fdatasync; sender SM acknowledgement then SIGKILL; authoritative queues and MAM checked before reauthentication; at-least-once replay"
}
@@ -0,0 +1,9 @@
Compiling rust-xmpp v0.8.3 (/workspace/scratch/362577cdf2e0/recovered/rust-xmpp)
Finished `test` profile [unoptimized] target(s) in 3.26s
Running tests/archive_mam.rs (target/debug/deps/archive_mam-1243a5e0f6034fe2)
running 1 test
test archive_larger_than_128mib_streams ... ok
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 13 filtered out; finished in 2.76s
@@ -0,0 +1,62 @@
{
"clients": 500,
"accounts": 100,
"rooms": 10,
"message_rounds": 3,
"concurrency": 20,
"binary": "/tmp/xmpp-durable-muc-load-ssixqqk9/rust-xmpp",
"binary_sha256": "8ceb8de3a2d02bc4399b8ae817dff7e3c3fe9bff83dc0c6669b7f19210a3847a",
"before": {
"rss_kib": 9516,
"cpu_seconds": 0.01,
"fd_count": 12
},
"connect_seconds": 12.048,
"connect_latency": {
"samples": 500,
"median_ms": 398.044,
"p95_ms": 1086.236,
"max_ms": 1403.7
},
"phase": "completed",
"status": "pass",
"elapsed_seconds": 128.38,
"join_seconds": 94.493,
"joined": {
"rss_kib": 41828,
"cpu_seconds": 55.61,
"fd_count": 513
},
"join_presence_deliveries": 25000,
"slots_joined": {
"count": 500,
"bytes": 309790
},
"message_seconds": 9.585,
"room_messages_sent": 30,
"room_messages_delivered": 1500,
"before_kill": {
"rss_kib": 41740,
"cpu_seconds": 62.6,
"fd_count": 512
},
"slots_before_kill": {
"count": 500,
"bytes": 514030
},
"restart_seconds": 7.557,
"resumed_resources": 10,
"replayed_pending_room_messages": 10,
"membership_groupchat_deliveries_after_resume": 10,
"after_resume": {
"rss_kib": 20896,
"cpu_seconds": 7.09,
"fd_count": 22
},
"slots_after_resume": {
"count": 500,
"bytes": 671070
},
"total_seconds": 128.38,
"peak_rss_kib": 41852
}
@@ -0,0 +1,62 @@
{
"clients": 500,
"accounts": 100,
"rooms": 10,
"message_rounds": 3,
"concurrency": 20,
"binary": "/tmp/xmpp-durable-muc-load-wgcesopv/rust-xmpp",
"binary_sha256": "130431025c4c7c2f72e28b77a31c83867f857ef8bb56d1f71b10e4b19d7caaf5",
"before": {
"rss_kib": 9468,
"cpu_seconds": 0.0,
"fd_count": 12
},
"connect_seconds": 8.677,
"connect_latency": {
"samples": 500,
"median_ms": 337.216,
"p95_ms": 475.311,
"max_ms": 632.696
},
"phase": "completed",
"status": "pass",
"elapsed_seconds": 151.334,
"join_seconds": 104.226,
"joined": {
"rss_kib": 41468,
"cpu_seconds": 39.74,
"fd_count": 513
},
"join_presence_deliveries": 25000,
"slots_joined": {
"count": 500,
"bytes": 309790
},
"message_seconds": 17.857,
"room_messages_sent": 30,
"room_messages_delivered": 1500,
"before_kill": {
"rss_kib": 41464,
"cpu_seconds": 48.58,
"fd_count": 512
},
"slots_before_kill": {
"count": 500,
"bytes": 514030
},
"restart_seconds": 10.083,
"resumed_resources": 10,
"replayed_pending_room_messages": 10,
"membership_groupchat_deliveries_after_resume": 10,
"after_resume": {
"rss_kib": 20660,
"cpu_seconds": 9.73,
"fd_count": 22
},
"slots_after_resume": {
"count": 500,
"bytes": 671070
},
"total_seconds": 151.334,
"peak_rss_kib": 41476
}
@@ -0,0 +1,6 @@
{
"status": "fail",
"cases": [],
"binary_sha256": "a5f5b3a552be44fb8ab8b6525dc228bc2dc97f81aecfbc559e12cc29b3e8c00b",
"failure": ".rredo recovery skipped journal sync and falsely reported successful handoff"
}
@@ -0,0 +1,6 @@
{
"status": "fail",
"cases": [],
"binary_sha256": "a5f5b3a552be44fb8ab8b6525dc228bc2dc97f81aecfbc559e12cc29b3e8c00b",
"failure": ".wal recovery skipped journal sync and falsely reported successful handoff"
}
@@ -0,0 +1,95 @@
{
"binary_source": "/workspace/scratch/362577cdf2e0/recovered/rust-xmpp/target/release/rust-xmpp",
"binary_sha256": "8ceb8de3a2d02bc4399b8ae817dff7e3c3fe9bff83dc0c6669b7f19210a3847a",
"scope": "real loopback TCP/STARTTLS/SASL2 Bind2; isolated personal MAM fixture; no production latency claim",
"timing": "raw ordered samples; auth timer starts before authenticate and ends at parsed success; full timer includes TCP/STARTTLS/features; metadata IQ timer runs from send to parsed result",
"result": {
"status": "pass",
"seeded_personal_archive_records": 10000,
"archive_bytes": 2138890,
"archive_expires_after": "never",
"index_present_before_start": false,
"reconnect_samples": 2,
"expected_metadata": {
"start": {
"id": "seed-0",
"timestamp": "2026-01-01T00:00:00Z"
},
"end": {
"id": "seed-9999",
"timestamp": "2026-01-01T02:46:39Z"
}
},
"observed_metadata": [
{
"start": {
"id": "seed-0",
"timestamp": "2026-01-01T00:00:00Z"
},
"end": {
"id": "seed-9999",
"timestamp": "2026-01-01T02:46:39Z"
}
},
{
"start": {
"id": "seed-0",
"timestamp": "2026-01-01T00:00:00Z"
},
"end": {
"id": "seed-9999",
"timestamp": "2026-01-01T02:46:39Z"
}
}
],
"mam_metadata_iq": {
"samples_ms": [
1.066574000105902,
0.6786549997741531
],
"median_ms": 0.8726144999400276,
"p95_ms": 1.066574000105902,
"max_ms": 1.066574000105902
},
"observed_iq_metadata": [
{
"start": {
"id": "seed-0",
"timestamp": "2026-01-01T00:00:00Z"
},
"end": {
"id": "seed-9999",
"timestamp": "2026-01-01T02:46:39Z"
}
},
{
"start": {
"id": "seed-0",
"timestamp": "2026-01-01T00:00:00Z"
},
"end": {
"id": "seed-9999",
"timestamp": "2026-01-01T02:46:39Z"
}
}
],
"sasl2_bind2_success": {
"samples_ms": [
5.694160000075499,
4.350195999904827
],
"median_ms": 5.022177999990163,
"p95_ms": 5.694160000075499,
"max_ms": 5.694160000075499
},
"tcp_starttls_to_bind2_success": {
"samples_ms": [
51.93691700014824,
9.144003000074008
],
"median_ms": 30.540460000111125,
"p95_ms": 51.93691700014824,
"max_ms": 51.93691700014824
}
}
}
@@ -0,0 +1,418 @@
{
"binary_sha256": "8ceb8de3a2d02bc4399b8ae817dff7e3c3fe9bff83dc0c6669b7f19210a3847a",
"scope": "loopback STARTTLS; fixture only; forced CSI dwell explicitly separated",
"cases": [
{
"active": {
"delivery": {
"samples": 1,
"median_ms": 42.84763399982694,
"p95_ms": 42.84763399982694,
"max_ms": 42.84763399982694,
"samples_ms": [
42.84763399982694
]
},
"sender_sm_handled": {
"samples": 1,
"median_ms": 41.45799199977773,
"p95_ms": 41.45799199977773,
"max_ms": 41.45799199977773,
"samples_ms": [
41.45799199977773
]
},
"client_receipt_end_to_end": {
"samples": 1,
"median_ms": 51.320845000191184,
"p95_ms": 51.320845000191184,
"max_ms": 51.320845000191184,
"samples_ms": [
51.320845000191184
]
},
"receipt_return_leg": {
"samples": 1,
"median_ms": 8.362257000044337,
"p95_ms": 8.362257000044337,
"max_ms": 8.362257000044337,
"samples_ms": [
8.362257000044337
]
}
},
"inactive": {
"delivery": {
"samples": 1,
"median_ms": 7.461921999947663,
"p95_ms": 7.461921999947663,
"max_ms": 7.461921999947663,
"samples_ms": [
7.461921999947663
]
},
"sender_sm_handled": {
"samples": 1,
"median_ms": 6.771670000034646,
"p95_ms": 6.771670000034646,
"max_ms": 6.771670000034646,
"samples_ms": [
6.771670000034646
]
},
"client_receipt_end_to_end": {
"samples": 1,
"median_ms": 12.700967000000674,
"p95_ms": 12.700967000000674,
"max_ms": 12.700967000000674,
"samples_ms": [
12.700967000000674
]
},
"receipt_return_leg": {
"samples": 1,
"median_ms": 5.201973000112048,
"p95_ms": 5.201973000112048,
"max_ms": 5.201973000112048,
"samples_ms": [
5.201973000112048
]
}
},
"receipt_only-active": {
"delivery_ms": 23.464398000214715,
"sender_sm_handled_ms": 23.388774000068224,
"durably_csi_buffered": false,
"forced_dwell_seconds": 0,
"after_activation_ms": null
},
"receipt_only-inactive": {
"delivery_ms": 11.619346000316,
"sender_sm_handled_ms": 11.170642000251974,
"durably_csi_buffered": false,
"forced_dwell_seconds": 0,
"after_activation_ms": null
},
"omemo_legacy_only-active": {
"delivery_ms": 3.3123490002253675,
"sender_sm_handled_ms": 3.2682549999663024,
"durably_csi_buffered": false,
"forced_dwell_seconds": 0,
"after_activation_ms": null
},
"omemo_legacy_only-inactive": {
"delivery_ms": 5.355494000014005,
"sender_sm_handled_ms": 4.585807000239583,
"durably_csi_buffered": false,
"forced_dwell_seconds": 0,
"after_activation_ms": null
},
"omemo_2_only-active": {
"delivery_ms": 6.6929370000252675,
"sender_sm_handled_ms": 5.942287999914697,
"durably_csi_buffered": false,
"forced_dwell_seconds": 0,
"after_activation_ms": null
},
"omemo_2_only-inactive": {
"delivery_ms": 7.484531000045536,
"sender_sm_handled_ms": 7.103434999862657,
"durably_csi_buffered": false,
"forced_dwell_seconds": 0,
"after_activation_ms": null
},
"omemo_with_eme-active": {
"delivery_ms": 8.187372000065807,
"sender_sm_handled_ms": 7.642809000117268,
"durably_csi_buffered": false,
"forced_dwell_seconds": 0,
"after_activation_ms": null
},
"omemo_with_eme-inactive": {
"delivery_ms": 8.972958999947878,
"sender_sm_handled_ms": 8.508876000178134,
"durably_csi_buffered": false,
"forced_dwell_seconds": 0,
"after_activation_ms": null
},
"marker_received-active": {
"delivery_ms": 7.674009000311344,
"sender_sm_handled_ms": 7.3800410000330885,
"durably_csi_buffered": false,
"forced_dwell_seconds": 0,
"after_activation_ms": null
},
"marker_received-inactive": {
"delivery_ms": 5.613406000065879,
"sender_sm_handled_ms": 5.646823999995831,
"durably_csi_buffered": false,
"forced_dwell_seconds": 0,
"after_activation_ms": null
},
"marker_displayed-active": {
"delivery_ms": 5.180439000014303,
"sender_sm_handled_ms": 4.853748000186897,
"durably_csi_buffered": false,
"forced_dwell_seconds": 0,
"after_activation_ms": null
},
"marker_displayed-inactive": {
"delivery_ms": 8.705914999609377,
"sender_sm_handled_ms": 8.28981999984535,
"durably_csi_buffered": false,
"forced_dwell_seconds": 0,
"after_activation_ms": null
},
"marker_acknowledged-active": {
"delivery_ms": 6.910242000230937,
"sender_sm_handled_ms": 6.799871000112034,
"durably_csi_buffered": false,
"forced_dwell_seconds": 0,
"after_activation_ms": null
},
"marker_acknowledged-inactive": {
"delivery_ms": 8.707511000011436,
"sender_sm_handled_ms": 8.398837999720854,
"durably_csi_buffered": false,
"forced_dwell_seconds": 0,
"after_activation_ms": null
},
"low_priority_headline-active": {
"delivery_ms": 4.267847999926744,
"sender_sm_handled_ms": 4.236566000145103,
"durably_csi_buffered": false,
"forced_dwell_seconds": 0,
"after_activation_ms": null
},
"low_priority_headline-inactive": {
"delivery_ms": 108.03854200003116,
"sender_sm_handled_ms": 2.853955999853497,
"durably_csi_buffered": true,
"forced_dwell_seconds": 0.1,
"after_activation_ms": 3.106009000021004
},
"sm_backlog": {
"seeded_messages": 30,
"durable_pending_before_sample": 30,
"delivery_ms": 8.070619000136503,
"sender_sm_handled_ms": 7.959626000229036
},
"status": "pass",
"final_personal_mam_records_per_user": [
1045,
1045
],
"csi_policy": "default",
"baseline_expect_held": false,
"roster_contacts_per_user": 0,
"seeded_personal_mam_records_per_user": 1000,
"archive_policy": "always"
},
{
"active": {
"delivery": {
"samples": 1,
"median_ms": 9.351153999887174,
"p95_ms": 9.351153999887174,
"max_ms": 9.351153999887174,
"samples_ms": [
9.351153999887174
]
},
"sender_sm_handled": {
"samples": 1,
"median_ms": 9.169040999950084,
"p95_ms": 9.169040999950084,
"max_ms": 9.169040999950084,
"samples_ms": [
9.169040999950084
]
},
"client_receipt_end_to_end": {
"samples": 1,
"median_ms": 13.487032000284671,
"p95_ms": 13.487032000284671,
"max_ms": 13.487032000284671,
"samples_ms": [
13.487032000284671
]
},
"receipt_return_leg": {
"samples": 1,
"median_ms": 4.101098000319325,
"p95_ms": 4.101098000319325,
"max_ms": 4.101098000319325,
"samples_ms": [
4.101098000319325
]
}
},
"inactive": {
"delivery": {
"samples": 1,
"median_ms": 5.198949999794422,
"p95_ms": 5.198949999794422,
"max_ms": 5.198949999794422,
"samples_ms": [
5.198949999794422
]
},
"sender_sm_handled": {
"samples": 1,
"median_ms": 5.164645999684581,
"p95_ms": 5.164645999684581,
"max_ms": 5.164645999684581,
"samples_ms": [
5.164645999684581
]
},
"client_receipt_end_to_end": {
"samples": 1,
"median_ms": 13.508577999800764,
"p95_ms": 13.508577999800764,
"max_ms": 13.508577999800764,
"samples_ms": [
13.508577999800764
]
},
"receipt_return_leg": {
"samples": 1,
"median_ms": 8.226426999954128,
"p95_ms": 8.226426999954128,
"max_ms": 8.226426999954128,
"samples_ms": [
8.226426999954128
]
}
},
"receipt_only-active": {
"delivery_ms": 8.37234799973885,
"sender_sm_handled_ms": 7.964516999891202,
"durably_csi_buffered": false,
"forced_dwell_seconds": 0,
"after_activation_ms": null
},
"receipt_only-inactive": {
"delivery_ms": 109.20343799989496,
"sender_sm_handled_ms": 5.452117000004364,
"durably_csi_buffered": true,
"forced_dwell_seconds": 0.1,
"after_activation_ms": 2.879265000046871
},
"omemo_legacy_only-active": {
"delivery_ms": 4.961447999903612,
"sender_sm_handled_ms": 4.207620999750361,
"durably_csi_buffered": false,
"forced_dwell_seconds": 0,
"after_activation_ms": null
},
"omemo_legacy_only-inactive": {
"delivery_ms": 107.77542400001039,
"sender_sm_handled_ms": 4.299592999814195,
"durably_csi_buffered": true,
"forced_dwell_seconds": 0.1,
"after_activation_ms": 1.5704200000072888
},
"omemo_2_only-active": {
"delivery_ms": 3.2654749998073385,
"sender_sm_handled_ms": 3.3216599999832397,
"durably_csi_buffered": false,
"forced_dwell_seconds": 0,
"after_activation_ms": null
},
"omemo_2_only-inactive": {
"delivery_ms": 109.47085399993739,
"sender_sm_handled_ms": 5.282463000185089,
"durably_csi_buffered": true,
"forced_dwell_seconds": 0.1,
"after_activation_ms": 2.073672999813425
},
"omemo_with_eme-active": {
"delivery_ms": 6.262568000238389,
"sender_sm_handled_ms": 6.371549000050436,
"durably_csi_buffered": false,
"forced_dwell_seconds": 0,
"after_activation_ms": null
},
"omemo_with_eme-inactive": {
"delivery_ms": 5.960880999737128,
"sender_sm_handled_ms": 5.667837000146392,
"durably_csi_buffered": false,
"forced_dwell_seconds": 0,
"after_activation_ms": null
},
"marker_received-active": {
"delivery_ms": 7.435425000039686,
"sender_sm_handled_ms": 7.0634520002386125,
"durably_csi_buffered": false,
"forced_dwell_seconds": 0,
"after_activation_ms": null
},
"marker_received-inactive": {
"delivery_ms": 110.35115000004225,
"sender_sm_handled_ms": 5.86408199978905,
"durably_csi_buffered": true,
"forced_dwell_seconds": 0.1,
"after_activation_ms": 2.434945000004518
},
"marker_displayed-active": {
"delivery_ms": 3.6287610000727,
"sender_sm_handled_ms": 3.696032999869203,
"durably_csi_buffered": false,
"forced_dwell_seconds": 0,
"after_activation_ms": null
},
"marker_displayed-inactive": {
"delivery_ms": 112.26732199975231,
"sender_sm_handled_ms": 8.923948999836284,
"durably_csi_buffered": true,
"forced_dwell_seconds": 0.1,
"after_activation_ms": 1.43316000003324
},
"marker_acknowledged-active": {
"delivery_ms": 4.8418340002172044,
"sender_sm_handled_ms": 4.916578000120353,
"durably_csi_buffered": false,
"forced_dwell_seconds": 0,
"after_activation_ms": null
},
"marker_acknowledged-inactive": {
"delivery_ms": 110.98447399990619,
"sender_sm_handled_ms": 5.764318000274216,
"durably_csi_buffered": true,
"forced_dwell_seconds": 0.1,
"after_activation_ms": 3.3109219998550543
},
"low_priority_headline-active": {
"delivery_ms": 4.907228000320174,
"sender_sm_handled_ms": 4.593587000272237,
"durably_csi_buffered": false,
"forced_dwell_seconds": 0,
"after_activation_ms": null
},
"low_priority_headline-inactive": {
"delivery_ms": 106.62631800005329,
"sender_sm_handled_ms": 2.990170999964903,
"durably_csi_buffered": true,
"forced_dwell_seconds": 0.1,
"after_activation_ms": 1.6857880000316072
},
"sm_backlog": {
"seeded_messages": 30,
"durable_pending_before_sample": 30,
"delivery_ms": 8.221272999890061,
"sender_sm_handled_ms": 8.186153000224294
},
"status": "pass",
"final_personal_mam_records_per_user": [
1045,
1045
],
"csi_policy": "native",
"baseline_expect_held": false,
"roster_contacts_per_user": 0,
"seeded_personal_mam_records_per_user": 1000,
"archive_policy": "always"
}
]
}
@@ -0,0 +1,25 @@
{
"status": "pass",
"recipient_state": "hibernating",
"durable_pending_at_gateway_stream": {
"push-first": true,
"push-second": true
},
"observation_seconds": 2,
"elapsed_seconds": 2.013597706999917,
"gateway_post_tls_stream_observed": true,
"gateway_post_tls_stream_at_bound": true,
"blocked_stage": "post-tls-features",
"durable_pending": {
"push-first": true,
"push-second": true
},
"sender_sm_ack_h": [
4
],
"sender_inputs": 4,
"sender_handled_before": 2,
"binary_version": "rust-xmpp 0.8.3 \u2014 experimental XMPP runtime",
"binary_sha256": "8ceb8de3a2d02bc4399b8ae817dff7e3c3fe9bff83dc0c6669b7f19210a3847a",
"scope": "loopback TLS fixture; stalled external XMPP push service; no APNs/FCM"
}
@@ -0,0 +1,25 @@
{
"status": "pass",
"recipient_state": "offline",
"durable_pending_at_gateway_stream": {
"push-first": true,
"push-second": true
},
"observation_seconds": 2,
"elapsed_seconds": 2.1132061280000016,
"gateway_post_tls_stream_observed": true,
"gateway_post_tls_stream_at_bound": true,
"blocked_stage": "post-tls-features",
"durable_pending": {
"push-first": true,
"push-second": true
},
"sender_sm_ack_h": [
4
],
"sender_inputs": 4,
"sender_handled_before": 2,
"binary_version": "rust-xmpp 0.8.3 \u2014 experimental XMPP runtime",
"binary_sha256": "8ceb8de3a2d02bc4399b8ae817dff7e3c3fe9bff83dc0c6669b7f19210a3847a",
"scope": "loopback TLS fixture; stalled external XMPP push service; no APNs/FCM"
}
@@ -0,0 +1,24 @@
{
"status": "pass",
"cases": [
{
"suffix": ".wal",
"status": "pass",
"checkpoint_rejected_sync_failure": true,
"journals_preserved_on_failure": 4,
"serve_rejected_before_listeners": true,
"clean_retry_preserved_sessions": 4,
"clean_retry_preserved_archives": 2
},
{
"suffix": ".rredo",
"status": "pass",
"checkpoint_rejected_sync_failure": true,
"journals_preserved_on_failure": 2,
"serve_rejected_before_listeners": false,
"clean_retry_preserved_sessions": 4,
"clean_retry_preserved_archives": 2
}
],
"binary_sha256": "8ceb8de3a2d02bc4399b8ae817dff7e3c3fe9bff83dc0c6669b7f19210a3847a"
}
@@ -0,0 +1,2 @@
Compiling rust-xmpp v0.8.3 (/workspace/scratch/362577cdf2e0/recovered/rust-xmpp)
Finished `release` profile [optimized] target(s) in 1m 43s
@@ -0,0 +1,99 @@
{
"binary_sha256": "8ceb8de3a2d02bc4399b8ae817dff7e3c3fe9bff83dc0c6669b7f19210a3847a",
"scope": "loopback STARTTLS; four SM resources/two accounts; MAM + pending avatar metadata PEP; fixture-only fsync/fdatasync including data directories; real sync retained; operation windows include background SM heartbeat",
"cases": [
{
"samples": [
{
"phase": "settled",
"round": 0,
"delivery_ms": {
"direct": 2042.56345,
"sent": 2239.283932,
"received": 2442.564925
},
"sender_sm_handled_ms": 2441.985416,
"publication_plus_delivery_ms": 2442.641984,
"sync": {
"total": 12,
"by_store_kind_op": {
"mam:directory:fsync": 2,
"mam:file:fsync": 6,
"sm:file:fdatasync": 4
}
},
"pending_per_resource": [
1,
2,
2,
2
]
},
{
"phase": "avatar_pending",
"round": 0,
"delivery_ms": {
"direct": 2676.283903,
"sent": 2476.200907,
"received": 3095.924978
},
"sender_sm_handled_ms": 3094.949323,
"publication_plus_delivery_ms": 3096.115223,
"sync": {
"total": 15,
"by_store_kind_op": {
"mam:file:fsync": 2,
"pep:directory:fsync": 2,
"pep:file:fsync": 2,
"sm:file:fdatasync": 9
}
},
"pending_per_resource": [
3,
2,
3,
2
]
}
],
"mam_records_per_account": [
102,
102
],
"status": "pass",
"delay_ms_per_sync": 200,
"server_message_processing_slow_events": [
{
"duration_ms": 2440,
"route": "local",
"pre_routing_ms": 0,
"sender_mam_ms": 813,
"recipient_mam_ms": 816,
"delivery_ms": 202,
"carbons_ms": 404,
"offline_push_ms": 0,
"handled_ms": 202,
"target_count": 1,
"carbon_count": 2
},
{
"duration_ms": 1259,
"route": "local",
"pre_routing_ms": 0,
"sender_mam_ms": 204,
"recipient_mam_ms": 208,
"delivery_ms": 223,
"carbons_ms": 410,
"offline_push_ms": 0,
"handled_ms": 211,
"target_count": 1,
"carbon_count": 2
}
],
"server_message_processing_slow_ms": [
2440,
1259
]
}
]
}
@@ -0,0 +1,74 @@
{
"binary_sha256": "130431025c4c7c2f72e28b77a31c83867f857ef8bb56d1f71b10e4b19d7caaf5",
"scope": "loopback STARTTLS; four SM resources/two accounts; MAM + pending avatar PEP; fixture-only fsync/fdatasync including data directories; real sync retained",
"cases": [
{
"samples": [
{
"phase": "settled",
"round": 0,
"delivery_ms": {
"direct": 2836.63993,
"sent": 3238.840305,
"received": 3643.707041
},
"sender_sm_handled_ms": 3643.297905,
"publication_plus_delivery_ms": 3643.772507,
"sync": {
"total": 24,
"by_store_kind_op": {
"mam:directory:fsync": 4,
"mam:file:fsync": 4,
"sm:directory:fsync": 8,
"sm:file:fsync": 8
}
},
"pending_per_resource": [
1,
2,
2,
2
]
},
{
"phase": "avatar_pending",
"round": 0,
"delivery_ms": {
"direct": 5782.018326,
"sent": 5384.85421,
"received": 6596.874338
},
"sender_sm_handled_ms": 6595.924675,
"publication_plus_delivery_ms": 6597.492339,
"sync": {
"total": 30,
"by_store_kind_op": {
"mam:directory:fsync": 4,
"mam:file:fsync": 4,
"pep:directory:fsync": 2,
"pep:file:fsync": 2,
"sm:directory:fsync": 9,
"sm:file:fsync": 9
}
},
"pending_per_resource": [
3,
2,
3,
2
]
}
],
"mam_records_per_account": [
102,
102
],
"status": "pass",
"delay_ms_per_sync": 200,
"server_message_processing_slow_ms": [
3245,
3235
]
}
]
}
@@ -0,0 +1,255 @@
{
"binary_sha256": "8ceb8de3a2d02bc4399b8ae817dff7e3c3fe9bff83dc0c6669b7f19210a3847a",
"scope": "loopback STARTTLS; four SM resources/two accounts; MAM + pending avatar metadata PEP; fixture-only fsync/fdatasync including data directories; real sync retained; operation windows include background SM heartbeat",
"cases": [
{
"samples": [
{
"phase": "settled",
"round": 0,
"delivery_ms": {
"direct": 1083.729337,
"sent": 1183.535171,
"received": 1301.736052
},
"sender_sm_handled_ms": 1300.213796,
"publication_plus_delivery_ms": 1309.067665,
"sync": {
"total": 12,
"by_store_kind_op": {
"mam:directory:fsync": 2,
"mam:file:fsync": 6,
"sm:file:fdatasync": 4
}
},
"pending_per_resource": [
1,
2,
2,
2
]
},
{
"phase": "settled",
"round": 1,
"delivery_ms": {
"direct": 416.413847,
"sent": 521.445561,
"received": 622.692061
},
"sender_sm_handled_ms": 622.557259,
"publication_plus_delivery_ms": 627.341892,
"sync": {
"total": 6,
"by_store_kind_op": {
"mam:file:fsync": 2,
"sm:file:fdatasync": 4
}
},
"pending_per_resource": [
1,
2,
2,
2
]
},
{
"phase": "settled",
"round": 2,
"delivery_ms": {
"direct": 419.683064,
"sent": 551.764241,
"received": 644.751777
},
"sender_sm_handled_ms": 644.082891,
"publication_plus_delivery_ms": 644.882621,
"sync": {
"total": 6,
"by_store_kind_op": {
"mam:file:fsync": 2,
"sm:file:fdatasync": 4
}
},
"pending_per_resource": [
1,
2,
2,
2
]
},
{
"phase": "avatar_pending",
"round": 0,
"delivery_ms": {
"direct": 1376.989787,
"sent": 1273.220821,
"received": 1478.990457
},
"sender_sm_handled_ms": 1477.870242,
"publication_plus_delivery_ms": 1479.132408,
"sync": {
"total": 14,
"by_store_kind_op": {
"mam:file:fsync": 2,
"pep:directory:fsync": 2,
"pep:file:fsync": 2,
"sm:file:fdatasync": 8
}
},
"pending_per_resource": [
3,
2,
3,
2
]
},
{
"phase": "avatar_pending",
"round": 1,
"delivery_ms": {
"direct": 1523.59144,
"sent": 1288.285577,
"received": 1626.311997
},
"sender_sm_handled_ms": 1626.005979,
"publication_plus_delivery_ms": 1626.574287,
"sync": {
"total": 14,
"by_store_kind_op": {
"mam:file:fsync": 2,
"pep:directory:fsync": 2,
"pep:file:fsync": 2,
"sm:file:fdatasync": 8
}
},
"pending_per_resource": [
3,
2,
3,
2
]
},
{
"phase": "avatar_pending",
"round": 2,
"delivery_ms": {
"direct": 1378.231158,
"sent": 1276.210346,
"received": 1582.305858
},
"sender_sm_handled_ms": 1581.975429,
"publication_plus_delivery_ms": 1582.463448,
"sync": {
"total": 15,
"by_store_kind_op": {
"mam:file:fsync": 2,
"pep:directory:fsync": 2,
"pep:file:fsync": 2,
"sm:file:fdatasync": 9
}
},
"pending_per_resource": [
3,
2,
3,
2
]
}
],
"mam_records_per_account": [
106,
106
],
"status": "pass",
"delay_ms_per_sync": 100,
"server_message_processing_slow_events": [
{
"duration_ms": 1298,
"route": "local",
"pre_routing_ms": 0,
"sender_mam_ms": 421,
"recipient_mam_ms": 416,
"delivery_ms": 109,
"carbons_ms": 232,
"offline_push_ms": 0,
"handled_ms": 117,
"target_count": 1,
"carbon_count": 2
},
{
"duration_ms": 621,
"route": "local",
"pre_routing_ms": 0,
"sender_mam_ms": 101,
"recipient_mam_ms": 104,
"delivery_ms": 104,
"carbons_ms": 206,
"offline_push_ms": 0,
"handled_ms": 103,
"target_count": 1,
"carbon_count": 2
},
{
"duration_ms": 638,
"route": "local",
"pre_routing_ms": 0,
"sender_mam_ms": 101,
"recipient_mam_ms": 102,
"delivery_ms": 104,
"carbons_ms": 219,
"offline_push_ms": 1,
"handled_ms": 108,
"target_count": 1,
"carbon_count": 2
},
{
"duration_ms": 625,
"route": "local",
"pre_routing_ms": 1,
"sender_mam_ms": 104,
"recipient_mam_ms": 102,
"delivery_ms": 106,
"carbons_ms": 208,
"offline_push_ms": 0,
"handled_ms": 101,
"target_count": 1,
"carbon_count": 2
},
{
"duration_ms": 756,
"route": "local",
"pre_routing_ms": 1,
"sender_mam_ms": 105,
"recipient_mam_ms": 103,
"delivery_ms": 102,
"carbons_ms": 340,
"offline_push_ms": 0,
"handled_ms": 102,
"target_count": 1,
"carbon_count": 2
},
{
"duration_ms": 617,
"route": "local",
"pre_routing_ms": 1,
"sender_mam_ms": 102,
"recipient_mam_ms": 104,
"delivery_ms": 101,
"carbons_ms": 204,
"offline_push_ms": 0,
"handled_ms": 101,
"target_count": 1,
"carbon_count": 2
}
],
"server_message_processing_slow_ms": [
1298,
621,
638,
625,
756,
617
]
}
]
}
@@ -0,0 +1,350 @@
{
"binary_sha256": "130431025c4c7c2f72e28b77a31c83867f857ef8bb56d1f71b10e4b19d7caaf5",
"scope": "loopback STARTTLS; four SM resources/two accounts; MAM + pending avatar PEP; fixture-only fsync/fdatasync including data directories; real sync retained",
"cases": [
{
"samples": [
{
"phase": "settled",
"round": 0,
"delivery_ms": {
"direct": 13.745165,
"sent": 16.132819,
"received": 18.746494
},
"sender_sm_handled_ms": 18.699594,
"publication_plus_delivery_ms": 18.823329,
"sync": {
"total": 16,
"by_store_kind_op": {
"mam:directory:fsync": 4,
"mam:file:fsync": 4,
"sm:directory:fsync": 4,
"sm:file:fsync": 4
}
},
"pending_per_resource": [
1,
2,
2,
2
]
},
{
"phase": "settled",
"round": 1,
"delivery_ms": {
"direct": 15.636949,
"sent": 18.138746,
"received": 20.807559
},
"sender_sm_handled_ms": 20.705943,
"publication_plus_delivery_ms": 20.843968,
"sync": {
"total": 16,
"by_store_kind_op": {
"mam:directory:fsync": 4,
"mam:file:fsync": 4,
"sm:directory:fsync": 4,
"sm:file:fsync": 4
}
},
"pending_per_resource": [
1,
2,
2,
2
]
},
{
"phase": "settled",
"round": 2,
"delivery_ms": {
"direct": 21.358818,
"sent": 24.788203,
"received": 26.125405
},
"sender_sm_handled_ms": 26.090729,
"publication_plus_delivery_ms": 26.187115,
"sync": {
"total": 16,
"by_store_kind_op": {
"mam:directory:fsync": 4,
"mam:file:fsync": 4,
"sm:directory:fsync": 4,
"sm:file:fsync": 4
}
},
"pending_per_resource": [
1,
2,
2,
2
]
},
{
"phase": "avatar_pending",
"round": 0,
"delivery_ms": {
"direct": 32.67965,
"sent": 29.361139,
"received": 35.870217
},
"sender_sm_handled_ms": 35.819252,
"publication_plus_delivery_ms": 35.991777,
"sync": {
"total": 28,
"by_store_kind_op": {
"mam:directory:fsync": 4,
"mam:file:fsync": 4,
"pep:directory:fsync": 2,
"pep:file:fsync": 2,
"sm:directory:fsync": 8,
"sm:file:fsync": 8
}
},
"pending_per_resource": [
3,
2,
3,
2
]
},
{
"phase": "avatar_pending",
"round": 1,
"delivery_ms": {
"direct": 30.47626,
"sent": 28.128456,
"received": 32.654735
},
"sender_sm_handled_ms": 32.540092,
"publication_plus_delivery_ms": 32.738534,
"sync": {
"total": 28,
"by_store_kind_op": {
"mam:directory:fsync": 4,
"mam:file:fsync": 4,
"pep:directory:fsync": 2,
"pep:file:fsync": 2,
"sm:directory:fsync": 8,
"sm:file:fsync": 8
}
},
"pending_per_resource": [
3,
2,
3,
2
]
},
{
"phase": "avatar_pending",
"round": 2,
"delivery_ms": {
"direct": 30.492529,
"sent": 28.027238,
"received": 33.697165
},
"sender_sm_handled_ms": 33.576139,
"publication_plus_delivery_ms": 33.786933,
"sync": {
"total": 28,
"by_store_kind_op": {
"mam:directory:fsync": 4,
"mam:file:fsync": 4,
"pep:directory:fsync": 2,
"pep:file:fsync": 2,
"sm:directory:fsync": 8,
"sm:file:fsync": 8
}
},
"pending_per_resource": [
3,
2,
3,
2
]
}
],
"mam_records_per_account": [
106,
106
],
"status": "pass",
"delay_ms_per_sync": 0
},
{
"samples": [
{
"phase": "settled",
"round": 0,
"delivery_ms": {
"direct": 1226.407308,
"sent": 1431.404759,
"received": 1837.301696
},
"sender_sm_handled_ms": 1633.73631,
"publication_plus_delivery_ms": 1837.391548,
"sync": {
"total": 18,
"by_store_kind_op": {
"mam:directory:fsync": 4,
"mam:file:fsync": 4,
"sm:directory:fsync": 5,
"sm:file:fsync": 5
}
},
"pending_per_resource": [
1,
2,
2,
2
]
},
{
"phase": "settled",
"round": 1,
"delivery_ms": {
"direct": 1234.414585,
"sent": 1436.607311,
"received": 1642.84668
},
"sender_sm_handled_ms": 1641.598264,
"publication_plus_delivery_ms": 1642.913856,
"sync": {
"total": 16,
"by_store_kind_op": {
"mam:directory:fsync": 4,
"mam:file:fsync": 4,
"sm:directory:fsync": 4,
"sm:file:fsync": 4
}
},
"pending_per_resource": [
1,
2,
2,
2
]
},
{
"phase": "settled",
"round": 2,
"delivery_ms": {
"direct": 1232.468024,
"sent": 1441.824206,
"received": 1872.009731
},
"sender_sm_handled_ms": 1652.92941,
"publication_plus_delivery_ms": 1872.108448,
"sync": {
"total": 18,
"by_store_kind_op": {
"mam:directory:fsync": 4,
"mam:file:fsync": 4,
"sm:directory:fsync": 5,
"sm:file:fsync": 5
}
},
"pending_per_resource": [
1,
2,
2,
2
]
},
{
"phase": "avatar_pending",
"round": 0,
"delivery_ms": {
"direct": 3209.381557,
"sent": 3419.876568,
"received": 3614.69826
},
"sender_sm_handled_ms": 3613.565949,
"publication_plus_delivery_ms": 3615.484237,
"sync": {
"total": 28,
"by_store_kind_op": {
"mam:directory:fsync": 4,
"mam:file:fsync": 4,
"pep:directory:fsync": 2,
"pep:file:fsync": 2,
"sm:directory:fsync": 8,
"sm:file:fsync": 8
}
},
"pending_per_resource": [
3,
2,
3,
2
]
},
{
"phase": "avatar_pending",
"round": 1,
"delivery_ms": {
"direct": 2665.684412,
"sent": 2461.378505,
"received": 2870.784696
},
"sender_sm_handled_ms": 2870.455915,
"publication_plus_delivery_ms": 2871.000834,
"sync": {
"total": 28,
"by_store_kind_op": {
"mam:directory:fsync": 4,
"mam:file:fsync": 4,
"pep:directory:fsync": 2,
"pep:file:fsync": 2,
"sm:directory:fsync": 8,
"sm:file:fsync": 8
}
},
"pending_per_resource": [
3,
2,
3,
2
]
},
{
"phase": "avatar_pending",
"round": 2,
"delivery_ms": {
"direct": 2683.292251,
"sent": 2475.225974,
"received": 3636.635597
},
"sender_sm_handled_ms": 3195.209226,
"publication_plus_delivery_ms": 3640.372412,
"sync": {
"total": 30,
"by_store_kind_op": {
"mam:directory:fsync": 4,
"mam:file:fsync": 4,
"pep:directory:fsync": 2,
"pep:file:fsync": 2,
"sm:directory:fsync": 9,
"sm:file:fsync": 9
}
},
"pending_per_resource": [
3,
2,
3,
2
]
}
],
"mam_records_per_account": [
106,
106
],
"status": "pass",
"delay_ms_per_sync": 100
}
]
}
+111
View File
@@ -0,0 +1,111 @@
Compiling rust-xmpp v0.8.3 (/workspace/scratch/362577cdf2e0/recovered/rust-xmpp)
Finished `test` profile [unoptimized] target(s) in 29.67s
Running unittests src/lib.rs (target/debug/deps/rust_xmpp-792bf902c86621ec)
running 50 tests
test csi::checkpoint_tests::failed_queue_checkpoints_restore_append_and_drain_without_copying_queue ... ok
test federation::tests::external_must_be_in_the_advertised_mechanisms ... ok
test federation::tests::starttls_must_be_an_immediate_stream_feature ... ok
test csi::checkpoint_tests::only_queue_changes_require_a_checkpoint ... ok
test federation::tests::federation_domains_are_dns_names_and_case_insensitive ... ok
test federation::tests::stream_restarts_compare_canonical_domains ... ok
test http_config::tests::literal_pchars_and_percent_encoding_are_supported ... ok
test http_config::tests::invalid_paths_types_urls_and_conflicts_are_errors ... ok
test http_config::tests::external_prefix_and_standard_port ... ok
test http_config::tests::routes_hosts_and_default_without_overriding_known_hosts ... ok
test http_proxy::tests::config_validation_and_network_boundaries ... ok
test http_proxy::tests::empty_native_lua_table_disables_proxy_trust ... ok
test http_config::tests::mounts_inherit_per_app_and_expand_virtual_host ... ok
test http_proxy::tests::forwarded_tls_termination_cannot_upgrade_a_plaintext_chain_hop ... ok
test http_proxy::tests::malformed_forwarded_and_unknown_identifiers_are_ignored ... ok
test http_proxy::tests::untrusted_socket_ignores_spoofed_headers ... ok
test http_proxy::tests::legacy_chain_uses_rightmost_untrusted_and_tls_is_preserved ... ok
test http_proxy::tests::forwarded_mode_stops_at_first_untrusted_hop ... ok
test http_transport::tests::bosh_processing_capacity_is_bounded ... ok
test http_transport::tests::configured_mount_host_origin_and_secure_dispatch ... ok
test http_transport::tests::independent_framing ... ok
test http_transport::tests::websocket_preserves_inherited_namespaces_and_spaced_declarations ... ok
test logging::tests::console_aliases_share_one_sink_with_combined_levels ... ok
test sasl::tests::reject_malformed_and_downgrade ... ok
test archive::redo_rewrite_tests::crash_gap_after_redo_retirement_before_native_rename_keeps_old_snapshot ... ok
test archive::redo_rewrite_tests::crash_gap_after_native_rename_has_no_old_redo_to_resurrect_deletions ... ok
test sasl::tests::rfc5802_sha1_vector ... ok
test server::authorization_tests::http_interfaces_follow_secure_override_wildcards_and_global_fallback ... ok
test server::authorization_tests::pep_diagnostics_count_only_first_item_devices_without_logging_ids ... ok
test server::authorization_tests::pep_diagnostics_exclude_payloads_and_identifiers ... ok
test http_service::tests::upload_claim_refreshes_completion_state_after_an_earlier_snapshot ... ok
test sessions::push_timeout_tests::first_important_push_shortens_extended_hibernation_once ... ok
test sessions::tests::acknowledgement_wraps ... ok
test sessions::tests::graceful_disconnect_is_not_resumable ... ok
test sessions::tests::resume_replays_only_unacked_and_checks_owner ... ok
test sessions::tests::traffic_before_sm_enable_is_not_counted ... ok
test upload_migration::tests::local_spool_checks_size_and_hash_and_preserves_source ... ok
test sasl::tests::sha256_independent_python_vector ... ok
test wire::tests::arbitrary_prefix_and_inherited_namespaces ... ok
test wire::tests::cdata_and_comments_frame_without_counting_brackets ... ok
test wire::tests::declarations_rejected ... ok
test wire::tests::fragmented_and_pipelined ... ok
test wire::tests::invalid_namespace_and_declarations_rejected ... ok
test upload_migration::tests::pending_and_corrupt_journals_block_startup ... ok
test http_service::tests::upload_internal_mount_is_independent_of_advertised_proxy_prefix ... ok
test http_service::tests::abandoned_slots_release_quota_without_pruning_existing_or_active_files ... ok
test sasl::tests::plus_channel_binding_and_ssdp_are_authenticated ... ok
test server::authorization_tests::upload_requires_registered_local_account_or_explicit_remote_access ... ok
test sasl::tests::wrong_proof_nonce_binding_and_duplicates ... ok
test admin::tests::real_admin_xml_protocol_over_duplex ... ok
test result: ok. 50 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.18s
Running tests/archive_edges.rs (target/debug/deps/archive_edges-da556b3567110d0b)
running 6 tests
test missing_empty_whitespace_and_single_record ... ok
test endpoint_read_recovers_partial_append_before_seeking ... ok
test multiline_endpoints_match_native_scanner_with_blank_lines ... ok
test oversized_prefix_and_malformed_endpoints_fail ... ok
test metadata_skips_large_malformed_middle_without_creating_cache ... ok
test tail_padding_respects_record_bound ... ok
test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 2.26s
Running tests/archive_redo.rs (target/debug/deps/archive_redo-d041d27370ba29bd)
running 13 tests
test atomic_native_replace_without_handoff_cannot_discard_committed_frames ... ok
test explicit_native_atomic_handoff_and_missing_baseline_fail_closed ... ok
running 1 test
test durable_handoff_allows_native_in_place_append_and_next_rust_append ... ok
test legacy_rollback_marker_is_migrated_before_redo_append ... ok
test committed_redo_recovers_missing_and_partial_native_without_duplicates ... ok
test redo_process_child ... ok
test killed_writer_retains_accepted_records_after_native_data_loss ... ok
test native_baseline_in_place_edits_are_rejected ... ok
test complete_checksum_corruption_fails_without_modifying_native_or_journal ... ok
test copied_archive_replays_committed_redo_despite_new_native_inode ... ok
test incomplete_final_frame_and_uncommitted_native_suffix_are_discarded ... ok
test byte_limit_rotates_before_frame_count_limit ... ok
test bounded_rotation_recovers_suffix_and_rewrites_never_resurrect_deleted_records ... ok
test result: ok. 13 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.91s
Running tests/sm_journal.rs (target/debug/deps/sm_journal-79fad6209c2b84bb)
running 14 tests
test offline_checkpoint_refuses_live_store_and_preserves_redo ... ok
test complete_corrupt_frame_fails_closed_even_with_valid_export ... ok
test journal_symlink_is_rejected_without_following_target ... ok
test export_failure_after_commit_advances_memory_and_fences_later_writes ... ok
test durable_tombstone_overrides_resurrected_stale_json ... ok
test offline_checkpoint_durably_applies_expiration_and_tombstones ... ok
test legacy_json_migrates_and_baseline_survives_partial_first_append ... ok
test offline_checkpoint_repairs_exports_removes_redo_and_allows_legacy_restart ... ok
test partial_tail_is_discarded_and_subsequent_commits_survive_restart ... ok
test redo_repairs_missing_or_torn_export_before_json_validation ... ok
test superseded_frames_still_require_valid_checksums_json_and_baseline_flags ... ok
test incomplete_baseline_or_magic_never_falls_back_to_json ... ok
test retired_journals_are_collected_in_bounded_batches ... ok
test rotation_preserves_latest_snapshot_and_bounds_redo_size ... ok
test result: ok. 14 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 5.97s
File diff suppressed because it is too large. Load diff
@@ -0,0 +1,169 @@
{
"binary_sha256": "8ceb8de3a2d02bc4399b8ae817dff7e3c3fe9bff83dc0c6669b7f19210a3847a",
"scope": "loopback STARTTLS; four SM resources/two accounts; MAM + pending avatar metadata PEP; fixture-only fsync/fdatasync including data directories; real sync retained; operation windows include background SM heartbeat",
"cases": [
{
"samples": [
{
"phase": "settled",
"round": 0,
"delivery_ms": {
"direct": 18.615095,
"sent": 19.940848,
"received": 26.953392
},
"sender_sm_handled_ms": 26.88493,
"publication_plus_delivery_ms": 27.065981,
"sync": {
"total": 12,
"by_store_kind_op": {
"mam:directory:fsync": 2,
"mam:file:fsync": 6,
"sm:file:fdatasync": 4
}
},
"pending_per_resource": [
1,
2,
2,
2
]
},
{
"phase": "settled",
"round": 1,
"delivery_ms": {
"direct": 6.003256,
"sent": 7.461811,
"received": 10.135311
},
"sender_sm_handled_ms": 9.993133,
"publication_plus_delivery_ms": 10.173245,
"sync": {
"total": 6,
"by_store_kind_op": {
"mam:file:fsync": 2,
"sm:file:fdatasync": 4
}
},
"pending_per_resource": [
1,
2,
2,
2
]
},
{
"phase": "settled",
"round": 2,
"delivery_ms": {
"direct": 7.871244,
"sent": 13.331419,
"received": 15.325955
},
"sender_sm_handled_ms": 15.079213,
"publication_plus_delivery_ms": 15.363148,
"sync": {
"total": 6,
"by_store_kind_op": {
"mam:file:fsync": 2,
"sm:file:fdatasync": 4
}
},
"pending_per_resource": [
1,
2,
2,
2
]
},
{
"phase": "avatar_pending",
"round": 0,
"delivery_ms": {
"direct": 38.754553,
"sent": 25.865646,
"received": 40.65703
},
"sender_sm_handled_ms": 40.40549,
"publication_plus_delivery_ms": 40.769686,
"sync": {
"total": 14,
"by_store_kind_op": {
"mam:file:fsync": 2,
"pep:directory:fsync": 2,
"pep:file:fsync": 2,
"sm:file:fdatasync": 8
}
},
"pending_per_resource": [
3,
2,
3,
2
]
},
{
"phase": "avatar_pending",
"round": 1,
"delivery_ms": {
"direct": 20.942607,
"sent": 20.204899,
"received": 22.05411
},
"sender_sm_handled_ms": 21.856674,
"publication_plus_delivery_ms": 23.679041,
"sync": {
"total": 14,
"by_store_kind_op": {
"mam:file:fsync": 2,
"pep:directory:fsync": 2,
"pep:file:fsync": 2,
"sm:file:fdatasync": 8
}
},
"pending_per_resource": [
3,
2,
3,
2
]
},
{
"phase": "avatar_pending",
"round": 2,
"delivery_ms": {
"direct": 17.980156,
"sent": 18.759152,
"received": 21.369599
},
"sender_sm_handled_ms": 21.332607,
"publication_plus_delivery_ms": 21.518131,
"sync": {
"total": 14,
"by_store_kind_op": {
"mam:file:fsync": 2,
"pep:directory:fsync": 2,
"pep:file:fsync": 2,
"sm:file:fdatasync": 8
}
},
"pending_per_resource": [
3,
2,
3,
2
]
}
],
"mam_records_per_account": [
106,
106
],
"status": "pass",
"delay_ms_per_sync": 0,
"server_message_processing_slow_events": [],
"server_message_processing_slow_ms": []
}
]
}
@@ -0,0 +1,180 @@
{
"binary_sha256": "130431025c4c7c2f72e28b77a31c83867f857ef8bb56d1f71b10e4b19d7caaf5",
"scope": "loopback STARTTLS; four SM resources/two accounts; MAM + pending avatar metadata PEP; fixture-only fsync/fdatasync including data directories; real sync retained; operation windows include background SM heartbeat",
"cases": [
{
"samples": [
{
"phase": "settled",
"round": 0,
"delivery_ms": {
"direct": 26.330895,
"sent": 29.575091,
"received": 33.552895
},
"sender_sm_handled_ms": 33.407559,
"publication_plus_delivery_ms": 33.592558,
"sync": {
"total": 16,
"by_store_kind_op": {
"mam:directory:fsync": 4,
"mam:file:fsync": 4,
"sm:directory:fsync": 4,
"sm:file:fsync": 4
}
},
"pending_per_resource": [
1,
2,
2,
2
]
},
{
"phase": "settled",
"round": 1,
"delivery_ms": {
"direct": 25.112248,
"sent": 31.688103,
"received": 39.725842
},
"sender_sm_handled_ms": 39.683719,
"publication_plus_delivery_ms": 39.867826,
"sync": {
"total": 16,
"by_store_kind_op": {
"mam:directory:fsync": 4,
"mam:file:fsync": 4,
"sm:directory:fsync": 4,
"sm:file:fsync": 4
}
},
"pending_per_resource": [
1,
2,
2,
2
]
},
{
"phase": "settled",
"round": 2,
"delivery_ms": {
"direct": 11.894093,
"sent": 11.977544,
"received": 13.638711
},
"sender_sm_handled_ms": 13.371964,
"publication_plus_delivery_ms": 13.720665,
"sync": {
"total": 16,
"by_store_kind_op": {
"mam:directory:fsync": 4,
"mam:file:fsync": 4,
"sm:directory:fsync": 4,
"sm:file:fsync": 4
}
},
"pending_per_resource": [
1,
2,
2,
2
]
},
{
"phase": "avatar_pending",
"round": 0,
"delivery_ms": {
"direct": 42.768972,
"sent": 32.889933,
"received": 48.207819
},
"sender_sm_handled_ms": 47.576405,
"publication_plus_delivery_ms": 51.552818,
"sync": {
"total": 28,
"by_store_kind_op": {
"mam:directory:fsync": 4,
"mam:file:fsync": 4,
"pep:directory:fsync": 2,
"pep:file:fsync": 2,
"sm:directory:fsync": 8,
"sm:file:fsync": 8
}
},
"pending_per_resource": [
3,
2,
3,
2
]
},
{
"phase": "avatar_pending",
"round": 1,
"delivery_ms": {
"direct": 28.176101,
"sent": 26.386914,
"received": 29.822606
},
"sender_sm_handled_ms": 29.714999,
"publication_plus_delivery_ms": 30.011235,
"sync": {
"total": 28,
"by_store_kind_op": {
"mam:directory:fsync": 4,
"mam:file:fsync": 4,
"pep:directory:fsync": 2,
"pep:file:fsync": 2,
"sm:directory:fsync": 8,
"sm:file:fsync": 8
}
},
"pending_per_resource": [
3,
2,
3,
2
]
},
{
"phase": "avatar_pending",
"round": 2,
"delivery_ms": {
"direct": 29.714366,
"sent": 26.01394,
"received": 32.367508
},
"sender_sm_handled_ms": 32.252144,
"publication_plus_delivery_ms": 33.076522,
"sync": {
"total": 28,
"by_store_kind_op": {
"mam:directory:fsync": 4,
"mam:file:fsync": 4,
"pep:directory:fsync": 2,
"pep:file:fsync": 2,
"sm:directory:fsync": 8,
"sm:file:fsync": 8
}
},
"pending_per_resource": [
3,
2,
3,
2
]
}
],
"mam_records_per_account": [
106,
106
],
"status": "pass",
"delay_ms_per_sync": 0,
"server_message_processing_slow_events": [],
"server_message_processing_slow_ms": []
}
]
}
File diff suppressed because it is too large. Load diff
+209
View File
@@ -0,0 +1,209 @@
{
"version": "0.8.3",
"date_utc": "2026-10-21",
"release_binary_sha256": "8ceb8de3a2d02bc4399b8ae817dff7e3c3fe9bff83dc0c6669b7f19210a3847a",
"ordinary_tests": {
"passed": 334,
"failed": 0,
"ignored": 11
},
"fmt": "passed",
"clippy": "passed",
"release": "passed",
"full_verify": "passed in one final frozen-source run",
"native_module_tests": 8,
"native_federation_tests": 1,
"extra_large_archive_tests": 1,
"ignored_not_run": [
"AF_UNIX admin_socket"
],
"load": {
"general_tls_clients": 500,
"general_accounts": 100,
"muc_sm": {
"clients": 500,
"accounts": 100,
"rooms": 10,
"message_rounds": 3,
"concurrency": 20,
"binary": "/tmp/xmpp-durable-muc-load-ssixqqk9/rust-xmpp",
"binary_sha256": "8ceb8de3a2d02bc4399b8ae817dff7e3c3fe9bff83dc0c6669b7f19210a3847a",
"before": {
"rss_kib": 9516,
"cpu_seconds": 0.01,
"fd_count": 12
},
"connect_seconds": 12.048,
"connect_latency": {
"samples": 500,
"median_ms": 398.044,
"p95_ms": 1086.236,
"max_ms": 1403.7
},
"phase": "completed",
"status": "pass",
"elapsed_seconds": 128.38,
"join_seconds": 94.493,
"joined": {
"rss_kib": 41828,
"cpu_seconds": 55.61,
"fd_count": 513
},
"join_presence_deliveries": 25000,
"slots_joined": {
"count": 500,
"bytes": 309790
},
"message_seconds": 9.585,
"room_messages_sent": 30,
"room_messages_delivered": 1500,
"before_kill": {
"rss_kib": 41740,
"cpu_seconds": 62.6,
"fd_count": 512
},
"slots_before_kill": {
"count": 500,
"bytes": 514030
},
"restart_seconds": 7.557,
"resumed_resources": 10,
"replayed_pending_room_messages": 10,
"membership_groupchat_deliveries_after_resume": 10,
"after_resume": {
"rss_kib": 20896,
"cpu_seconds": 7.09,
"fd_count": 22
},
"slots_after_resume": {
"count": 500,
"bytes": 671070
},
"total_seconds": 128.38,
"peak_rss_kib": 41852
}
},
"personal_delivery_comparisons": [
{
"delay_ms_per_sync": 0,
"phase": "settled",
"samples_per_binary": 3,
"body_before_ms": 25.112248,
"body_after_ms": 7.871244,
"sender_ack_before_ms": 33.407559,
"sender_ack_after_ms": 15.079213
},
{
"delay_ms_per_sync": 0,
"phase": "avatar_pending",
"samples_per_binary": 3,
"body_before_ms": 29.714366,
"body_after_ms": 20.942607,
"sender_ack_before_ms": 32.252144,
"sender_ack_after_ms": 21.856674
},
{
"delay_ms_per_sync": 100,
"phase": "settled",
"samples_per_binary": 3,
"body_before_ms": 1232.468024,
"body_after_ms": 419.683064,
"sender_ack_before_ms": 1641.598264,
"sender_ack_after_ms": 644.082891
},
{
"delay_ms_per_sync": 100,
"phase": "avatar_pending",
"samples_per_binary": 3,
"body_before_ms": 2683.292251,
"body_after_ms": 1378.231158,
"sender_ack_before_ms": 3195.209226,
"sender_ack_after_ms": 1581.975429
},
{
"delay_ms_per_sync": 200,
"phase": "settled",
"samples_per_binary": 1,
"body_before_ms": 2836.63993,
"body_after_ms": 2042.56345,
"sender_ack_before_ms": 3643.297905,
"sender_ack_after_ms": 2441.985416
},
{
"delay_ms_per_sync": 200,
"phase": "avatar_pending",
"samples_per_binary": 1,
"body_before_ms": 5782.018326,
"body_after_ms": 2676.283903,
"sender_ack_before_ms": 6595.924675,
"sender_ack_after_ms": 3094.949323
}
],
"durability_crash": {
"status": "pass",
"delay_ms_per_sync": 100,
"acknowledged_messages": 3,
"mam_records_per_account": [
3,
3
],
"resumed_resources": 4,
"replayed_personal_deliveries": 9,
"offline_checkpoint": {
"exported_sessions": 4,
"native_archives": 2,
"journals_removed": true,
"second_invocation_idempotent": true
},
"pending_before_reauthentication": [
2,
3,
4,
3
],
"sync_between_publication_and_sigkill": {
"total": 32,
"by_store_kind_op": {
"mam:directory:fsync": 2,
"mam:file:fsync": 10,
"pep:directory:fsync": 2,
"pep:file:fsync": 2,
"sm:file:fdatasync": 16
}
},
"binary_sha256": "8ceb8de3a2d02bc4399b8ae817dff7e3c3fe9bff83dc0c6669b7f19210a3847a",
"scope": "loopback STARTTLS; real delayed fsync/fdatasync; sender SM acknowledgement then SIGKILL; authoritative queues and MAM checked before reauthentication; at-least-once replay"
},
"recovery_sync_faults": {
"status": "pass",
"cases": [
{
"suffix": ".wal",
"status": "pass",
"checkpoint_rejected_sync_failure": true,
"journals_preserved_on_failure": 4,
"serve_rejected_before_listeners": true,
"clean_retry_preserved_sessions": 4,
"clean_retry_preserved_archives": 2
},
{
"suffix": ".rredo",
"status": "pass",
"checkpoint_rejected_sync_failure": true,
"journals_preserved_on_failure": 2,
"serve_rejected_before_listeners": false,
"clean_retry_preserved_sessions": 4,
"clean_retry_preserved_archives": 2
}
],
"binary_sha256": "8ceb8de3a2d02bc4399b8ae817dff7e3c3fe9bff83dc0c6669b7f19210a3847a"
},
"limitations": [
"Loopback fixture, not production client or disk qualification",
"0ms pair ran sequentially without full verify; 100/200ms final runs had background final verify",
"One200ms sample; startup/migration/rotation and background heartbeat affect sync counts",
"SM global IO mutex and full snapshots remain; no atomic transaction spanning MAM/SM/network",
"Offline storage-checkpoint required before legacy/native writer handoff",
"10GiB transfers, real MinIO/AWS, Docker/Gitea/GitHub run were not repeated in this release"
]
}
+182 -1667
View File
File diff suppressed because it is too large. Load diff
+3
View File
@@ -30,6 +30,9 @@ python3 tests/preauth_network.py target/release/rust-xmpp
python3 tests/muc_restart_network.py target/release/rust-xmpp
python3 tests/muc_crash_network.py target/release/rust-xmpp
python3 tests/sm_restart_network.py target/release/rust-xmpp
python3 tests/slow_storage_latency_network.py --binary target/release/rust-xmpp --delay-ms 0 25 --rounds 1 --output /tmp/slow-storage-latency.json
python3 tests/slow_storage_crash_network.py --binary target/release/rust-xmpp --delay-ms 25 --messages 3 --output /tmp/slow-storage-crash.json
python3 tests/journal_recovery_sync_network.py --binary target/release/rust-xmpp --output /tmp/journal-recovery-sync.json
python3 tests/push_latency_network.py --binary target/release/rust-xmpp --recipient hibernating --output /tmp/push-latency.json
python3 tests/push_latency_network.py --binary target/release/rust-xmpp --recipient offline --output /tmp/push-offline-latency.json
python3 tests/personal_message_latency_network.py --binary target/release/rust-xmpp --rounds 1 --history-records 1000 --roster-sizes 0 --backlog 30 --csi-dwell 0.1 --output /tmp/personal-message-latency.json
+380 -20
View File
@@ -174,7 +174,103 @@ pub fn scan_store<F: FnMut(Entry) -> Result<()>>(
let _guard = lock.lock().map_err(|_| anyhow!("archive lock poisoned"))?;
scan_unlocked(root, host, user, store, visit)
}
fn recover_append(p: &Path) -> Result<()> {
// The native file remains directly readable by Prosody. While this sidecar exists,
// its complete frames define the committed suffix: do not edit a live native file
// in place or replace it until checkpoint_all has durably retired the journal.
const REDO_MAGIC: &[u8; 8] = b"XARRED01";
const REDO_HEADER: usize = 96;
const REDO_LIMIT: u64 = 1024 * 1024;
const REDO_FRAMES: usize = 128;
#[derive(Clone)]
struct RedoState {
length: u64,
bytes: u64,
frames: usize,
}
#[derive(Clone, PartialEq, Eq)]
struct FileStamp {
len: u64,
modified: Option<std::time::SystemTime>,
identity: (u64, u64),
#[cfg(unix)]
changed: (i64, i64),
}
fn identity(meta: &std::fs::Metadata) -> (u64, u64) {
#[cfg(unix)]
{
use std::os::unix::fs::MetadataExt;
(meta.dev(), meta.ino())
}
#[cfg(not(unix))]
{
let _ = meta;
(0, 0)
}
}
fn stamp(p: &Path) -> Result<FileStamp> {
let m = std::fs::metadata(p)?;
Ok(FileStamp {
len: m.len(),
modified: m.modified().ok(),
identity: identity(&m),
#[cfg(unix)]
changed: {
use std::os::unix::fs::MetadataExt;
(m.ctime(), m.ctime_nsec())
},
})
}
type RedoCache = std::collections::HashMap<std::path::PathBuf, (FileStamp, FileStamp, RedoState)>;
static REDO_CACHE: OnceLock<Mutex<RedoCache>> = OnceLock::new();
fn remember_redo(p: &Path, state: &RedoState) -> Result<()> {
let native = stamp(p)?;
let journal = stamp(&p.with_extension("rredo"))?;
let mut cache = REDO_CACHE
.get_or_init(|| Mutex::new(RedoCache::new()))
.lock()
.map_err(|_| anyhow!("archive redo cache poisoned"))?;
if cache.len() >= 1024 {
cache.clear();
}
cache.insert(p.to_path_buf(), (native, journal, state.clone()));
Ok(())
}
fn digest(data: &[u8]) -> [u8; 32] {
use sha2::{Digest, Sha256};
Sha256::digest(data).into()
}
fn baseline_anchor(file: &mut std::fs::File, length: u64) -> Result<[u8; 32]> {
let count = length.min(4096);
file.seek(SeekFrom::Start(length - count))?;
let mut bytes = vec![0; count as usize];
file.read_exact(&mut bytes)?;
Ok(digest(&bytes))
}
/// Native fsync precedes journal replacement. Either old or new header can
/// survive a crash, and both refer to an already durable native baseline.
fn checkpoint_redo(p: &Path) -> Result<RedoState> {
let mut file = std::fs::OpenOptions::new().read(true).write(true).open(p)?;
file.sync_all()?;
let meta = file.metadata()?;
let length = meta.len();
let (dev, ino) = identity(&meta);
let mut header = Vec::with_capacity(REDO_HEADER);
header.extend_from_slice(REDO_MAGIC);
for n in [length, dev, ino] {
header.extend_from_slice(&n.to_le_bytes());
}
header.extend_from_slice(&baseline_anchor(&mut file, length)?);
header.extend_from_slice(&digest(&header));
storage::atomic_write(&p.with_extension("rredo"), &header)?;
let state = RedoState {
length,
bytes: REDO_HEADER as u64,
frames: 0,
};
remember_redo(p, &state)?;
Ok(state)
}
fn recover_legacy_append(p: &Path) -> Result<()> {
let journal = p.with_extension("rappend");
match std::fs::read(&journal) {
Ok(data) => {
@@ -195,6 +291,194 @@ fn recover_append(p: &Path) -> Result<()> {
}
Ok(())
}
fn recover_redo(p: &Path) -> Result<Option<RedoState>> {
recover_legacy_append(p)?;
let journal = p.with_extension("rredo");
let jm = match stamp(&journal) {
Ok(m) => m,
Err(e)
if e.downcast_ref::<std::io::Error>()
.is_some_and(|e| e.kind() == std::io::ErrorKind::NotFound) =>
{
return Ok(None);
}
Err(e) => return Err(e),
};
let nm = stamp(p).ok();
if let Some(native) = &nm {
let cache = REDO_CACHE
.get_or_init(|| Mutex::new(RedoCache::new()))
.lock()
.map_err(|_| anyhow!("archive redo cache poisoned"))?;
if let Some((n, j, state)) = cache.get(p)
&& n == native
&& j == &jm
{
return Ok(Some(state.clone()));
}
}
// One oversized record is allowed, but the journal never grows without bound.
anyhow::ensure!(
jm.len <= REDO_LIMIT + MAX_RECORD_BYTES as u64 + REDO_HEADER as u64 + 72,
"archive redo journal exceeds bound"
);
let data = std::fs::read(&journal)?;
anyhow::ensure!(
data.len() >= REDO_HEADER && &data[..8] == REDO_MAGIC,
"invalid archive redo header"
);
anyhow::ensure!(
digest(&data[..64]) == data[64..96],
"archive redo header checksum mismatch"
);
let number = |offset: usize| u64::from_le_bytes(data[offset..offset + 8].try_into().unwrap());
let base = number(8);
// Inodes change when an archive is copied/restored. The journal remains
// authoritative across that change; identity is only a cache stamp.
anyhow::ensure!(nm.is_some() || base == 0, "missing archive redo baseline");
let mut opts = std::fs::OpenOptions::new();
opts.read(true).write(true).create(true).truncate(false);
#[cfg(unix)]
{
use std::os::unix::fs::OpenOptionsExt;
opts.mode(0o600);
}
let mut file = opts.open(p)?;
anyhow::ensure!(
file.metadata()?.len() >= base,
"archive shorter than redo baseline"
);
anyhow::ensure!(
baseline_anchor(&mut file, base)? == data[32..64],
"archive redo baseline changed in place"
);
let mut pos = REDO_HEADER;
let mut frames = Vec::new();
let mut length = base;
while pos < data.len() {
if data.len() - pos < 72 {
break;
}
anyhow::ensure!(
digest(&data[pos..pos + 8]) == data[pos + 8..pos + 40],
"archive redo frame length checksum mismatch"
);
let size = number(pos);
anyhow::ensure!(
size > 0 && size <= MAX_RECORD_BYTES as u64,
"invalid archive redo frame length"
);
let end = pos + 72 + size as usize;
if end > data.len() {
break;
}
let payload = &data[pos + 72..end];
anyhow::ensure!(
digest(payload) == data[pos + 40..pos + 72],
"archive redo frame checksum mismatch"
);
frames.push((length, payload));
anyhow::ensure!(
frames.len() <= REDO_FRAMES,
"archive redo frame count exceeds bound"
);
length = length
.checked_add(size)
.ok_or_else(|| anyhow!("archive redo offset overflow"))?;
pos = end;
}
// Native indexes are rebuildable. A crash may have resurrected an unlink
// from the append path; invalidate it again before exposing recovered data.
match std::fs::remove_file(p.with_extension("lidx")) {
Ok(()) => std::fs::File::open(p.parent().unwrap())?.sync_all()?,
Err(e) if e.kind() == std::io::ErrorKind::NotFound => (),
Err(e) => return Err(e.into()),
}
// A complete frame can survive a process crash in page cache even if its
// writer never completed sync. Durably adopt all validated frames before
// exposing them or allowing later recovery to truncate the native file.
let redo = std::fs::OpenOptions::new().write(true).open(&journal)?;
if pos < data.len() {
redo.set_len(pos as u64)?;
}
redo.sync_all()?;
// Validate the complete journal before changing native data. Replaying is
// idempotent, including after a second crash midway through recovery.
let mut changed = file.metadata()?.len() != length;
for (offset, payload) in &frames {
file.seek(SeekFrom::Start(*offset))?;
let mut native = vec![0; payload.len()];
let equal = file.read_exact(&mut native).is_ok() && native.as_slice() == *payload;
if !equal {
file.seek(SeekFrom::Start(*offset))?;
file.write_all(payload)?;
changed = true;
}
}
if changed {
file.set_len(length)?;
file.sync_all()?;
}
let state = RedoState {
length,
bytes: pos as u64,
frames: frames.len(),
};
if nm.is_none() {
return checkpoint_redo(p).map(Some);
}
remember_redo(p, &state)?;
Ok(Some(state))
}
fn recover_append(p: &Path) -> Result<()> {
recover_redo(p)?;
Ok(())
}
/// Durable handoff to a native writer. Call after stopping archive users and
/// before letting Prosody modify `.list` files in place. Abrupt shutdown leaves
/// redo recovery active; unjournaled native suffixes are then uncommitted.
pub fn checkpoint_all(root: &Path) -> Result<usize> {
with_exclusive_access(|| {
let mut pending = vec![root.to_path_buf()];
let mut count = 0;
while let Some(dir) = pending.pop() {
let entries = match std::fs::read_dir(&dir) {
Ok(entries) => entries,
Err(e) if e.kind() == std::io::ErrorKind::NotFound => continue,
Err(e) => return Err(e.into()),
};
for entry in entries {
let entry = entry?;
let kind = entry.file_type()?;
if kind.is_dir() {
pending.push(entry.path());
} else if kind.is_file() && entry.path().extension().is_some_and(|e| e == "rredo") {
let path = entry.path().with_extension("list");
prepare_rewrite(&path)?;
count += 1;
}
}
}
Ok(count)
})
}
/// Retire redo before an atomic snapshot replacement. The old native snapshot
/// is synced first, so a crash after unlink and before rename retains every
/// committed record. After rename there is no old redo to resurrect deletions.
fn prepare_rewrite(p: &Path) -> Result<()> {
if recover_redo(p)?.is_none() {
return Ok(());
}
std::fs::File::open(p)?.sync_all()?;
std::fs::remove_file(p.with_extension("rredo"))?;
std::fs::File::open(p.parent().unwrap())?.sync_all()?;
REDO_CACHE
.get_or_init(|| Mutex::new(RedoCache::new()))
.lock()
.map_err(|_| anyhow!("archive redo cache poisoned"))?
.remove(p);
Ok(())
}
fn scan_unlocked<F: FnMut(Entry) -> Result<()>>(
root: &Path,
host: &str,
@@ -458,7 +742,7 @@ fn save_unlocked(
data.push_str(&encode_entry(e)?);
}
let p = storage::path(root, host, user, store).with_extension("list");
recover_append(&p)?;
prepare_rewrite(&p)?;
match std::fs::remove_file(p.with_extension("lidx")) {
Ok(()) => (),
Err(e) if e.kind() == std::io::ErrorKind::NotFound => (),
@@ -519,37 +803,57 @@ fn append_unlocked(
entry: &Entry,
) -> Result<()> {
let data = encode_entry(entry)?;
anyhow::ensure!(
data.len() <= MAX_RECORD_BYTES,
"archive record exceeds 16 MiB"
);
let path = storage::path(root, host, user, store).with_extension("list");
let parent = path.parent().unwrap();
std::fs::create_dir_all(parent)?;
recover_append(&path)?;
std::fs::create_dir_all(path.parent().unwrap())?;
let state = recover_redo(&path)?;
let mut opts = std::fs::OpenOptions::new();
opts.create(true).append(true);
opts.create(true).read(true).write(true).truncate(false);
#[cfg(unix)]
{
use std::os::unix::fs::OpenOptionsExt;
opts.mode(0o600);
}
let mut file = opts.open(&path)?;
let length = file.metadata()?.len();
let journal = path.with_extension("rappend");
storage::atomic_write(&journal, &length.to_le_bytes())?;
let mut state = match state {
Some(state) => state,
None => checkpoint_redo(&path)?,
};
if state.bytes >= REDO_LIMIT || state.frames >= REDO_FRAMES {
state = checkpoint_redo(&path)?;
}
match std::fs::remove_file(path.with_extension("lidx")) {
Ok(()) => (),
Err(e) if e.kind() == std::io::ErrorKind::NotFound => (),
Err(e) => return Err(e.into()),
}
let result = (|| -> Result<()> {
file.write_all(data.as_bytes())?;
file.sync_all()?;
std::fs::remove_file(&journal)?;
std::fs::File::open(parent)?.sync_all()?;
Ok(())
})();
if result.is_err() {
recover_append(&path)?;
}
result
let mut frame = Vec::with_capacity(72 + data.len());
frame.extend_from_slice(&(data.len() as u64).to_le_bytes());
frame.extend_from_slice(&digest(&(data.len() as u64).to_le_bytes()));
frame.extend_from_slice(&digest(data.as_bytes()));
frame.extend_from_slice(data.as_bytes());
// Invalidate the cache before any write, including error paths. A failed
// sync is indeterminate and must be recovered, never blindly rolled back.
REDO_CACHE
.get_or_init(|| Mutex::new(RedoCache::new()))
.lock()
.map_err(|_| anyhow!("archive redo cache poisoned"))?
.remove(&path);
file.seek(SeekFrom::Start(state.length))?;
file.write_all(data.as_bytes())?;
let mut journal = std::fs::OpenOptions::new()
.append(true)
.open(path.with_extension("rredo"))?;
journal.write_all(&frame)?;
journal.sync_all()?;
state.length += data.len() as u64;
state.bytes += frame.len() as u64;
state.frames += 1;
remember_redo(&path, &state)?;
Ok(())
}
/// Owner is localpart, peer is bare JID. Caller enforces archive preferences.
pub fn archive_message(
@@ -614,6 +918,7 @@ fn retain_unlocked<F: FnMut(&Entry) -> bool>(
mut retain: F,
) -> Result<usize> {
let path = storage::path(root, host, user, store).with_extension("list");
recover_append(&path)?;
if !path.exists() {
return Ok(0);
}
@@ -641,6 +946,7 @@ fn retain_unlocked<F: FnMut(&Entry) -> bool>(
return Ok(0);
}
file.sync_all()?;
prepare_rewrite(&path)?;
match std::fs::remove_file(path.with_extension("lidx")) {
Ok(()) => (),
Err(e) if e.kind() == std::io::ErrorKind::NotFound => (),
@@ -653,3 +959,57 @@ fn retain_unlocked<F: FnMut(&Entry) -> bool>(
let _ = std::fs::remove_file(&temp);
result
}
#[cfg(test)]
mod redo_rewrite_tests {
use super::*;
fn fixture() -> (tempfile::TempDir, std::path::PathBuf) {
let root = tempfile::tempdir().unwrap();
append_new(
root.path(),
"example.org",
Some("alice"),
"archive",
Entry {
id: "committed-before-rewrite".into(),
when: 42.,
peer: String::new(),
xml: "<message><body>durable</body></message>".into(),
},
)
.unwrap();
let path = storage::path(root.path(), "example.org", Some("alice"), "archive")
.with_extension("list");
(root, path)
}
#[test]
fn crash_gap_after_redo_retirement_before_native_rename_keeps_old_snapshot() {
let (root, path) = fixture();
let expected = std::fs::read(&path).unwrap();
prepare_rewrite(&path).unwrap();
// Crash boundary: redo unlink and directory sync completed, replacement
// has not happened. Restart must retain the complete durable old list.
assert!(!path.with_extension("rredo").exists());
assert_eq!(std::fs::read(&path).unwrap(), expected);
let rows = load(root.path(), "example.org", "alice").unwrap();
assert_eq!(rows.len(), 1);
assert_eq!(rows[0].id, "committed-before-rewrite");
}
#[test]
fn crash_gap_after_native_rename_has_no_old_redo_to_resurrect_deletions() {
let (root, path) = fixture();
prepare_rewrite(&path).unwrap();
storage::atomic_write(&path, b"").unwrap();
// Crash boundary: the atomic replacement committed. Recovery sees the
// new snapshot directly and cannot replay the removed old record.
assert!(!path.with_extension("rredo").exists());
assert!(
load(root.path(), "example.org", "alice")
.unwrap()
.is_empty()
);
}
}
+20 -1
View File
@@ -12,6 +12,25 @@ async fn main() {
async fn run() -> Result<()> {
let args: Vec<_> = env::args().skip(1).collect();
match args.first().map(String::as_str) {
Some("storage-checkpoint") if args.len() == 2 => {
let (c, domain) = rust_xmpp::server::settings_config(Path::new(&args[1]))?;
let data = Path::new(
c.option(&domain, "data_path")
.and_then(|v| v.as_str())
.context("data_path missing")?,
);
let _data_guard = rust_xmpp::data_lock::DataDirectoryGuard::acquire(data)?;
let archives = rust_xmpp::archive::checkpoint_all(data)?;
let sm_path = data
.join(rust_xmpp::storage::encode_name(&domain))
.join("rust_smacks");
let sessions = if sm_path.try_exists()? {
rust_xmpp::session_store::checkpoint_exports(&sm_path)?
} else {
0
};
println!("Durable native archive handoff: {archives}; SM JSON exports: {sessions}");
}
Some("upload-migrate") if args.len() == 5 || args.len() == 6 => {
anyhow::ensure!(
args[3] == "--component" && (args.len() == 5 || args[5] == "--execute"),
@@ -121,7 +140,7 @@ async fn run() -> Result<()> {
println!("Account created: {}@{}", args[2], domain);
}
Some("--help") | None => println!(
"rust-xmpp upload-migrate OLD_CONFIG NEW_CONFIG --component HOST [--execute]\nrust-xmpp check-config PATH\nrust-xmpp convert PATH json|yaml|toml OUTPUT\nrust-xmpp --version\n\nExperimental runtime: serve CONFIG --experimental\ncreate-user CONFIG USER --password-stdin\ncheck [CONFIG] (running server healthcheck)\nshell CONFIG COMMAND\n\nSTARTTLS, SCRAM/PLAIN, local messaging, roster, MAM, MUC, PEP/pubsub, stream management, push, federation and HTTP transports are implemented with limitations; see docs/COMPATIBILITY.md."
"rust-xmpp upload-migrate OLD_CONFIG NEW_CONFIG --component HOST [--execute]\nrust-xmpp check-config PATH\nrust-xmpp convert PATH json|yaml|toml OUTPUT\nrust-xmpp --version\nrust-xmpp storage-checkpoint CONFIG (offline native handoff/downgrade)\n\nExperimental runtime: serve CONFIG --experimental\ncreate-user CONFIG USER --password-stdin\ncheck [CONFIG] (running server healthcheck)\nshell CONFIG COMMAND\n\nSTARTTLS, SCRAM/PLAIN, local messaging, roster, MAM, MUC, PEP/pubsub, stream management, push, federation and HTTP transports are implemented with limitations; see docs/COMPATIBILITY.md."
),
_ => bail!("invalid command; use --help"),
}
+42 -1
View File
@@ -3303,6 +3303,12 @@ async fn authenticated_transport<S: AsyncRead + AsyncWrite + Unpin>(
ctx.event(Level::Debug, "stanza-received", &[("kind", kind)]);
let handled_session = session.clone();
let processing_started = Instant::now();
// Fixed, payload-free phase metrics; unvisited phases stay zero for
// implicit destinations, blocked messages and MUC handling.
let mut message_phases = [Duration::ZERO; 7];
let mut personal_message_started = false;
let mut target_count = 0;
let mut carbon_count = 0;
async {
if kind == "iq" {
let typ = n.attribute("type").unwrap_or("");
@@ -3612,7 +3618,10 @@ async fn authenticated_transport<S: AsyncRead + AsyncWrite + Unpin>(
.unwrap_or(to),
);
let typ = n.attribute("type").unwrap_or("normal");
personal_message_started = true;
message_phases[0] = processing_started.elapsed();
if rt.enabled("mam") {
let sender_mam_started = Instant::now();
xml = crate::mam::strip_owner_stanza_ids(&xml, &rt.domain, &user)?;
let now = chrono::Utc::now().timestamp_millis() as f64 / 1000.0;
if crate::mam::should_store(n)
@@ -3637,7 +3646,9 @@ async fn authenticated_transport<S: AsyncRead + AsyncWrite + Unpin>(
} else {
sent_xml = xml.clone();
}
message_phases[1] = sender_mam_started.elapsed();
if let Some(u) = &local {
let recipient_mam_started = Instant::now();
xml = crate::mam::strip_owner_stanza_ids(&xml, &rt.domain, u)?;
if u == &user {
xml = sent_xml.clone();
@@ -3658,14 +3669,17 @@ async fn authenticated_transport<S: AsyncRead + AsyncWrite + Unpin>(
)?;
xml = stamped;
}
message_phases[2] = recipient_mam_started.elapsed();
}
}
let delivery_started = Instant::now();
let remote_delivered = remote && route_delivery(&rt, to, &xml).await.is_ok();
let targets = if local.is_some() {
rt.hub.message_targets_for_type(to, typ)
} else {
vec![]
};
target_count = targets.len();
let mut delivered = usize::from(remote_delivered);
let mut target_jids = vec![];
for target in targets {
@@ -3675,6 +3689,8 @@ async fn authenticated_transport<S: AsyncRead + AsyncWrite + Unpin>(
target_jids.push(target.jid.clone());
}
}
message_phases[3] = delivery_started.elapsed();
let carbons_started = Instant::now();
if rt.enabled("carbons")
&& matches!(typ, "chat" | "normal")
&& !n.children().any(|n| {
@@ -3701,10 +3717,14 @@ async fn authenticated_transport<S: AsyncRead + AsyncWrite + Unpin>(
escape(&s.owner),
escape(&s.jid)
);
let _ = s.transmit(carbon);
if s.transmit(carbon).is_ok() {
carbon_count += 1;
}
}
}
}
message_phases[4] = carbons_started.elapsed();
let offline_push_started = Instant::now();
// A push can wake the client immediately. Commit offline delivery
// before publishing its notification, just as for durable SM output.
if delivered == 0 && !matches!(typ, "error" | "headline") {
@@ -3727,16 +3747,22 @@ async fn authenticated_transport<S: AsyncRead + AsyncWrite + Unpin>(
enqueue_push_route(&rt, to, payload);
}
}
message_phases[5] = offline_push_started.elapsed();
}
Ok::<(), anyhow::Error>(())
}.await?;
if !personal_message_started {
message_phases[0] = processing_started.elapsed();
}
let handled_started = Instant::now();
if let Some(s) = handled_session {
let mut state = s.lock().unwrap();
if !state.closed {
state.mark_handled()?;
}
}
message_phases[6] = handled_started.elapsed();
if kind == "message" && processing_started.elapsed() >= Duration::from_millis(250) {
let route = n.attribute("to").map_or("implicit", |to| {
if is_muc_destination(&rt.config, to) {
@@ -3756,6 +3782,21 @@ async fn authenticated_transport<S: AsyncRead + AsyncWrite + Unpin>(
&processing_started.elapsed().as_millis().to_string(),
),
("route", route),
("pre_routing_ms", &message_phases[0].as_millis().to_string()),
("sender_mam_ms", &message_phases[1].as_millis().to_string()),
(
"recipient_mam_ms",
&message_phases[2].as_millis().to_string(),
),
("delivery_ms", &message_phases[3].as_millis().to_string()),
("carbons_ms", &message_phases[4].as_millis().to_string()),
(
"offline_push_ms",
&message_phases[5].as_millis().to_string(),
),
("handled_ms", &message_phases[6].as_millis().to_string()),
("target_count", &target_count.to_string()),
("carbon_count", &carbon_count.to_string()),
],
);
}
+382 -35
View File
@@ -2,10 +2,12 @@
//!
//! Recovery is deliberately strict: malformed, oversized, or inconsistent committed
//! slots fail startup instead of silently losing acknowledged session state. Uncommitted
//! UUID temporary files are discarded; the last renamed JSON slot remains authoritative.
//! UUID temporary files are discarded; checksummed redo journals are authoritative.
//! The readable JSON files are exports and are repaired from redo before loading.
use crate::sessions::Session;
use anyhow::{Result, ensure};
use serde::{Deserialize, Serialize};
use sha2::{Digest, Sha256};
use std::{
collections::BTreeMap,
fs::{self, File, OpenOptions},
@@ -122,6 +124,9 @@ pub(crate) struct Store {
root: PathBuf,
_lock: File,
slots: Mutex<BTreeMap<String, (String, usize)>>,
failed: std::sync::atomic::AtomicBool,
retired: Mutex<Vec<String>>,
journals: Mutex<BTreeMap<String, u64>>,
}
fn store(session: &Session) -> Result<Option<Arc<Store>>> {
session
@@ -180,6 +185,7 @@ impl Store {
use std::os::unix::fs::OpenOptionsExt;
options.mode(0o600);
}
regular_or_missing(&path.join("store.lock"))?;
let lock = options.open(path.join("store.lock"))?;
#[cfg(unix)]
{
@@ -192,6 +198,9 @@ impl Store {
root: path.into(),
_lock: lock,
slots: Default::default(),
failed: std::sync::atomic::AtomicBool::new(false),
retired: Default::default(),
journals: Default::default(),
});
let mut recovered = vec![];
let mut total = 0usize;
@@ -202,6 +211,7 @@ impl Store {
let name = entry.file_name();
let name = name.to_string_lossy();
if let Some(id) = name.strip_suffix(".tmp") {
let id = id.strip_suffix(".wal").unwrap_or(id);
let owned = safe_id(id).is_ok()
&& uuid::Uuid::parse_str(id).is_ok_and(|uuid| uuid.to_string() == id);
if owned && entry.file_type()?.is_file() {
@@ -210,6 +220,50 @@ impl Store {
}
}
File::open(path)?.sync_all()?;
// Redo precedes JSON validation: the export may be stale or torn.
for entry in &entries {
let name = entry.file_name();
let name = name.to_string_lossy();
if let Some(id) = name.strip_suffix(".wal") {
safe_id(id)?;
ensure!(
entry.file_type()?.is_file(),
"invalid stream management journal"
);
let snapshot = replay(&entry.path())?;
// A process crash can leave a complete, checksummed append
// only in page cache. Make its adoption durable before any
// recovered slot is exposed to bootstrap or resume consumers.
OpenOptions::new()
.write(true)
.open(entry.path())?
.sync_data()?;
{
let mut journals = store.journals.lock().unwrap();
journals.insert(id.to_string(), entry.metadata()?.len());
ensure!(
journals.len() <= SESSION_LIMIT + 64
&& journals.values().sum::<u64>() <= JOURNAL_TOTAL_LIMIT,
"stream management aggregate journal limit"
);
}
if let Some(record) = &snapshot {
validate_record(record, id)?;
}
store.export(
id,
snapshot
.as_ref()
.map(serde_json::to_vec)
.transpose()?
.as_deref(),
)?;
if snapshot.is_none() {
store.retired.lock().unwrap().push(id.to_string());
}
}
}
let entries = fs::read_dir(path)?.collect::<std::io::Result<Vec<_>>>()?;
for entry in entries {
let name = entry.file_name();
let name = name.to_string_lossy();
@@ -237,20 +291,10 @@ impl Store {
.take(SLOT_LIMIT as u64 + 1)
.read_to_end(&mut bytes)?;
let record: Record = serde_json::from_slice(&bytes)?;
ensure!(
record.version == 1
&& record.pending.len() <= 4096
&& record.sent.wrapping_sub(record.acked) as usize == record.pending.len()
&& crate::protocol::bare(&record.jid) == record.owner
&& record.jid.contains('/'),
"invalid durable session"
);
ensure!(
name == format!("{}.json", safe_id(&record.resume_id)?),
"invalid slot token"
);
let id = name.strip_suffix(".json").unwrap();
validate_record(&record, id)?;
if record.deadline_ms <= now_ms() {
fs::remove_file(entry.path())?;
store.commit(id, None)?;
continue;
}
store
@@ -261,6 +305,7 @@ impl Store {
recovered.push(record);
}
File::open(path)?.sync_all()?;
store.collect_retired()?;
Ok((store, recovered))
}
fn write(&self, s: &Session, expected: Option<&str>) -> Result<()> {
@@ -282,7 +327,8 @@ impl Store {
);
} else {
ensure!(
!slots.contains_key(&record.resume_id),
!slots.contains_key(&record.resume_id)
&& !self.retired.lock().unwrap().contains(&record.resume_id),
"duplicate stream token"
);
}
@@ -296,24 +342,7 @@ impl Store {
total - previous + bytes.len() <= TOTAL_LIMIT,
"stream management storage limit"
);
let tmp = self.root.join(format!("{name}.tmp"));
let mut options = OpenOptions::new();
options.write(true).create(true).truncate(true);
#[cfg(unix)]
{
use std::os::unix::fs::OpenOptionsExt;
options.mode(0o600);
}
let mut file = options.open(&tmp)?;
#[cfg(unix)]
{
use std::os::unix::fs::PermissionsExt;
file.set_permissions(fs::Permissions::from_mode(0o600))?;
}
file.write_all(&bytes)?;
file.sync_all()?;
fs::rename(tmp, self.root.join(format!("{name}.json")))?;
File::open(&self.root)?.sync_all()?;
self.commit(name, Some(&bytes))?;
slots.insert(record.resume_id, (record.connection, bytes.len()));
Ok(())
}
@@ -322,12 +351,189 @@ impl Store {
if slots.get(id).is_none_or(|v| v.0 != connection) {
return Ok(());
}
fs::remove_file(self.root.join(format!("{}.json", safe_id(id)?)))?;
File::open(&self.root)?.sync_all()?;
self.commit(safe_id(id)?, None)?;
slots.remove(id);
Ok(())
}
// A journal failure may have reached stable storage. Fence this store rather
// than allowing an in-memory rollback to overwrite an uncertain commit.
fn commit(&self, id: &str, bytes: Option<&[u8]>) -> Result<()> {
use std::sync::atomic::Ordering;
ensure!(
!self.failed.load(Ordering::SeqCst),
"stream management store failed"
);
let result = self.commit_inner(id, bytes);
if result.is_err() {
self.failed.store(true, Ordering::SeqCst);
}
result
}
fn commit_inner(&self, id: &str, bytes: Option<&[u8]>) -> Result<()> {
let path = self.root.join(format!("{id}.wal"));
regular_or_missing(&path)?;
if !path.try_exists()? {
// The baseline embeds the previously committed JSON, so recovery
// never depends on an export modified without a subsequent fsync.
let old = self.root.join(format!("{id}.json"));
let baseline = if old.try_exists()? {
regular_or_missing(&old)?;
let mut baseline = Vec::new();
File::open(old)?
.take(SLOT_LIMIT as u64 + 1)
.read_to_end(&mut baseline)?;
ensure!(baseline.len() <= SLOT_LIMIT, "stream management slot limit");
let record: Record = serde_json::from_slice(&baseline)?;
validate_record(&record, id)?;
Some(baseline)
} else {
None
};
let baseline_frame = frame(true, baseline.as_deref())?;
if self.journals.lock().unwrap().values().sum::<u64>()
+ JOURNAL_MAGIC.len() as u64
+ baseline_frame.len() as u64
> JOURNAL_TOTAL_LIMIT
{
self.compact_journals()?;
}
self.reset_journal(id, baseline.as_deref())?;
}
// Rotation is amortized over at least several small commits. Sync the
// exported snapshot and its rename BEFORE replacing the redo history.
let len = fs::metadata(&path)?.len();
if len >= JOURNAL_ROTATE {
let old = replay(&path)?;
let old = old.as_ref().map(serde_json::to_vec).transpose()?;
self.checkpoint_journal(id, old.as_deref())?;
}
let frame = frame(false, bytes)?;
if self.journals.lock().unwrap().values().sum::<u64>() + frame.len() as u64
> JOURNAL_TOTAL_LIMIT
{
// An aggregate cap prevents many small slots each growing a full
// per-slot rotation budget. Compaction is rare and amortized over
// hundreds of MiB of durable appends.
self.compact_journals()?;
}
let mut journal = private_options().append(true).open(&path)?;
let new_len = journal.metadata()?.len() + frame.len() as u64;
journal.write_all(&frame)?;
journal.sync_data()?;
self.journals
.lock()
.unwrap()
.insert(id.to_string(), new_len);
// This remains readable for diagnostics and wire assertions, but only
// the journal is authoritative after a crash.
if self.export(id, bytes).is_err() {
// The durable commit succeeded. Its caller must advance memory to
// the same state rather than rolling it back. Fence future writes;
// startup will reconstruct the export from redo.
self.failed.store(true, std::sync::atomic::Ordering::SeqCst);
return Ok(());
}
if bytes.is_none() {
let mut retired = self.retired.lock().unwrap();
retired.push(id.to_string());
let collect = retired.len() >= 64;
drop(retired);
if collect && self.collect_retired().is_err() {
self.failed.store(true, std::sync::atomic::Ordering::SeqCst);
}
}
Ok(())
}
fn compact_journals(&self) -> Result<()> {
let ids: Vec<String> = self.journals.lock().unwrap().keys().cloned().collect();
for id in ids {
let snapshot = replay(&self.root.join(format!("{id}.wal")))?;
let snapshot = snapshot.as_ref().map(serde_json::to_vec).transpose()?;
self.checkpoint_journal(&id, snapshot.as_deref())?;
}
Ok(())
}
fn collect_retired(&self) -> Result<()> {
let mut retired = self.retired.lock().unwrap();
if retired.is_empty() {
return Ok(());
}
// Deletions become durable before their redo tombstones disappear.
File::open(&self.root)?.sync_all()?;
for id in retired.iter() {
let path = self.root.join(format!("{id}.wal"));
if path.try_exists()? {
fs::remove_file(path)?;
}
self.journals.lock().unwrap().remove(id);
}
File::open(&self.root)?.sync_all()?;
retired.clear();
Ok(())
}
fn checkpoint_journal(&self, id: &str, bytes: Option<&[u8]>) -> Result<()> {
self.export(id, bytes)?;
if bytes.is_some() {
File::open(self.root.join(format!("{id}.json")))?.sync_all()?;
}
File::open(&self.root)?.sync_all()?;
self.reset_journal(id, bytes)
}
fn reset_journal(&self, id: &str, baseline: Option<&[u8]>) -> Result<()> {
let tmp = self.root.join(format!("{id}.wal.tmp"));
regular_or_missing(&tmp)?;
let mut file = private_options().create(true).truncate(true).open(&tmp)?;
private_permissions(&file)?;
file.write_all(JOURNAL_MAGIC)?;
file.write_all(&frame(true, baseline)?)?;
file.sync_all()?;
let len = file.metadata()?.len();
fs::rename(tmp, self.root.join(format!("{id}.wal")))?;
File::open(&self.root)?.sync_all()?;
self.journals.lock().unwrap().insert(id.to_string(), len);
Ok(())
}
fn export(&self, id: &str, bytes: Option<&[u8]>) -> Result<()> {
let dest = self.root.join(format!("{id}.json"));
if let Some(bytes) = bytes {
let tmp = self.root.join(format!("{id}.tmp"));
regular_or_missing(&tmp)?;
let mut file = private_options().create(true).truncate(true).open(&tmp)?;
private_permissions(&file)?;
file.write_all(bytes)?;
fs::rename(tmp, dest)?;
} else if dest.try_exists()? {
fs::remove_file(dest)?;
}
Ok(())
}
}
/// Materialize the authoritative redo state as durable legacy JSON snapshots.
///
/// This is an offline handoff operation: it acquires the same exclusive store
/// lock as a persistent Hub and fails while a live owner exists. Recovery and
/// expiry run first. All current JSON data and directory entries become durable
/// before any redo journal is removed. A failed checkpoint may leave redundant
/// journals, but never removes the only durable copy of an acknowledged state.
/// The caller should additionally hold its server-wide data-directory lock.
pub fn checkpoint_exports(path: &Path) -> Result<usize> {
let (store, records) = Store::open(path)?;
for record in &records {
let json = path.join(format!("{}.json", safe_id(&record.resume_id)?));
regular_or_missing(&json)?;
File::open(json)?.sync_all()?;
}
File::open(path)?.sync_all()?;
let journals: Vec<String> = store.journals.lock().unwrap().keys().cloned().collect();
for id in journals {
let journal = path.join(format!("{id}.wal"));
regular_or_missing(&journal)?;
fs::remove_file(journal)?;
}
File::open(path)?.sync_all()?;
Ok(records.len())
}
fn safe_id(id: &str) -> Result<&str> {
ensure!(
!id.is_empty()
@@ -337,3 +543,144 @@ fn safe_id(id: &str) -> Result<&str> {
);
Ok(id)
}
const JOURNAL_MAGIC: &[u8; 8] = b"XMPPSM01";
const JOURNAL_ROTATE: u64 = 16 * 1024 * 1024;
const JOURNAL_TOTAL_LIMIT: u64 = (TOTAL_LIMIT * 2) as u64;
const FRAME_LIMIT: usize = SLOT_LIMIT + 1024;
#[derive(Serialize, Deserialize)]
struct JournalEntry<T> {
baseline: bool,
snapshot: Option<T>,
}
fn private_options() -> OpenOptions {
let mut options = OpenOptions::new();
options.write(true);
#[cfg(unix)]
{
use std::os::unix::fs::OpenOptionsExt;
options.mode(0o600);
}
options
}
fn private_permissions(file: &File) -> Result<()> {
#[cfg(unix)]
{
use std::os::unix::fs::PermissionsExt;
file.set_permissions(fs::Permissions::from_mode(0o600))?;
}
Ok(())
}
fn frame(baseline: bool, bytes: Option<&[u8]>) -> Result<Vec<u8>> {
// The snapshot was already validated and serialized by write() or replay().
// Copy its JSON directly into the frame rather than parsing and serializing
// the entire pending queue again on every checkpoint.
if let Some(bytes) = bytes {
ensure!(bytes.len() <= SLOT_LIMIT, "stream management slot limit");
}
let mut payload = Vec::with_capacity(bytes.map_or(4, |snapshot| snapshot.len()) + 32);
payload.extend_from_slice(if baseline {
b"{\"baseline\":true,\"snapshot\":"
} else {
b"{\"baseline\":false,\"snapshot\":"
});
payload.extend_from_slice(bytes.unwrap_or(b"null"));
payload.push(b'}');
ensure!(
payload.len() <= FRAME_LIMIT,
"stream management journal frame limit"
);
let len = payload.len() as u32;
let mut result = Vec::with_capacity(payload.len() + 40);
result.extend_from_slice(&len.to_le_bytes());
result.extend_from_slice(&(!len).to_le_bytes());
result.extend_from_slice(&Sha256::digest(&payload));
result.extend_from_slice(&payload);
Ok(result)
}
fn replay(path: &Path) -> Result<Option<Record>> {
let mut file = OpenOptions::new().read(true).write(true).open(path)?;
private_permissions(&file)?;
ensure!(
file.metadata()?.len() <= JOURNAL_ROTATE + (FRAME_LIMIT + 40) as u64 + 8,
"stream management journal limit"
);
let mut bytes = Vec::new();
(&mut file)
.take(JOURNAL_ROTATE + FRAME_LIMIT as u64 + 49)
.read_to_end(&mut bytes)?;
ensure!(
bytes.starts_with(JOURNAL_MAGIC),
"invalid stream management journal header"
);
let mut offset = 8;
let mut first = true;
let mut latest_payload = None;
while offset < bytes.len() {
if bytes.len() - offset < 8 {
break;
}
let len = u32::from_le_bytes(bytes[offset..offset + 4].try_into().unwrap());
let complement = u32::from_le_bytes(bytes[offset + 4..offset + 8].try_into().unwrap());
ensure!(
len == !complement && len as usize <= FRAME_LIMIT,
"corrupt stream management journal frame"
);
let end = offset + 40 + len as usize;
if end > bytes.len() {
break;
}
let payload = &bytes[offset + 40..end];
ensure!(
Sha256::digest(payload).as_slice() == &bytes[offset + 8..offset + 40],
"stream management journal checksum mismatch"
);
// Historical snapshots were validated before their original commit.
// Check every byte and the full JSON syntax again, without allocating
// old pending queues or serializing superseded snapshots during replay.
let entry: JournalEntry<serde::de::IgnoredAny> = serde_json::from_slice(payload)?;
ensure!(
entry.baseline == first,
"invalid stream management journal baseline"
);
latest_payload = Some(payload);
first = false;
offset = end;
}
ensure!(!first, "incomplete stream management journal baseline");
let latest: JournalEntry<Record> = serde_json::from_slice(latest_payload.unwrap())?;
if let Some(record) = &latest.snapshot {
let id = path.file_stem().unwrap().to_string_lossy();
validate_record(record, &id)?;
ensure!(
serde_json::to_vec(record)?.len() <= SLOT_LIMIT,
"stream management slot limit"
);
}
if offset < bytes.len() {
file.set_len(offset as u64)?;
file.sync_all()?;
}
Ok(latest.snapshot)
}
fn validate_record(record: &Record, id: &str) -> Result<()> {
ensure!(
record.version == 1
&& record.pending.len() <= 4096
&& record.sent.wrapping_sub(record.acked) as usize == record.pending.len()
&& crate::protocol::bare(&record.jid) == record.owner
&& record.jid.contains('/'),
"invalid durable session"
);
ensure!(safe_id(&record.resume_id)? == id, "invalid slot token");
Ok(())
}
fn regular_or_missing(path: &Path) -> Result<()> {
match fs::symlink_metadata(path) {
Ok(metadata) => ensure!(metadata.is_file(), "invalid stream management file"),
Err(error) if error.kind() == std::io::ErrorKind::NotFound => {}
Err(error) => return Err(error.into()),
}
Ok(())
}
+1 -1
View File
@@ -427,7 +427,7 @@ pub fn purge_user(root: &Path, host: &str, user: &str) -> Result<()> {
if !ty.is_dir() {
continue;
}
for extension in ["dat", "list", "lidx", "rappend"] {
for extension in ["dat", "list", "lidx", "rappend", "rredo"] {
let p = entry.path().join(format!("{user}.{extension}"));
match fs::symlink_metadata(&p) {
Ok(m) => {
+347
View File
@@ -0,0 +1,347 @@
//! Crash fixtures model independent loss of unsynced native writes. The redo
//! sidecar is the durable commit source until a native checkpoint or handoff.
use rust_xmpp::{archive, storage};
use std::{
fs,
io::Write,
path::{Path, PathBuf},
process::Command,
time::{Duration, Instant},
};
fn path(root: &Path) -> PathBuf {
storage::path(root, "example.org", Some("alice"), "archive").with_extension("list")
}
fn entry(id: &str) -> archive::Entry {
archive::Entry {
id: id.into(),
when: 42.,
peer: "bob@example.org".into(),
xml: "<message xmlns='jabber:client'><body>hello</body></message>".into(),
}
}
fn append(root: &Path, id: &str) {
archive::append_new(root, "example.org", Some("alice"), "archive", entry(id)).unwrap();
}
fn ids(root: &Path) -> Vec<String> {
archive::load(root, "example.org", "alice")
.unwrap()
.into_iter()
.map(|e| e.id)
.collect()
}
#[test]
fn committed_redo_recovers_missing_and_partial_native_without_duplicates() {
let t = tempfile::tempdir().unwrap();
append(t.path(), "one");
append(t.path(), "two");
let p = path(t.path());
let expected = fs::read(&p).unwrap();
for length in [0, 1, expected.len() / 2, expected.len() - 1] {
fs::OpenOptions::new()
.write(true)
.open(&p)
.unwrap()
.set_len(length as u64)
.unwrap();
assert_eq!(ids(t.path()), ["one", "two"]);
assert_eq!(fs::read(&p).unwrap(), expected);
assert_eq!(ids(t.path()), ["one", "two"]);
}
fs::remove_file(&p).unwrap();
assert_eq!(ids(t.path()), ["one", "two"]);
assert_eq!(fs::read(&p).unwrap(), expected);
}
#[test]
fn incomplete_final_frame_and_uncommitted_native_suffix_are_discarded() {
for extra in [0, 1, 7, 20, 39, 40, 71, 72, 90] {
let t = tempfile::tempdir().unwrap();
append(t.path(), "one");
let p = path(t.path());
let committed = fs::read(&p).unwrap();
let journal = p.with_extension("rredo");
let committed_journal = fs::metadata(&journal).unwrap().len();
append(t.path(), "two");
// Construct a crash before the second frame was fully persisted.
fs::OpenOptions::new()
.write(true)
.open(&journal)
.unwrap()
.set_len(committed_journal + extra)
.unwrap();
assert_eq!(ids(t.path()), ["one"]);
assert_eq!(fs::read(&p).unwrap(), committed);
assert_eq!(fs::metadata(&journal).unwrap().len(), committed_journal);
append(t.path(), "three");
assert_eq!(ids(t.path()), ["one", "three"]);
}
}
#[test]
fn complete_checksum_corruption_fails_without_modifying_native_or_journal() {
for offset in [8, 64, 96, 104, 136, 168] {
let t = tempfile::tempdir().unwrap();
append(t.path(), "one");
let p = path(t.path());
let native = fs::read(&p).unwrap();
let journal = p.with_extension("rredo");
let mut corrupt = fs::read(&journal).unwrap();
corrupt[offset] ^= 1;
fs::write(&journal, &corrupt).unwrap();
assert!(archive::load(t.path(), "example.org", "alice").is_err());
assert_eq!(fs::read(&p).unwrap(), native);
assert_eq!(fs::read(&journal).unwrap(), corrupt);
}
}
#[test]
fn bounded_rotation_recovers_suffix_and_rewrites_never_resurrect_deleted_records() {
let t = tempfile::tempdir().unwrap();
for i in 0..300 {
append(t.path(), &format!("id{i}"));
}
let p = path(t.path());
let journal = p.with_extension("rredo");
let data = fs::read(&journal).unwrap();
assert!(data.len() < 1024 * 1024);
let baseline = u64::from_le_bytes(data[8..16].try_into().unwrap());
assert!(baseline > 0);
fs::OpenOptions::new()
.write(true)
.open(&p)
.unwrap()
.set_len(baseline)
.unwrap();
assert_eq!(ids(t.path()).len(), 300);
archive::delete_ids(
t.path(),
"example.org",
Some("alice"),
"archive",
&["id299".into()],
)
.unwrap();
assert_eq!(ids(t.path()).len(), 299);
append(t.path(), "after-delete");
assert!(!ids(t.path()).contains(&"id299".into()));
archive::save_store(
t.path(),
"example.org",
Some("alice"),
"archive",
&[entry("replacement")],
)
.unwrap();
assert_eq!(ids(t.path()), ["replacement"]);
append(t.path(), "after-save");
assert_eq!(ids(t.path()), ["replacement", "after-save"]);
archive::expire_before(t.path(), "example.org", Some("alice"), "archive", 43.).unwrap();
assert!(ids(t.path()).is_empty());
append(t.path(), "after-expire");
assert_eq!(ids(t.path()), ["after-expire"]);
}
#[test]
fn byte_limit_rotates_before_frame_count_limit() {
let t = tempfile::tempdir().unwrap();
for i in 0..20 {
let mut e = entry(&format!("big{i}"));
e.xml = format!("<message><body>{}</body></message>", "x".repeat(100_000));
archive::append_new(t.path(), "example.org", Some("alice"), "archive", e).unwrap();
}
let data = fs::read(path(t.path()).with_extension("rredo")).unwrap();
assert!(u64::from_le_bytes(data[8..16].try_into().unwrap()) > 0);
assert!(data.len() < 1024 * 1024 + 110_000);
assert_eq!(ids(t.path()).len(), 20);
}
#[test]
fn explicit_native_atomic_handoff_and_missing_baseline_fail_closed() {
let t = tempfile::tempdir().unwrap();
append(t.path(), "old");
let p = path(t.path());
archive::checkpoint_all(t.path()).unwrap();
storage::atomic_write(
&p,
b"item({name='message';attr={};key='native';when=42});\n",
)
.unwrap();
assert_eq!(ids(t.path()), ["native"]);
append(t.path(), "rust");
assert_eq!(ids(t.path()), ["native", "rust"]);
fs::OpenOptions::new()
.append(true)
.open(&p)
.unwrap()
.write_all(b"item(bad uncommitted);")
.unwrap();
assert_eq!(ids(t.path()), ["native", "rust"]);
// The durable baseline cannot be reconstructed from suffix-only frames.
fs::remove_file(&p).unwrap();
assert!(archive::load(t.path(), "example.org", "alice").is_err());
}
#[test]
fn native_baseline_in_place_edits_are_rejected() {
let t = tempfile::tempdir().unwrap();
let p = path(t.path());
fs::create_dir_all(p.parent().unwrap()).unwrap();
let original = b"item({name='message';attr={};key='native';when=42});\n";
fs::write(&p, original).unwrap();
append(t.path(), "rust");
let mut data = fs::read(&p).unwrap();
data[5] ^= 1;
fs::write(&p, &data).unwrap();
assert!(archive::load(t.path(), "example.org", "alice").is_err());
assert_eq!(fs::read(&p).unwrap(), data);
}
#[test]
fn redo_process_child() {
let Some(root) = std::env::var_os("XMPP_ARCHIVE_REDO_CHILD") else {
return;
};
let root = Path::new(&root);
append(root, "accepted-before-kill");
fs::write(root.join("ready"), b"ready").unwrap();
loop {
std::thread::park();
}
}
#[test]
fn killed_writer_retains_accepted_records_after_native_data_loss() {
let t = tempfile::tempdir().unwrap();
let mut child = Command::new(std::env::current_exe().unwrap())
.args(["--exact", "redo_process_child", "--nocapture"])
.env("XMPP_ARCHIVE_REDO_CHILD", t.path())
.spawn()
.unwrap();
let deadline = Instant::now() + Duration::from_secs(10);
while !t.path().join("ready").exists() {
if Instant::now() >= deadline {
let _ = child.kill();
let _ = child.wait();
panic!("redo child not ready");
}
assert!(
child.try_wait().unwrap().is_none(),
"redo child exited early"
);
std::thread::sleep(Duration::from_millis(10));
}
child.kill().unwrap();
child.wait().unwrap();
// Simulate loss of dirty native pages independently of the synced journal.
fs::OpenOptions::new()
.write(true)
.open(path(t.path()))
.unwrap()
.set_len(0)
.unwrap();
assert_eq!(ids(t.path()), ["accepted-before-kill"]);
assert_eq!(ids(t.path()), ["accepted-before-kill"]);
}
#[test]
fn durable_handoff_allows_native_in_place_append_and_next_rust_append() {
let t = tempfile::tempdir().unwrap();
append(t.path(), "rust-before");
let p = path(t.path());
assert_eq!(archive::checkpoint_all(t.path()).unwrap(), 1);
assert!(!p.with_extension("rredo").exists());
fs::OpenOptions::new()
.append(true)
.open(&p)
.unwrap()
.write_all(b"item({name='message';attr={};key='native';when=42});\n")
.unwrap();
assert_eq!(ids(t.path()), ["rust-before", "native"]);
append(t.path(), "rust-after");
assert_eq!(ids(t.path()), ["rust-before", "native", "rust-after"]);
assert_eq!(archive::checkpoint_all(t.path()).unwrap(), 1);
assert_eq!(archive::checkpoint_all(t.path()).unwrap(), 0);
}
#[test]
fn legacy_rollback_marker_is_migrated_before_redo_append() {
let t = tempfile::tempdir().unwrap();
let p = path(t.path());
fs::create_dir_all(p.parent().unwrap()).unwrap();
let original = b"item({name='message';attr={};key='legacy';when=42});\n";
fs::write(&p, [original.as_slice(), b"item(broken"].concat()).unwrap();
fs::write(
p.with_extension("rappend"),
(original.len() as u64).to_le_bytes(),
)
.unwrap();
append(t.path(), "redo");
assert_eq!(ids(t.path()), ["legacy", "redo"]);
assert!(!p.with_extension("rappend").exists());
assert!(p.with_extension("rredo").exists());
}
#[test]
fn copied_archive_replays_committed_redo_despite_new_native_inode() {
for native_baseline in [false, true] {
for lose_entire_suffix in [false, true] {
let source = tempfile::tempdir().unwrap();
let source_path = path(source.path());
if native_baseline {
fs::create_dir_all(source_path.parent().unwrap()).unwrap();
fs::write(
&source_path,
b"item({name='message';attr={};key='baseline';when=42});\n",
)
.unwrap();
}
append(source.path(), "accepted-one");
append(source.path(), "accepted-two");
let expected = fs::read(&source_path).unwrap();
let redo = fs::read(source_path.with_extension("rredo")).unwrap();
let baseline = u64::from_le_bytes(redo[8..16].try_into().unwrap());
let restored = tempfile::tempdir().unwrap();
let restored_path = path(restored.path());
fs::create_dir_all(restored_path.parent().unwrap()).unwrap();
fs::copy(&source_path, &restored_path).unwrap();
fs::copy(
source_path.with_extension("rredo"),
restored_path.with_extension("rredo"),
)
.unwrap();
let length = if lose_entire_suffix {
baseline
} else {
expected.len() as u64 - 1
};
fs::OpenOptions::new()
.write(true)
.open(&restored_path)
.unwrap()
.set_len(length)
.unwrap();
let recovered = ids(restored.path());
assert_eq!(recovered.last().unwrap(), "accepted-two");
assert_eq!(recovered.len(), if native_baseline { 3 } else { 2 });
assert_eq!(fs::read(&restored_path).unwrap(), expected);
append(restored.path(), "after-restore");
assert_eq!(ids(restored.path()).last().unwrap(), "after-restore");
}
}
}
#[test]
fn atomic_native_replace_without_handoff_cannot_discard_committed_frames() {
let t = tempfile::tempdir().unwrap();
append(t.path(), "accepted");
let p = path(t.path());
storage::atomic_write(
&p,
b"item({name='message';attr={};key='replacement';when=42});\n",
)
.unwrap();
// A replacement without handoff is not authority to discard committed WAL.
assert_eq!(ids(t.path()), ["accepted"]);
}
+74
View File
@@ -0,0 +1,74 @@
/* Test-only LD_PRELOAD shim: delay, then perform the real durable operation.
* Scope is strictly the canonical fixture data root selected by the harness.
* No host mount or durability settings are changed. Log contains no file data.
* Exceptional XMPP_SYNC_FAIL_SUFFIX=.wal|.rredo injects EIO for matching
* fixture regular files. Without that explicit fault mode all syncs are real.
*/
#define _GNU_SOURCE
#include <dlfcn.h>
#include <errno.h>
#include <fcntl.h>
#include <limits.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/stat.h>
#include <time.h>
#include <unistd.h>
static long long ns(void) {
struct timespec t;
clock_gettime(CLOCK_MONOTONIC, &t);
return (long long)t.tv_sec * 1000000000LL + t.tv_nsec;
}
static int sync_call(int fd, const char *operation) {
int (*real_sync)(int) = dlsym(RTLD_NEXT, operation);
if (!real_sync) { errno = ENOSYS; return -1; }
const char *prefix = getenv("XMPP_SLOW_SYNC_ROOT");
const char *log = getenv("XMPP_SLOW_SYNC_LOG");
char link[64], path[PATH_MAX];
snprintf(link, sizeof link, "/proc/self/fd/%d", fd);
ssize_t length = readlink(link, path, sizeof path - 1);
if (!prefix || !log || length < 0) return real_sync(fd);
path[length] = '\0';
size_t size = strlen(prefix);
if (strncmp(path, prefix, size) || (path[size] && path[size] != '/'))
return real_sync(fd);
struct stat st;
if (fstat(fd, &st) || (!S_ISREG(st.st_mode) && !S_ISDIR(st.st_mode)))
return real_sync(fd);
const char *delay = getenv("XMPP_SLOW_SYNC_MS");
long ms = delay ? strtol(delay, NULL, 10) : 100;
if (ms < 0 || ms > 10000) ms = 100;
long long begin = ns();
struct timespec wait = {.tv_sec = ms / 1000, .tv_nsec = (ms % 1000) * 1000000};
while (nanosleep(&wait, &wait) && errno == EINTR) {}
const char *suffix = getenv("XMPP_SYNC_FAIL_SUFFIX");
int injected = 0;
if (suffix && (!strcmp(suffix, ".wal") || !strcmp(suffix, ".rredo")) && S_ISREG(st.st_mode)) {
size_t suffix_length = strlen(suffix);
injected = (size_t)length >= suffix_length &&
!strcmp(path + length - suffix_length, suffix);
}
int result;
if (injected) { errno = EIO; result = -1; }
else result = real_sync(fd);
int saved_errno = errno;
long long end = ns();
/* One append write per operation avoids partial concurrent log records.
* Paths are not logged: only a path-derived store category. */
const char *store = strstr(path, "/rust_smacks") ? "sm" :
strstr(path, "/archive") ? "mam" :
strstr(path, "/pep") ? "pep" : "other";
char row[512];
int count = snprintf(row, sizeof row,
"{\"op\":\"%s\",\"kind\":\"%s\",\"store\":\"%s\",\"begin_ns\":%lld,\"end_ns\":%lld,\"result\":%d,\"injected\":%s}\n",
operation, S_ISDIR(st.st_mode) ? "directory" : "file", store, begin, end, result,
injected ? "true" : "false");
int output = open(log, O_WRONLY | O_CREAT | O_APPEND | O_CLOEXEC, 0600);
if (output >= 0) { ssize_t ignored = write(output, row, count); (void)ignored; close(output); }
errno = saved_errno;
return result;
}
int fsync(int fd) { return sync_call(fd, "fsync"); }
int fdatasync(int fd) { return sync_call(fd, "fdatasync"); }
+199
View File
@@ -0,0 +1,199 @@
#!/usr/bin/env python3
"""Fail recovery journal syncs explicitly; never publish a false durable handoff.
Uses independent real TLS fixtures for .wal and .rredo so archive native inode
identity remains genuine. SIGKILL leaves complete valid journals. The scoped
test interposer returns EIO only for the selected suffix beneath fixture data.
"""
import argparse
import asyncio
import hashlib
import json
import os
from pathlib import Path
import shutil
import signal
import socket
import ssl
import subprocess
import tempfile
import time
from muc_restart_network import stop
from sm_restart_network import durable_slot
from slow_storage_latency_network import CARBONS, MeasuredClient, check_shim_scope, iq, sync_rows
from slow_storage_crash_network import committed_archive, start
async def seed(root, binary, config, port, log, env):
context = ssl.create_default_context(cafile=str(root/'cert.pem'))
clients, server = [], None
identifier = 'recovery-sync-body'
try:
server = await asyncio.to_thread(start, binary, config, port, log, env)
for number in range(4):
client = await MeasuredClient.connect(number, port, context, 2)
clients.append(client)
client.auto_ack = False
await client.enable()
await iq(client, f'carbons-{number}', f"<enable xmlns='{CARBONS}'/>")
await client.stanza('<presence/>')
await client.fence(f'ready-{number}')
alice, bob, _, _ = clients
await alice.stanza(f"<message to='{bob.jid}' type='chat' id='{identifier}'><body>{identifier}</body></message>")
await alice.handled()
for client, kind in zip(clients[1:], ('direct', 'sent', 'received')):
await client.wait(lambda c=client,k=kind: c.arrivals[(identifier,k)], 'message/carbon before crash')
server.send_signal(signal.SIGKILL)
assert await asyncio.to_thread(server.wait, 10) == -signal.SIGKILL
records = {client.token: durable_slot(root, client.token) for client in clients}
assert len(records) == 4
for client in clients[1:]:
assert sum(identifier in xml for xml in records[client.token]['pending']) == 1
archives = {p: committed_archive(p) for p in (root/'data'/'localhost'/'archive').glob('*.list')}
assert len(archives) == 2
assert all(sum(identifier in row for row in content.splitlines()) == 1 for content in archives.values())
return records, archives
finally:
if server is not None and server.poll() is None:
await asyncio.to_thread(stop, server)
await asyncio.gather(*(c.close_raw() for c in clients), return_exceptions=True)
def journal_bytes(root, suffix):
return {p: p.read_bytes() for p in (root/'data').rglob('*'+suffix)}
def assert_serve_rejects(binary, config, port, log, env):
server = subprocess.Popen([str(binary), 'serve', str(config), '--experimental'],
env=env, stdout=log, stderr=log)
listened = False
try:
deadline = time.monotonic()+15
while server.poll() is None and time.monotonic() < deadline:
try:
with socket.create_connection(('127.0.0.1', port), timeout=.05):
listened = True
break
except OSError:
time.sleep(.01)
assert not listened, 'SM recovery sync failure exposed a listener'
assert server.wait(timeout=5) != 0, 'SM recovery sync failure reported successful serve'
finally:
if server.poll() is None:
server.kill()
server.wait()
def scope_fault(work, shim):
root = work/'fault-scope'
root.mkdir()
code = """import errno,json,os,pathlib,sys
root,outside,suffix=pathlib.Path(sys.argv[1]),pathlib.Path(sys.argv[2]),sys.argv[3]
statuses=[]
for path in (root/('inside'+suffix),root/'unmatched.txt',outside):
with path.open('wb') as stream:
stream.write(b'fixture');stream.flush()
for sync in (os.fsync,os.fdatasync):
try:sync(stream.fileno());statuses.append(0)
except OSError as error:statuses.append(error.errno)
assert statuses==[errno.EIO,errno.EIO,0,0,0,0],statuses
"""
for suffix in ('.wal', '.rredo'):
log = work/('scope-'+suffix[1:]+'.jsonl')
env = dict(os.environ, LD_PRELOAD=str(shim), XMPP_SLOW_SYNC_ROOT=str(root),
XMPP_SLOW_SYNC_LOG=str(log), XMPP_SLOW_SYNC_MS='0', XMPP_SYNC_FAIL_SUFFIX=suffix)
subprocess.run(['python3', '-c', code, str(root), str(work/('outside'+suffix)), suffix],
env=env, check=True, capture_output=True)
rows = sync_rows(log)
assert len(rows) == 4 and sum(row['injected'] for row in rows) == 2
def run_case(work, binary, shim, suffix):
root = work/suffix[1:]
root.mkdir()
subprocess.run(['openssl', 'req', '-x509', '-newkey', 'ec', '-pkeyopt', 'ec_paramgen_curve:P-256',
'-nodes', '-days', '1', '-subj', '/CN=localhost', '-addext', 'subjectAltName=DNS:localhost',
'-keyout', str(root/'key.pem'), '-out', str(root/'cert.pem')], check=True, capture_output=True)
with socket.socket() as probe:
probe.bind(('127.0.0.1', 0))
port = probe.getsockname()[1]
config = root/'config.json'
config.write_text(json.dumps({'global': {'data_path': str(root/'data'), 'interfaces': ['127.0.0.1'],
'c2s_ports': [port], 's2s_ports': [], 'http_ports': [], 'https_ports': [],
'modules_enabled': ['tls', 'saslauth', 'smacks', 'ping', 'mam', 'carbons'],
'smacks_hibernation_time': 600, 'archive_expires_after': 'never', 'default_archive_policy': True,
'ssl': {'key': str(root/'key.pem'), 'certificate': str(root/'cert.pem')}},
'hosts': {'localhost': {'options': {'authentication': 'internal_hashed', 'password_hash': 'SHA-256'}}}}))
for number in range(2):
subprocess.run([str(binary), 'create-user', str(config), f'load-{number}', '--password-stdin'],
input='load-test-password\n', text=True, check=True, capture_output=True)
sync_log = root/'sync.jsonl'
env = dict(os.environ, LD_PRELOAD=str(shim), XMPP_SLOW_SYNC_ROOT=str(root/'data'),
XMPP_SLOW_SYNC_LOG=str(sync_log), XMPP_SLOW_SYNC_MS='0')
env.pop('XMPP_SYNC_FAIL_SUFFIX', None)
with (root/'server.log').open('w+') as log:
records, archives = asyncio.run(seed(root, binary, config, port, log, env))
journals = journal_bytes(root, suffix)
assert journals, 'network fixture did not leave requested journals'
fault_env = dict(env, XMPP_SYNC_FAIL_SUFFIX=suffix)
before = len(sync_rows(sync_log))
command = [str(binary), 'storage-checkpoint', str(config)]
result = subprocess.run(command, env=fault_env, capture_output=True, text=True)
assert result.returncode != 0, f'{suffix} recovery skipped journal sync and falsely reported successful handoff'
assert 'Durable native archive handoff:' not in result.stdout, 'failed checkpoint advertised handoff success'
attempts = [row for row in sync_rows(sync_log)[before:] if row.get('injected')]
assert attempts and all(row['result'] == -1 for row in attempts), 'recovery did not attempt faulted journal sync'
assert journal_bytes(root, suffix) == journals, 'failed checkpoint removed or changed authoritative journal'
if suffix == '.wal':
before_serve = len(sync_rows(sync_log))
assert_serve_rejects(binary, config, port, log, fault_env)
assert any(row.get('injected') and row['store'] == 'sm'
for row in sync_rows(sync_log)[before_serve:]), 'serve failed without attempting SM recovery sync'
assert journal_bytes(root, suffix) == journals, 'failed serve changed authoritative SM journals'
subprocess.run(command, env=env, check=True, capture_output=True, text=True)
assert not journal_bytes(root, '.wal') and not journal_bytes(root, '.rredo'), 'successful handoff left journals'
for token, record in records.items():
paths = list((root/'data').rglob(token+'.json'))
assert len(paths) == 1
exported = json.loads(paths[0].read_text())
assert {k:v for k,v in exported.items() if k != 'deadline_ms'} == {
k:v for k,v in record.items() if k != 'deadline_ms'}, 'handoff changed acknowledged SM state'
assert all(path.read_text() == content for path, content in archives.items()), 'handoff lost acknowledged MAM body'
return {'suffix': suffix, 'status': 'pass', 'checkpoint_rejected_sync_failure': True,
'journals_preserved_on_failure': len(journals),
'serve_rejected_before_listeners': suffix == '.wal',
'clean_retry_preserved_sessions': len(records), 'clean_retry_preserved_archives': len(archives)}
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('--binary', default='target/release/rust-xmpp')
parser.add_argument('--output', default='/tmp/journal-recovery-sync.json')
parser.add_argument('--suffix', nargs='+', choices=['.wal', '.rredo'], default=['.wal', '.rredo'])
args = parser.parse_args()
report = {'status': 'running', 'cases': []}
try:
with tempfile.TemporaryDirectory(prefix='xmpp-journal-recovery-sync-') as directory:
work = Path(directory)
binary = work/'rust-xmpp'
shutil.copy2(Path(args.binary).resolve(), binary)
report['binary_sha256'] = hashlib.sha256(binary.read_bytes()).hexdigest()
shim = work/'slow_sync.so'
subprocess.run(['cc', '-shared', '-fPIC', '-O2', '-Wall', '-Wextra', '-Werror',
str(Path(__file__).parent/'fixtures'/'slow_sync.c'), '-o', str(shim), '-ldl'], check=True)
check_shim_scope(work, shim)
scope_fault(work, shim)
for suffix in args.suffix:
report['cases'].append(run_case(work, binary, shim, suffix))
report['status'] = 'pass'
except BaseException as error:
report.update(status='fail', failure=str(error))
raise
finally:
Path(args.output).write_text(json.dumps(report, indent=2)+'\n')
print(json.dumps(report, indent=2))
if __name__ == '__main__':
main()
+2
View File
@@ -25,6 +25,8 @@ fn actual_prosody_muc_mam_reads_rust_and_writes_native_archive() {
"<message type='groupchat' to='room@rooms.example.org' id='rust-message'><body>Rust archived body</body></message>",
);
drop(muc);
// Transfer archive write ownership before native Prosody appends in place.
assert_eq!(rust_xmpp::archive::checkpoint_all(root.path()).unwrap(), 1);
let bootstrap = std::fs::read_to_string(concat!(
env!("CARGO_MANIFEST_DIR"),
"/scripts/prosody_compat/oracle.lua"
+3
View File
@@ -47,6 +47,8 @@ fn real_prosody_storage_and_auth_roundtrip() {
for (i, id) in ["rust-one", "rust-two"].iter().enumerate() {
archive::append_new(root,host,Some("alice"),"archive",archive::Entry{id:id.to_string(),when:1767225600.+i as f64,peer:"bob@example.org".into(),xml:"<message xmlns='jabber:client' xml:lang='ru' xmlns:o='urn:oracle' o:flag='yes'><body>Rust body</body></message>".into()}).unwrap();
}
// Transfer write ownership only after durable native checkpoint.
archive::checkpoint_all(root).unwrap();
oracle(root, "roundtrip", hash);
let v = storage::load(root, host, Some("alice"), "oracle")
.unwrap()
@@ -97,6 +99,7 @@ fn real_prosody_storage_and_auth_roundtrip() {
)
.unwrap();
assert!(!p.exists());
archive::checkpoint_all(root).unwrap();
oracle(root, "verify-final", hash);
}
}
+287
View File
@@ -0,0 +1,287 @@
#!/usr/bin/env python3
"""SIGKILL after sender SM acknowledgement under real delayed durable IO.
Four STARTTLS resources/two accounts, MAM, carbons and unacknowledged avatar
PEP. Inspect authoritative recipient journals and both archives before any
reauthentication; then resume all resources and verify replay and counters.
Uses the latency fixture's fsync/fdatasync interposer, retaining real syncs.
"""
import argparse
import asyncio
import hashlib
import json
import os
from pathlib import Path
import shutil
import signal
import socket
import ssl
import subprocess
import sys
import tempfile
import time
from muc_sm_load_network import SM
from muc_restart_network import stop
from sm_restart_network import durable_slot
from slow_storage_latency_network import (
CARBONS, PUBSUB, MeasuredClient, check_shim_scope, counts, iq, sync_rows,
)
def start(binary, config, port, log, env):
server = subprocess.Popen([str(binary), 'serve', str(config), '--experimental'],
env=env, stdout=log, stderr=log)
deadline = time.monotonic()+60
while time.monotonic() < deadline:
assert server.poll() is None, 'server stopped during startup'
try:
with socket.create_connection(('127.0.0.1', port), timeout=.1):
return server
except OSError:
time.sleep(.05)
server.kill()
server.wait()
raise AssertionError('server startup timeout')
def committed_archive(path):
"""Reconstruct native MAM bytes using its authoritative checksummed redo."""
native = path.read_bytes()
data = path.with_suffix('.rredo').read_bytes()
assert len(data) >= 96 and data[:8] == b'XARRED01', 'invalid MAM redo header'
assert hashlib.sha256(data[:64]).digest() == data[64:96], 'invalid MAM redo header checksum'
baseline = int.from_bytes(data[8:16], 'little')
assert baseline <= len(native), 'missing MAM native baseline'
committed, offset = native[:baseline], 96
while offset < len(data):
if len(data)-offset < 72:
break
size = int.from_bytes(data[offset:offset+8], 'little')
assert hashlib.sha256(data[offset:offset+8]).digest() == data[offset+8:offset+40], 'invalid MAM redo length checksum'
assert 0 < size <= 16*1024*1024, 'invalid MAM redo frame size'
end = offset+72+size
if end > len(data):
break
payload = data[offset+72:end]
assert hashlib.sha256(payload).digest() == data[offset+40:offset+72], 'invalid MAM redo frame checksum'
committed += payload
offset = end
# Sender handled acknowledgement must not depend on a native suffix that
# lacks a complete, durable redo frame.
assert native == committed, 'native MAM suffix is not committed in redo'
return committed.decode()
def assert_committed(root, expected, identifiers, avatar):
snapshots = [durable_slot(root, old['token']) for old in expected]
for index, (record, old) in enumerate(zip(snapshots, expected)):
assert record['jid'] == old['jid'], (index, record)
assert record['handled'] == old['handled'], (index, record['handled'], old)
assert record['acked'] == old['acked'], (index, record['acked'], old)
assert record['sent']-record['acked'] == len(record['pending']), (index, record)
if index in (1, 2, 3):
for identifier in identifiers:
assert sum(identifier in xml for xml in record['pending']) == 1, (index, identifier, record)
if index in (0, 2):
assert any(avatar in xml and 'event' in xml and 'crash-avatar' in xml
for xml in record['pending']), ('PEP missing before reauthentication', index, record)
paths = sorted((root/'data'/'localhost'/'archive').glob('*.list'))
assert len(paths) == 2, paths
for path in paths:
rows = [row for row in committed_archive(path).splitlines() if row.startswith('item(')]
assert len(rows) == len(identifiers), (path.name, len(rows), identifiers)
for identifier in identifiers:
assert sum(identifier in row for row in rows) == 1, ('acknowledged body missing/duplicated in MAM', path.name, identifier)
return snapshots
async def exercise(root, binary, config, port, log, env, sync_log, args):
context = ssl.create_default_context(cafile=str(root/'cert.pem'))
clients, server = [], None
avatar = 'urn:xmpp:avatar:metadata'
try:
server = await asyncio.to_thread(start, binary, config, port, log, env)
for number in range(4):
client = await MeasuredClient.connect(number, port, context, 2)
clients.append(client)
client.auto_ack = False
await client.enable()
await iq(client, f'carbons-{number}', f"<enable xmlns='{CARBONS}'/>")
await client.stanza('<presence/>')
await client.fence(f'ready-{number}')
alice, bob, alice_other, _ = clients
await iq(alice, 'avatar-initial', f"<pubsub xmlns='{PUBSUB}'><publish node='{avatar}'><item id='initial'><metadata xmlns='{avatar}'/></item></publish></pubsub>")
for client in (alice, alice_other):
await iq(client, f'subscribe-{client.number}', f"<pubsub xmlns='{PUBSUB}'><subscribe node='{avatar}' jid='{client.jid}'/></pubsub>")
# Ack every setup stanza, then fence acknowledgement persistence. No
# resource sends another ACK until after the crash replay assertions.
for client in clients:
await client.fence(f'setup-{client.number}')
await client.ack()
await client.handled()
before = len(sync_rows(sync_log))
await iq(alice, 'avatar-pending', f"<pubsub xmlns='{PUBSUB}'><publish node='{avatar}'><item id='crash-avatar'><metadata xmlns='{avatar}'><info id='fixture-avatar' bytes='1' type='image/png'/></metadata></item></publish></pubsub>")
identifiers = [f'crash-body-{number}' for number in range(args.messages)]
for identifier in identifiers:
await alice.stanza(f"<message to='{bob.jid}' type='chat' id='{identifier}'><body>{identifier}</body></message>")
await alice.handled()
expected = [{'token': c.token, 'jid': c.jid, 'acked': c.acked, 'handled': c.inputs}
for c in clients]
# No receiver fence/delivery wait between sender's final <a/> and kill:
# the acknowledged sender operation itself must make the data durable.
server.send_signal(signal.SIGKILL)
assert await asyncio.to_thread(server.wait, 10) == -signal.SIGKILL
snapshots = assert_committed(root, expected, identifiers, avatar)
rows = sync_rows(sync_log)[before:]
summary = counts(rows)
assert rows and all(row['end_ns']-row['begin_ns'] >= args.delay_ms*1000000 for row in rows)
assert {'sm', 'mam', 'pep'}.issubset({row['store'] for row in rows}), summary
assert any(row['kind'] == 'directory' for row in rows), summary
await asyncio.gather(*(c.close_raw() for c in clients), return_exceptions=True)
clients.clear()
server = await asyncio.to_thread(start, binary, config, port, log, env)
# Startup must recover the same queues before a single SASL login.
recovered = assert_committed(root, expected, identifiers, avatar)
for old, new in zip(snapshots, recovered):
assert old['pending'] == new['pending']
for number, old in enumerate(expected):
client = await MeasuredClient.connect(number, port, context, 2, bind=False)
clients.append(client)
client.auto_ack = False
client.received = client.acked = old['acked']
client.inputs, client.token = old['handled'], old['token']
await client.send(f"<resume xmlns='{SM}' previd='{client.token}' h='{client.acked}'/>")
result = await client.control('resumed')
assert result.get('previd') == client.token
assert int(result.get('h')) == old['handled'], (number, result.attrib, old)
kind = {1: 'direct', 2: 'sent', 3: 'received'}.get(number)
if kind:
for identifier in identifiers:
await client.wait(lambda c=client,i=identifier,k=kind: c.arrivals[(i,k)], 'durable replay '+identifier)
assert len(client.arrivals[(identifier,kind)]) == 1, (number, identifier, client.arrivals)
await client.fence(f'after-resume-{number}')
# The fence follows every replay, so the exact output counter also
# detects missing/extra PEP, carbon or IQ replay, beyond body counts.
assert client.received == snapshots[number]['sent']+1, (number, client.received, snapshots[number]['sent'])
await client.ack()
await client.handled()
record = durable_slot(root, client.token)
assert record['acked'] == client.received and not record['pending'], (number, record)
assert not client.errors, client.errors
# Native handoff is explicitly offline: release the server's data/store
# locks before asking the CLI to durably export and remove journals.
await asyncio.to_thread(stop, server)
server = None
final_slots = {c.token: durable_slot(root, c.token) for c in clients}
final_archives = {p: committed_archive(p) for p in
(root/'data'/'localhost'/'archive').glob('*.list')}
checkpoint = [str(binary), 'storage-checkpoint', str(config)]
await asyncio.to_thread(subprocess.run, checkpoint, env=env, check=True, capture_output=True, text=True)
assert not list((root/'data').rglob('*.wal')), 'SM WAL survived offline checkpoint'
assert not list((root/'data').rglob('*.rredo')), 'MAM redo survived offline checkpoint'
exported = {}
for token, authoritative in final_slots.items():
paths = list((root/'data').rglob(token+'.json'))
assert len(paths) == 1, ('checkpoint lost SM export', token)
exported[token] = json.loads(paths[0].read_text())
previous = dict(authoritative)
current = dict(exported[token])
previous.pop('deadline_ms')
current.pop('deadline_ms')
assert current == previous, ('checkpoint changed authoritative SM fields', token, current, previous)
assert all(p.read_text() == content for p, content in final_archives.items()), 'checkpoint changed committed MAM records'
await asyncio.to_thread(subprocess.run, checkpoint, env=env, check=True, capture_output=True, text=True)
assert not list((root/'data').rglob('*.wal')) and not list((root/'data').rglob('*.rredo'))
for token, previous in exported.items():
paths = list((root/'data').rglob(token+'.json'))
assert len(paths) == 1 and json.loads(paths[0].read_text()) == previous, 'second checkpoint changed SM export'
assert all(p.read_text() == content for p, content in final_archives.items()), 'second checkpoint changed MAM records'
return {'status': 'pass', 'delay_ms_per_sync': args.delay_ms,
'acknowledged_messages': len(identifiers), 'mam_records_per_account': [len(identifiers)]*2,
'resumed_resources': len(clients), 'replayed_personal_deliveries': 3*len(identifiers),
'offline_checkpoint': {'exported_sessions': len(exported), 'native_archives': len(final_archives),
'journals_removed': True, 'second_invocation_idempotent': True},
'pending_before_reauthentication': [len(record['pending']) for record in snapshots],
'sync_between_publication_and_sigkill': summary}
finally:
if server is not None and server.poll() is None:
failing = sys.exc_info()[0] is not None
try:
await asyncio.to_thread(stop, server)
except BaseException:
if not failing:
raise
await asyncio.gather(*(c.close_raw() for c in clients), return_exceptions=True)
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('--binary', default='target/release/rust-xmpp')
parser.add_argument('--delay-ms', type=int, default=100)
parser.add_argument('--messages', type=int, default=3)
parser.add_argument('--output', default='/tmp/slow-storage-crash.json')
args = parser.parse_args()
assert 0 <= args.delay_ms <= 10000 and args.messages > 0
with tempfile.TemporaryDirectory(prefix='xmpp-slow-storage-crash-') as directory:
root = Path(directory)
binary = root/'rust-xmpp'
shutil.copy2(Path(args.binary).resolve(), binary)
shim = root/'slow_sync.so'
subprocess.run(['cc', '-shared', '-fPIC', '-O2', '-Wall', '-Wextra', '-Werror',
str(Path(__file__).parent/'fixtures'/'slow_sync.c'), '-o', str(shim), '-ldl'], check=True)
check_shim_scope(root, shim)
subprocess.run(['openssl', 'req', '-x509', '-newkey', 'ec', '-pkeyopt', 'ec_paramgen_curve:P-256',
'-nodes', '-days', '1', '-subj', '/CN=localhost', '-addext', 'subjectAltName=DNS:localhost',
'-keyout', str(root/'key.pem'), '-out', str(root/'cert.pem')], check=True, capture_output=True)
with socket.socket() as probe:
probe.bind(('127.0.0.1', 0))
port = probe.getsockname()[1]
config = root/'config.json'
config.write_text(json.dumps({'global': {'data_path': str(root/'data'), 'interfaces': ['127.0.0.1'],
'c2s_ports': [port], 's2s_ports': [], 'http_ports': [], 'https_ports': [],
'smacks_hibernation_time': 600,
'modules_enabled': ['tls', 'saslauth', 'smacks', 'csi_simple', 'ping', 'mam', 'pep', 'carbons'],
'archive_expires_after': 'never', 'default_archive_policy': True,
'ssl': {'key': str(root/'key.pem'), 'certificate': str(root/'cert.pem')}},
'hosts': {'localhost': {'options': {'authentication': 'internal_hashed', 'password_hash': 'SHA-256'}}}}))
for number in range(2):
subprocess.run([str(binary), 'create-user', str(config), f'load-{number}', '--password-stdin'],
input='load-test-password\n', text=True, check=True, capture_output=True)
sync_log = root/'sync.jsonl'
env = dict(os.environ, LD_PRELOAD=str(shim), XMPP_SLOW_SYNC_ROOT=str(root/'data'),
XMPP_SLOW_SYNC_LOG=str(sync_log), XMPP_SLOW_SYNC_MS=str(args.delay_ms))
with (root/'server.log').open('w+') as log:
try:
report = asyncio.run(exercise(root, binary, config, port, log, env, sync_log, args))
except BaseException as error:
# Preserve the first failing run's evidence outside the
# TemporaryDirectory before cleanup; never replace its error
# with a later shutdown or diagnostics failure.
try:
log.flush()
output = Path(args.output)
output.parent.mkdir(parents=True, exist_ok=True)
evidence = Path(tempfile.mkdtemp(prefix=output.stem+'-failure-', dir=output.parent))
shutil.copy2(root/'server.log', evidence/'server.log')
if sync_log.exists():
shutil.copy2(sync_log, evidence/'sync.jsonl')
Path(args.output).write_text(json.dumps({'status': 'fail',
'failure': repr(error), 'delay_ms_per_sync': args.delay_ms,
'binary_sha256': hashlib.sha256(binary.read_bytes()).hexdigest(),
'evidence': str(evidence)}, indent=2)+'\n')
except Exception as diagnostic_error:
print(f'Could not preserve failure evidence: {diagnostic_error}', file=sys.stderr)
log.seek(0)
print(log.read(), file=sys.stderr)
raise
report['binary_sha256'] = hashlib.sha256(binary.read_bytes()).hexdigest()
report['scope'] = 'loopback STARTTLS; real delayed fsync/fdatasync; sender SM acknowledgement then SIGKILL; authoritative queues and MAM checked before reauthentication; at-least-once replay'
result = json.dumps(report, indent=2)+'\n'
if args.output:
Path(args.output).write_text(result)
print(result, end='')
if __name__ == '__main__':
main()
+299
View File
@@ -0,0 +1,299 @@
#!/usr/bin/env python3
"""Fixture-only slow durable IO: two accounts, four TLS/SM resources, MAM/PEP.
Builds only a tiny C LD_PRELOAD interposer; uses an isolated copy of --binary.
Every intercepted fsync/fdatasync still executes. Directory syncs under the
fixture data root are delayed too and counted separately. No mount changes.
Timing includes durable work, with client ACKs withheld during each sample.
"""
import argparse
import asyncio
import collections
import hashlib
import json
import os
import pathlib
import re
import shutil
import socket
import ssl
import subprocess
import sys
import tempfile
import time
import xml.etree.ElementTree as ET
from muc_sm_load_network import Client, CLIENT, SM
from muc_restart_network import stop
from message_latency_network import slot
CARBONS = 'urn:xmpp:carbons:2'
PUBSUB = 'http://jabber.org/protocol/pubsub'
PHASE_FIELDS = ('pre_routing_ms', 'sender_mam_ms', 'recipient_mam_ms',
'delivery_ms', 'carbons_ms', 'offline_push_ms', 'handled_ms')
SLOW_FIELDS = {'duration_ms', 'route', 'target_count', 'carbon_count', *PHASE_FIELDS}
def slow_events(path, delay, rounds, allow_legacy=False):
events = []
for line in path.read_text().splitlines():
if 'event="message-processing-slow"' not in line:
continue
fields = dict(re.findall(r'(?:^|\s)"([^"\s]+)"="([^"\n]*)"', line))
legacy = allow_legacy and set(fields) == {'duration_ms', 'route'}
assert legacy or set(fields) == SLOW_FIELDS, ('unexpected slow-event fields', sorted(fields))
assert not any(value in line for value in
('slow fixture ', 'body-settled-', 'body-avatar_pending-', '<body')), 'payload leaked into slow diagnostics'
event = {key: value if key == 'route' else int(value) for key,value in fields.items()}
assert event['duration_ms'] >= 250 and event['route'] == 'local', event
if not legacy:
assert all(event[key] >= 0 for key in PHASE_FIELDS), event
assert sum(event[key] for key in PHASE_FIELDS) <= event['duration_ms']+len(PHASE_FIELDS), event
assert event['target_count'] == 1 and event['carbon_count'] == 2, event
events.append(event)
# At >=100ms the retained durable commits alone exceed the 250ms threshold.
# Smaller optional delays may legitimately produce no warning.
if delay >= 100:
assert len(events) == 2*rounds, ('missing/extra slow diagnostics',len(events),2*rounds)
return events
class WireReader:
def __init__(self, reader, owner):
self.reader, self.owner = reader, owner
self.parser = ET.XMLPullParser(events=('start', 'end'))
self.parser.feed("<root xmlns='jabber:client'>")
self.root = next(self.parser.read_events())[1]
self.depth = 0
def __getattr__(self, name):
return getattr(self.reader, name)
async def read(self, size):
data = await self.reader.read(size)
self.parser.feed(data)
for event, node in self.parser.read_events():
self.depth += 1 if event == 'start' else -1
if event != 'end' or self.depth:
continue
now = time.monotonic_ns()
if node.tag == f'{{{CLIENT}}}message':
for message in node.iter(f'{{{CLIENT}}}message'):
if message.get('id'):
kind = 'direct'
for carbon in ('sent', 'received'):
if node.find(f'{{{CARBONS}}}{carbon}') is not None:
kind = carbon
self.owner.arrivals[(message.get('id'), kind)].append(now)
if node.tag == f'{{{SM}}}a':
self.owner.ack_times[int(node.get('h'))] = now
self.root.remove(node)
return data
class MeasuredClient(Client):
def __init__(self, *args):
super().__init__(*args)
self.arrivals = collections.defaultdict(list)
self.ack_times = {}
self.reader = WireReader(self.reader, self)
async def iq(client, identifier, payload):
await client.stanza(f"<iq type='set' id='{identifier}'>{payload}</iq>")
result = await client.wait(lambda: client.iqs.pop(identifier, None), identifier)
assert result.get('type') == 'result', ET.tostring(result)
def sync_rows(path):
return [json.loads(row) for row in path.read_text().splitlines()] if path.exists() else []
def counts(rows):
result = collections.Counter(f"{row['store']}:{row['kind']}:{row['op']}" for row in rows)
assert all(row['result'] == 0 for row in rows), 'real sync failed'
return {'total': len(rows), 'by_store_kind_op': dict(sorted(result.items()))}
async def exercise(root, port, args, log):
context = ssl.create_default_context(cafile=str(root/'cert.pem'))
clients = []
report = []
try:
for number in range(4):
client = await MeasuredClient.connect(number, port, context, 2)
clients.append(client)
client.auto_ack = False
await client.enable()
await iq(client, f'carbons-{number}', f"<enable xmlns='{CARBONS}'/>")
await client.stanza('<presence/>')
await client.fence(f'ready-{number}')
alice, bob, alice_other, bob_other = clients
avatar = 'urn:xmpp:avatar:metadata'
# Self-account subscriptions make avatar events reach sibling resources.
await iq(alice, 'avatar-initial', f"<pubsub xmlns='{PUBSUB}'><publish node='{avatar}'><item id='initial'><metadata xmlns='{avatar}'/></item></publish></pubsub>")
for client in (alice, alice_other):
await iq(client, f'subscribe-{client.number}', f"<pubsub xmlns='{PUBSUB}'><subscribe node='{avatar}' jid='{client.jid}'/></pubsub>")
for phase in ('settled', 'avatar_pending'):
for turn in range(args.rounds):
# Fence all previous ACK work before taking operation counts.
for client in clients:
await client.ack()
await client.fence(f'before-{phase}-{turn}-{client.number}')
await client.handled()
before = len(sync_rows(log))
start_ns = time.monotonic_ns()
if phase == 'avatar_pending':
# Publication remains ahead of the body on this stream; no
# client ACK is sent for its IQ result or PEP notifications.
await alice.stanza(f"<iq type='set' id='publish-{turn}'><pubsub xmlns='{PUBSUB}'><publish node='{avatar}'><item id='sample-{turn}'><metadata xmlns='{avatar}'><info id='fixture-avatar' bytes='1' type='image/png'/></metadata></item></publish></pubsub></iq>")
identifier = f'body-{phase}-{turn}'
begun = time.monotonic_ns()
await alice.stanza(f"<message to='{bob.jid}' type='chat' id='{identifier}'><body>slow fixture {identifier}</body></message>")
await alice.handled()
events = []
for client, kind in ((bob, 'direct'), (alice_other, 'sent'), (bob_other, 'received')):
await client.wait(lambda c=client,k=kind: c.arrivals[(identifier,k)], identifier+' '+kind)
arrivals = client.arrivals[(identifier, kind)]
assert len(arrivals) == 1, (identifier, kind, arrivals)
events.append((kind, (arrivals[0]-begun)/1e6))
end_ns = time.monotonic_ns()
rows = sync_rows(log)[before:]
assert rows, 'interposer recorded no data-root sync operations'
assert all(row['end_ns']-row['begin_ns'] >= args.current_delay_ms*1000000
for row in rows), 'configured sync delay was not applied'
snapshots = [slot(root,c) for c in clients]
for index in (1,2,3):
assert sum(identifier in xml for xml in snapshots[index]['pending']) == 1, (identifier,index)
if phase == 'avatar_pending':
assert all(any(avatar in xml and 'event' in xml for xml in snapshots[index]['pending'])
for index in (0,2)), 'avatar PEP notifications were not durably pending'
report.append({'phase': phase, 'round': turn,
'delivery_ms': dict(events),
'sender_sm_handled_ms': (alice.ack_times[alice.inputs]-begun)/1e6,
'publication_plus_delivery_ms': (end_ns-start_ns)/1e6,
'sync': counts(rows),
'pending_per_resource': [len(snapshot['pending']) for snapshot in snapshots]})
assert all(not client.errors for client in clients), [c.errors for c in clients]
# Both ends archive each personal body exactly once, despite carbons.
archive_counts = [sum(line.startswith('item(') for line in path.open())
for path in sorted((root/'data'/'localhost'/'archive').glob('*.list'))]
expected = args.history_records + 2*args.rounds
assert archive_counts == [expected, expected], (archive_counts, expected)
return {'samples': report, 'mam_records_per_account': archive_counts,
'status': 'pass'}
finally:
await asyncio.gather(*(c.close_raw() for c in clients), return_exceptions=True)
def check_shim_scope(work, shim):
"""Prove that both sync APIs preserve success and exclude non-data files."""
root = work/'shim-scope-data'
root.mkdir()
log = work/'shim-scope.jsonl'
code = """import os,pathlib,sys
root,outside=map(pathlib.Path,sys.argv[1:])
for path in (root/'inside',outside):
with path.open('wb') as stream:
stream.write(b'fixture')
stream.flush()
os.fsync(stream.fileno())
os.fdatasync(stream.fileno())
fd=os.open(root,os.O_RDONLY)
os.fsync(fd)
os.close(fd)
"""
env = dict(os.environ, LD_PRELOAD=str(shim), XMPP_SLOW_SYNC_ROOT=str(root),
XMPP_SLOW_SYNC_LOG=str(log), XMPP_SLOW_SYNC_MS='0')
subprocess.run([sys.executable,'-c',code,str(root),str(work/'outside')],
env=env,check=True,capture_output=True)
rows = sync_rows(log)
assert len(rows) == 3, rows
assert collections.Counter((r['op'],r['kind']) for r in rows) == {
('fsync','file'):1, ('fdatasync','file'):1, ('fsync','directory'):1}
assert all(row['result'] == 0 for row in rows)
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('--binary', default='target/release/rust-xmpp')
parser.add_argument('--delay-ms', type=int, nargs='+', default=[0, 100])
parser.add_argument('--rounds', type=int, default=3)
parser.add_argument('--history-records', type=int, default=100)
parser.add_argument('--output')
parser.add_argument('--allow-legacy-phase-logs', action='store_true',
help='baseline binaries only: permit old duration/route diagnostics without phase metrics')
args = parser.parse_args()
assert args.rounds > 0 and args.history_records >= 0
assert all(0 <= delay <= 10000 for delay in args.delay_ms)
with tempfile.TemporaryDirectory(prefix='xmpp-slow-storage-') as directory:
work = pathlib.Path(directory)
binary = work/'rust-xmpp'
shutil.copy2(pathlib.Path(args.binary).resolve(), binary)
shim = work/'slow_sync.so'
subprocess.run(['cc', '-shared', '-fPIC', '-O2', '-Wall', '-Wextra', '-Werror',
str(pathlib.Path(__file__).parent/'fixtures'/'slow_sync.c'), '-o', str(shim), '-ldl'], check=True)
check_shim_scope(work, shim)
report = {'binary_sha256': hashlib.sha256(binary.read_bytes()).hexdigest(),
'scope': 'loopback STARTTLS; four SM resources/two accounts; MAM + pending avatar metadata PEP; fixture-only fsync/fdatasync including data directories; real sync retained; operation windows include background SM heartbeat',
'cases': []}
for delay in args.delay_ms:
root = work/f'delay-{delay}'
root.mkdir()
subprocess.run(['openssl', 'req', '-x509', '-newkey', 'ec', '-pkeyopt', 'ec_paramgen_curve:P-256',
'-nodes', '-days', '1', '-subj', '/CN=localhost', '-addext', 'subjectAltName=DNS:localhost',
'-keyout', str(root/'key.pem'), '-out', str(root/'cert.pem')], check=True, capture_output=True)
with socket.socket() as probe:
probe.bind(('127.0.0.1',0))
port = probe.getsockname()[1]
config = root/'config.json'
config.write_text(json.dumps({'global': {'data_path': str(root/'data'), 'interfaces':['127.0.0.1'],
'c2s_ports':[port], 's2s_ports':[], 'http_ports':[], 'https_ports':[],
'modules_enabled':['tls','saslauth','smacks','csi_simple','ping','mam','pep','carbons'],
'archive_expires_after':'never', 'default_archive_policy':True,
'ssl':{'key':str(root/'key.pem'),'certificate':str(root/'cert.pem')}},
'hosts':{'localhost':{'options':{'authentication':'internal_hashed','password_hash':'SHA-256'}}}}))
for number in range(2):
subprocess.run([str(binary),'create-user',str(config),f'load-{number}','--password-stdin'],
input='load-test-password\n',text=True,check=True,capture_output=True)
archive = root/'data'/'localhost'/'archive'/f'load%2d{number}.list'
archive.parent.mkdir(parents=True,exist_ok=True)
with archive.open('w') as stream:
for index in range(args.history_records):
stream.write("item({name='message';attr={xmlns='jabber:client';type='chat'};"
f"key='seed-{index}';when={time.time()};with='load-{1-number}@localhost';"
"{name='body';attr={};'history'}});\n")
sync_log = root/'sync.jsonl'
env = dict(os.environ, LD_PRELOAD=str(shim), XMPP_SLOW_SYNC_ROOT=str(root/'data'),
XMPP_SLOW_SYNC_LOG=str(sync_log), XMPP_SLOW_SYNC_MS=str(delay))
with (root/'server.log').open('w+') as log:
server = subprocess.Popen([str(binary),'serve',str(config),'--experimental'],env=env,stdout=log,stderr=log)
try:
deadline = time.monotonic()+60
while True:
assert server.poll() is None, log.read()
try:
with socket.create_connection(('127.0.0.1',port),timeout=.1):
break
except OSError:
assert time.monotonic()<deadline, 'server startup timeout'
time.sleep(.05)
args.current_delay_ms = delay
case = asyncio.run(exercise(root,port,args,sync_log))
case['delay_ms_per_sync'] = delay
events = slow_events(root/'server.log',delay,args.rounds,args.allow_legacy_phase_logs)
case['server_message_processing_slow_events'] = events
case['server_message_processing_slow_ms'] = [event['duration_ms'] for event in events]
report['cases'].append(case)
finally:
stop(server)
if args.output:
pathlib.Path(args.output).write_text(json.dumps(report,indent=2)+'\n')
print(f'slow storage: delay={delay}ms pass',flush=True)
print(json.dumps(report,indent=2))
if __name__ == '__main__':
main()
+322
View File
@@ -0,0 +1,322 @@
//! Redo recovery is independent of the best-effort readable JSON export.
use rust_xmpp::sessions::Hub;
use std::{fs, io::Write, path::Path, time::Duration};
use tokio::sync::mpsc;
fn committed(dir: &Path) -> (String, Vec<u8>) {
let hub = Hub::persistent(dir).unwrap();
let (tx, _rx) = mpsc::channel(8);
let session = hub.bind("a@x/r", "c", tx, Duration::from_secs(60)).unwrap();
let mut state = session.lock().unwrap();
state.enable(true).unwrap();
state.transmit("durable".into()).unwrap();
let token = state.resume_id.clone().unwrap();
let json = fs::read(dir.join(format!("{token}.json"))).unwrap();
drop(state);
drop(hub);
(token, json)
}
#[test]
fn redo_repairs_missing_or_torn_export_before_json_validation() {
for torn in [false, true] {
let dir = tempfile::tempdir().unwrap();
let (token, _) = committed(dir.path());
let json = dir.path().join(format!("{token}.json"));
if torn {
fs::write(&json, b"{torn").unwrap();
} else {
fs::remove_file(&json).unwrap();
}
let hub = Hub::persistent(dir.path()).unwrap();
assert_eq!(
hub.get("a@x/r")
.unwrap()
.lock()
.unwrap()
.pending
.front()
.map(String::as_str),
Some("durable")
);
let export: serde_json::Value = serde_json::from_slice(&fs::read(json).unwrap()).unwrap();
assert_eq!(export["pending"][0], "durable");
}
}
#[test]
fn partial_tail_is_discarded_and_subsequent_commits_survive_restart() {
let dir = tempfile::tempdir().unwrap();
let (token, _) = committed(dir.path());
let wal = dir.path().join(format!("{token}.wal"));
let good_len = fs::metadata(&wal).unwrap().len();
fs::OpenOptions::new()
.append(true)
.open(&wal)
.unwrap()
.write_all(&[42, 0, 0])
.unwrap();
let hub = Hub::persistent(dir.path()).unwrap();
assert_eq!(fs::metadata(&wal).unwrap().len(), good_len);
hub.get("a@x/r")
.unwrap()
.lock()
.unwrap()
.mark_handled()
.unwrap();
drop(hub);
let hub = Hub::persistent(dir.path()).unwrap();
assert_eq!(hub.get("a@x/r").unwrap().lock().unwrap().handled, 1);
}
#[test]
fn complete_corrupt_frame_fails_closed_even_with_valid_export() {
let dir = tempfile::tempdir().unwrap();
let (token, _) = committed(dir.path());
let wal = dir.path().join(format!("{token}.wal"));
let mut bytes = fs::read(&wal).unwrap();
*bytes.last_mut().unwrap() ^= 1;
fs::write(wal, bytes).unwrap();
assert!(Hub::persistent(dir.path()).is_err());
}
#[test]
fn durable_tombstone_overrides_resurrected_stale_json() {
let dir = tempfile::tempdir().unwrap();
let (token, old) = committed(dir.path());
let hub = Hub::persistent(dir.path()).unwrap();
hub.kick_owner("a@x");
drop(hub);
fs::write(dir.path().join(format!("{token}.json")), old).unwrap();
let hub = Hub::persistent(dir.path()).unwrap();
assert!(hub.get("a@x/r").is_none());
}
#[test]
fn legacy_json_migrates_and_baseline_survives_partial_first_append() {
let dir = tempfile::tempdir().unwrap();
let (token, _) = committed(dir.path());
let wal = dir.path().join(format!("{token}.wal"));
fs::remove_file(&wal).unwrap();
let hub = Hub::persistent(dir.path()).unwrap();
hub.get("a@x/r")
.unwrap()
.lock()
.unwrap()
.mark_handled()
.unwrap();
drop(hub);
let bytes = fs::read(&wal).unwrap();
let baseline_len = u32::from_le_bytes(bytes[8..12].try_into().unwrap()) as usize;
fs::write(&wal, &bytes[..8 + 40 + baseline_len + 12]).unwrap();
let hub = Hub::persistent(dir.path()).unwrap();
let session = hub.get("a@x/r").unwrap();
let state = session.lock().unwrap();
assert_eq!(state.handled, 0);
assert_eq!(state.pending.front().map(String::as_str), Some("durable"));
}
#[test]
fn rotation_preserves_latest_snapshot_and_bounds_redo_size() {
let dir = tempfile::tempdir().unwrap();
let hub = Hub::persistent(dir.path()).unwrap();
let (tx, _rx) = mpsc::channel(8);
let session = hub.bind("a@x/r", "c", tx, Duration::from_secs(60)).unwrap();
let token;
{
let mut state = session.lock().unwrap();
state.enable(true).unwrap();
token = state.resume_id.clone().unwrap();
state.transmit("x".repeat(1024 * 1024)).unwrap();
for _ in 0..35 {
state.mark_handled().unwrap();
}
}
let wal = dir.path().join(format!("{token}.wal"));
assert!(fs::metadata(wal).unwrap().len() < 18 * 1024 * 1024);
drop(hub);
let hub = Hub::persistent(dir.path()).unwrap();
let session = hub.get("a@x/r").unwrap();
let state = session.lock().unwrap();
assert_eq!(state.handled, 35);
assert_eq!(state.pending.front().unwrap().len(), 1024 * 1024);
}
#[test]
fn incomplete_baseline_or_magic_never_falls_back_to_json() {
for keep in [0, 4, 8, 12, 48] {
let dir = tempfile::tempdir().unwrap();
let (token, _) = committed(dir.path());
let wal = dir.path().join(format!("{token}.wal"));
let bytes = fs::read(&wal).unwrap();
fs::write(wal, &bytes[..keep]).unwrap();
assert!(
Hub::persistent(dir.path()).is_err(),
"truncated baseline at {keep}"
);
}
}
#[test]
#[cfg(unix)]
fn journal_symlink_is_rejected_without_following_target() {
use std::os::unix::fs::symlink;
let dir = tempfile::tempdir().unwrap();
let (token, _) = committed(dir.path());
let wal = dir.path().join(format!("{token}.wal"));
let other = tempfile::NamedTempFile::new().unwrap();
fs::remove_file(&wal).unwrap();
symlink(other.path(), wal).unwrap();
assert!(Hub::persistent(dir.path()).is_err());
}
#[test]
fn export_failure_after_commit_advances_memory_and_fences_later_writes() {
let dir = tempfile::tempdir().unwrap();
let hub = Hub::persistent(dir.path()).unwrap();
let (tx, _rx) = mpsc::channel(8);
let session = hub.bind("a@x/r", "c", tx, Duration::from_secs(60)).unwrap();
session.lock().unwrap().enable(true).unwrap();
let token = session.lock().unwrap().resume_id.clone().unwrap();
let obstruction = dir.path().join(format!("{token}.tmp"));
fs::create_dir(&obstruction).unwrap();
{
let mut state = session.lock().unwrap();
state.mark_handled().unwrap();
assert_eq!(state.handled, 1);
assert!(state.mark_handled().is_err());
assert_eq!(state.handled, 1);
}
drop(hub);
fs::remove_dir(obstruction).unwrap();
let hub = Hub::persistent(dir.path()).unwrap();
assert_eq!(hub.get("a@x/r").unwrap().lock().unwrap().handled, 1);
}
#[test]
fn retired_journals_are_collected_in_bounded_batches() {
let dir = tempfile::tempdir().unwrap();
let hub = Hub::persistent(dir.path()).unwrap();
for index in 0..64 {
let (tx, _rx) = mpsc::channel(1);
let connection = format!("c{index}");
let session = hub
.bind("a@x/r", &connection, tx, Duration::from_secs(60))
.unwrap();
session.lock().unwrap().enable(true).unwrap();
hub.disconnect(&connection, true);
}
assert_eq!(
fs::read_dir(dir.path())
.unwrap()
.filter(|entry| {
entry
.as_ref()
.unwrap()
.path()
.extension()
.is_some_and(|ext| ext == "wal")
})
.count(),
0
);
drop(hub);
assert!(Hub::persistent(dir.path()).unwrap().get("a@x/r").is_none());
}
#[test]
fn offline_checkpoint_repairs_exports_removes_redo_and_allows_legacy_restart() {
let dir = tempfile::tempdir().unwrap();
let (token, _) = committed(dir.path());
let json = dir.path().join(format!("{token}.json"));
fs::write(&json, b"{torn").unwrap();
assert_eq!(
rust_xmpp::session_store::checkpoint_exports(dir.path()).unwrap(),
1
);
assert!(!dir.path().join(format!("{token}.wal")).exists());
let export: serde_json::Value = serde_json::from_slice(&fs::read(json).unwrap()).unwrap();
assert_eq!(export["pending"][0], "durable");
let hub = Hub::persistent(dir.path()).unwrap();
let session = hub.get("a@x/r").unwrap();
assert_eq!(
session.lock().unwrap().pending.front().map(String::as_str),
Some("durable")
);
session.lock().unwrap().mark_handled().unwrap();
assert!(dir.path().join(format!("{token}.wal")).exists());
}
#[test]
fn offline_checkpoint_refuses_live_store_and_preserves_redo() {
let dir = tempfile::tempdir().unwrap();
let hub = Hub::persistent(dir.path()).unwrap();
let (tx, _rx) = mpsc::channel(8);
let session = hub.bind("a@x/r", "c", tx, Duration::from_secs(60)).unwrap();
session.lock().unwrap().enable(true).unwrap();
let token = session.lock().unwrap().resume_id.clone().unwrap();
assert!(rust_xmpp::session_store::checkpoint_exports(dir.path()).is_err());
assert!(dir.path().join(format!("{token}.wal")).exists());
}
#[test]
fn offline_checkpoint_durably_applies_expiration_and_tombstones() {
let dir = tempfile::tempdir().unwrap();
let hub = Hub::persistent(dir.path()).unwrap();
let (tx, _rx) = mpsc::channel(8);
let session = hub.bind("a@x/r", "c", tx, Duration::from_secs(60)).unwrap();
let token;
{
let mut state = session.lock().unwrap();
state.enable(true).unwrap();
token = state.resume_id.clone().unwrap();
state.expires = Some(std::time::Instant::now());
state.checkpoint().unwrap();
}
drop(hub);
assert_eq!(
rust_xmpp::session_store::checkpoint_exports(dir.path()).unwrap(),
0
);
assert!(!dir.path().join(format!("{token}.json")).exists());
assert!(!dir.path().join(format!("{token}.wal")).exists());
assert!(Hub::persistent(dir.path()).unwrap().get("a@x/r").is_none());
}
#[test]
fn superseded_frames_still_require_valid_checksums_json_and_baseline_flags() {
use sha2::{Digest, Sha256};
for damage in ["checksum", "json", "baseline"] {
let dir = tempfile::tempdir().unwrap();
let (token, _) = committed(dir.path());
let wal = dir.path().join(format!("{token}.wal"));
let mut bytes = fs::read(&wal).unwrap();
// Damage the baseline, leaving both later committed snapshots intact.
let mut len = u32::from_le_bytes(bytes[8..12].try_into().unwrap());
let mut payload = 48..48 + len as usize;
match damage {
"checksum" => bytes[payload.end - 1] ^= 1,
"json" => bytes[payload.end - 1] = b'!',
"baseline" => {
let start = bytes[payload.clone()]
.windows(4)
.position(|part| part == b"true")
.unwrap()
+ payload.start;
// Preserve valid JSON, but make the first frame a continuation.
bytes.splice(start..start + 4, b"false".iter().copied());
len += 1;
payload.end += 1;
bytes[8..12].copy_from_slice(&len.to_le_bytes());
bytes[12..16].copy_from_slice(&(!len).to_le_bytes());
}
_ => unreachable!(),
}
if damage != "checksum" {
let checksum = Sha256::digest(&bytes[payload]);
bytes[16..48].copy_from_slice(&checksum);
}
fs::write(wal, bytes).unwrap();
assert!(Hub::persistent(dir.path()).is_err(), "{damage}");
}
}
+45 -6
View File
@@ -5,6 +5,7 @@ Usage: python3 tests/sm_restart_network.py BINARY
Pending server stanzas are at-least-once: a received but unacknowledged stanza
must replay. This test does not assert exactly-once client stanza processing.
"""
import hashlib
import json
from pathlib import Path
import signal
@@ -23,6 +24,36 @@ SM = 'urn:xmpp:sm:3'
ROOM = 'durable@rooms.localhost'
def durable_slot(root, token):
"""Read the authoritative checksummed WAL, never a potentially stale export."""
paths = list((root/'data').rglob(token + '.wal'))
assert len(paths) == 1, ('missing or duplicate durable SM journal', paths)
data = paths[0].read_bytes()
assert data[:8] == b'XMPPSM01', 'invalid durable SM journal header'
offset, frames, record = 8, 0, None
while offset < len(data):
# An abrupt kill can leave an incomplete final append. Recovery uses
# the preceding complete frame; an incomplete baseline is never valid.
if len(data)-offset < 8:
break
size = int.from_bytes(data[offset:offset+4], 'little')
inverse = int.from_bytes(data[offset+4:offset+8], 'little')
assert size ^ inverse == 0xffffffff and size <= 8*1024*1024+1024, 'invalid durable SM frame length'
end = offset+40+size
if end > len(data):
break
payload = data[offset+40:end]
assert hashlib.sha256(payload).digest() == data[offset+8:offset+40], 'invalid durable SM frame checksum'
frame = json.loads(payload)
assert frame['baseline'] == (frames == 0), 'invalid durable SM baseline sequence'
record = frame['snapshot']
frames += 1
offset = end
assert frames and record is not None, 'missing committed live SM snapshot'
assert record['resume_id'] == token, 'durable SM token mismatch'
return record
def enable(client):
client.send(f"<enable xmlns='{SM}' resume='true'/>")
response = client.take(lambda e: e.tag == f'{{{SM}}}enabled')
@@ -115,7 +146,7 @@ def run(binary):
client = Client(port, cert, user, 'secret', resource)
clients.append(client)
return client
def restart(how):
def restart(how, before_start=None):
nonlocal server
if how == 'kill':
server.send_signal(signal.SIGKILL)
@@ -125,6 +156,8 @@ def run(binary):
for client in clients:
client.close()
clients.clear()
if before_start:
before_start()
server = start(binary, config, port, log)
try:
server = start(binary, config, port, log)
@@ -139,7 +172,15 @@ def run(binary):
alice.send("<message to='bob@localhost'><body>incoming before crash</body></message>")
bob.body('incoming before crash')
assert handled(alice) == 3
restart('kill')
def corrupt_export():
record = durable_slot(root, token)
assert record['handled'] == 3
exports = list((root/'data').rglob(token + '.json'))
assert len(exports) == 1
exports[0].write_text('{torn JSON export')
# The fsynced journal must repair a torn readable export and
# retain the latest handled counter and pending IQ.
restart('kill', corrupt_export)
for user, ident, h in [('bob', token, 1), ('alice', 'unknown-token', 1), ('alice', token, 99)]:
resume(port, cert, ident, h, clients, user=user, success=False)
alice, result = resume(port, cert, token, 1, clients)
@@ -203,9 +244,7 @@ def run(binary):
assert handled(alice) == 5
# Inspect the committed slot before any Bob authentication.
# A history replay on rejoining is not offline SM delivery.
slots = list((root/'data').rglob(bob_token + '.json'))
assert len(slots) == 1, slots
record = json.loads(slots[0].read_text())
record = durable_slot(root, bob_token)
assert record['jid'] == 'bob@localhost/room', record
queued = [ET.fromstring(xml) for xml in record['pending']]
assert any(packet.get('from') == ROOM + '/Alice'
@@ -256,7 +295,7 @@ def run(binary):
if sys.exc_info()[0] is not None:
log.seek(0)
print(log.read(), file=sys.stderr)
print('PASS: TLS legacy/SASL2 durable SM after SIGKILL/SIGTERM, counters, ack pruning, at-least-once pending replay, owner/token/h rejection, transport/resource conflict, MUC activation/offline fanout, grace expiry and account deletion/recreation revocation')
print('PASS: TLS legacy/SASL2 durable SM after SIGKILL/SIGTERM, WAL repair of torn JSON export, counters, ack pruning, at-least-once pending replay, owner/token/h rejection, transport/resource conflict, MUC activation/offline fanout, grace expiry and account deletion/recreation revocation')
if __name__ == '__main__':
+2 -2
View File
@@ -353,7 +353,7 @@ fn purge_account_exact_files_preserves_others_and_extras() {
let root = t.path();
for store in ["accounts", "roster", "pep_aabbcc", "custom_store"] {
for user in ["alice", "bob"] {
for ext in ["dat", "list", "lidx", "rappend"] {
for ext in ["dat", "list", "lidx", "rappend", "rredo"] {
let p = storage::path(root, "example.test", Some(user), store).with_extension(ext);
std::fs::create_dir_all(p.parent().unwrap()).unwrap();
std::fs::write(&p, b"fixture").unwrap();
@@ -367,7 +367,7 @@ fn purge_account_exact_files_preserves_others_and_extras() {
std::fs::write(&hostwide, b"keep").unwrap();
a::remove_account(root, "example.test", "alice").unwrap();
for store in ["accounts", "roster", "pep_aabbcc", "custom_store"] {
for ext in ["dat", "list", "lidx", "rappend"] {
for ext in ["dat", "list", "lidx", "rappend", "rredo"] {
assert!(
!storage::path(root, "example.test", Some("alice"), store)
.with_extension(ext)