initial commit

scripts/prosody_compat/README.md

@@ -0,0 +1,66 @@
+# Exact Prosody compatibility oracle
+
+This harness executes the supplied **Prosody cafd74b9c5ea** implementation with a real Lua 5.4 interpreter and its compiled native modules. It uses the original `startup.make_dummy_hosts` (the upstream prosodyctl bootstrap), `core.modulemanager`, `mod_storage_internal`, and `mod_auth_internal_hashed`. Storage, authentication, stanza serialization and binary archive indexing are not replaced with test doubles. It does not launch a network listener or access a production data directory.
+
+The storage/auth ignored integration test creates a new temporary directory for each of SHA-1 and SHA-256. It verifies:
+
+1. Rust creates mixed-key/binary key/value data, preferences, accounts and archives.
+2. Original Prosody loads those records, verifies passwords, rotates passwords, creates another account with a binary salt, reads XML namespace attributes, appends and replaces archive entries, and builds a real `.lidx` binary index.
+3. Rust reads the modified records and verifies the new passwords, appends another record without rewriting history, and invalidates the upstream index.
+4. Original Prosody reads the final records again and authenticates the password changed by Rust.
+
+This caught and fixed real namespace interoperability problems: canonical Prosody attributes use the `\1` namespace separator, and reserved XML attributes are represented as literal `xml:lang`, etc. Reading also accepts the historical `|` separator.
+
+## Reproduce on Debian/Ubuntu
+
+Install build dependencies using your normal package manager: `lua5.4`, `liblua5.4-dev`, `lua-filesystem`, `lua-expat`, `lua-socket`, `lua-sec`, `libssl-dev`, `libicu-dev`, a C compiler, and `make`.
+
+Extract the supplied source archive into a new test-only directory, then build:
+
+```bash
+cd prosody-cafd74b9c5ea
+./configure --ostype=debian --lua-version=5.4 --lua-suffix=5.4
+make -j2
+```
+
+From the Rust project:
+
+```bash
+scripts/prosody_compat/verify.sh /absolute/path/to/prosody-cafd74b9c5ea lua5.4
+```
+
+`CARGO` can select a cargo executable. `RUSTUP_TOOLCHAIN` defaults to the project toolchain `1.99.1`. If Lua dependencies are extracted to a private prefix instead of installed, set `LUA_PATH` and `LUA_CPATH` to that prefix before invoking the script.
+
+## Validation performed in this workspace
+
+The source was built with GCC against Lua 5.4.6, OpenSSL 3, and ICU 74. Real Lua modules came from Ubuntu packages extracted into a private prefix. The oracle completed all four native phases for SHA-1 and SHA-256 successfully. This verifies the tested persistence and password interoperability, **not full Prosody protocol parity**.
+
+## Additional native module oracles
+
+The same verification script also runs additional ignored tests against unmodified native modules:
+
+- `preauth_native`: actual mod_invites consumes a Rust token and creates a native token consumed by Rust; scoped tokens remain inaccessible to native code.
+- `pubsub_native`: actual `mod_pep`/`util.pubsub` read Rust items, publish a replacement, and Rust reads it.
+- `muc_native`: actual `mod_muc` restores a Rust room/affiliations; `mod_muc_mam` reads Rust `message<groupchat` records and archives a real native room broadcast; Rust reads the result.
+- `push` (ignored native lookup): original `util.stanza` confirms body extraction excludes foreign namespaces and nested extension elements.
+- `csi_native`: fifteen importance decisions are compared with the actual `mod_csi_simple`, including presence, chatstates, carbons, EME, Jingle, MUC invites and a configured custom payload.
+
+These use actual Prosody modules and storage. They do not emulate a networked fleet of real clients or external servers.
+
+## Native federation transport test
+
+The optional harness launches the built original Prosody in a temporary directory
+and exchanges real stanzas with Rust over two TCP STARTTLS + SASL EXTERNAL
+connections. A generated test CA anchors both certificate chains. Identity and
+chain checks remain enabled. Only the test TCP routing resolver is overridden;
+the asserted XMPP domain is unchanged.
+
+```bash
+PROSODY_SOURCE=/absolute/path/to/built/prosody-cafd74b9c5ea \
+PROSODY_LUA=lua5.4 scripts/native-federation.sh
+```
+
+For dependencies extracted to a private prefix, set `PROSODY_NATIVE_PREFIX` to
+its `usr` directory. The test checks the expected archive revision before
+launching. It is isolated from live configuration/data and validates both message
+IDs and bodies, plus Prosody's TLS/EXTERNAL authentication status.

scripts/prosody_compat/oracle.lua

@@ -0,0 +1,77 @@
+-- Runs the supplied Prosody's REAL bootstrap/module manager/storage/auth implementations.
+local source, root, phase = assert(arg[1]), assert(arg[2]), assert(arg[3]);
+package.path = source .. '/?.lua;' .. package.path;
+package.cpath = source .. '/?.so;' .. package.cpath;
+dofile(source .. '/loader.lua');
+CFG_SOURCEDIR, CFG_PLUGINDIR, CFG_DATADIR = source, source .. '/plugins', root;
+local startup = require 'prosody.util.startup';
+startup.init_global_state();
+local config = require 'prosody.core.configmanager';
+config.set('*', 'data_path', root);
+config.set('*', 'network_backend', 'select');
+config.set('example.org', 'authentication', 'internal_hashed');
+config.set('example.org', 'password_hash', arg[4] or 'SHA-256');
+startup.setup_datadir();
+startup.load_libraries();
+startup.make_dummy_hosts(); -- The original prosodyctl bootstrap, not a test replacement.
+local sm = require 'prosody.core.storagemanager';
+sm.initialize_host('example.org');
+local kv = assert(sm.open('example.org', 'oracle'));
+local archive = assert(sm.open('example.org', 'archive', 'archive'));
+local st = require 'prosody.util.stanza';
+local mm = require 'prosody.core.modulemanager';
+assert(mm.load('example.org', 'auth_internal_hashed'));
+local authmod = assert(mm.get_module('example.org', 'auth_internal_hashed'));
+local auth = assert(mm.get_items('auth-provider', 'example.org')[1]);
+if phase == 'probe' then
+  assert(kv:set('probe', {hello='actual Prosody'}));
+  assert(kv:get('probe').hello == 'actual Prosody');
+  print('PROSODY_NATIVE_PROBE_OK');
+elseif phase == 'roundtrip' then
+  local value = assert(kv:get('alice'));
+  assert(value[false] == 'roster');
+  assert(value[3] == 'sparse');
+  assert(value.binary == string.char(0, 1, 10, 255));
+  assert(value.text == 'Привет & <xml>');
+  value.native = {changed=true, number=1.25};
+  assert(kv:set('alice', value));
+  local prefs = assert(sm.open('example.org', 'archive_prefs'));
+  assert(prefs:get('alice')[false] == 'roster');
+  assert(prefs:set('alice', {[false]=false, ['bob@example.org']=true}));
+  assert(auth.test_password('alice', 'пароль-Original'));
+  assert(not auth.test_password('alice', 'wrong'));
+  assert(auth.set_password('alice', 'пароль-Native'));
+  assert(auth.create_user('bob', 'native-bob'));
+  local ok, stored, server = require('prosody.util.sasl.scram').hashers[arg[4] or 'SHA-256']('binary-password', string.char(0,255)..'salt', 10000);
+  assert(ok);
+  local hex = require 'prosody.util.hex';
+  assert(sm.open('example.org', 'accounts'):set('binary', {salt=string.char(0,255)..'salt', iteration_count=10000,stored_key=hex.encode(stored),server_key=hex.encode(server)}));
+  assert(auth.test_password('binary', 'binary-password'));
+  local found = {};
+  for id, stanza, when, peer in assert(archive:find('alice')) do
+    found[#found+1] = id;
+    assert(stanza.name == 'message');
+    assert(stanza.attr['xml:lang'] == 'ru');
+    assert(stanza.attr['urn:oracle'..string.char(1)..'flag'] == 'yes');
+    assert(stanza:get_child_text('body') == 'Rust body');
+    assert(when == 1767225600 + #found - 1);
+    assert(peer == 'bob@example.org');
+    assert(stanza.attr.stamp == nil);
+  end
+  assert(#found == 2 and found[1]=='rust-one' and found[2]=='rust-two');
+  local native = st.message({xmlns='jabber:client',from='bob@example.org',to='alice@example.org',['xml:lang']='ru', ['urn:oracle'..string.char(1)..'flag']='native'}, 'Native body');
+  assert(archive:append('alice', 'native-three', native, 1767225602, 'bob@example.org'));
+  assert(archive:append('alice', 'rust-two', st.message({xmlns='jabber:client'}, 'Replaced by native'), 1767225603, 'bob@example.org'));
+  local iter = assert(archive:find('alice', {after='rust-one', with='bob@example.org'}));
+  assert(iter() == 'native-three');assert(iter() == 'rust-two');assert(iter() == nil);
+  -- This exercises original binary .lidx creation, not just .list loading.
+  print('PROSODY_NATIVE_ROUNDTRIP_OK '..(arg[4] or 'SHA-256'));
+elseif phase == 'verify-final' then
+  assert(kv:get('alice').rust_final == true);
+  assert(auth.test_password('bob', 'rust-bob-new'));
+  local count=0;for id, stanza in assert(archive:find('alice')) do count=count+1;if count==4 then assert(id=='rust-final');assert(stanza:get_child_text('body')=='Final');end end
+  assert(count==4);
+  print('PROSODY_NATIVE_FINAL_OK '..(arg[4] or 'SHA-256'));
+else
+  error('unknown phase');
+end

scripts/prosody_compat/verify.sh

@@ -0,0 +1,20 @@
+#!/usr/bin/env bash
+set -euo pipefail
+if [[ $# -lt 1 || $# -gt 2 ]]; then
+  echo 'Usage: verify.sh /absolute/path/to/built/prosody-cafd74b9c5ea [lua5.4]' >&2
+  exit 2
+fi
+project_dir=$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/../.." && pwd)
+reference_dir=$(cd -- "$1" && pwd)
+if ! grep -q '^node: cafd74b9c5ea' "$reference_dir/.hg_archival.txt"; then
+  echo 'Expected the supplied Prosody cafd74b9c5ea source revision.' >&2
+  exit 2
+fi
+for required in util/encodings.so util/hashes.so util/pposix.so plugins/mod_storage_internal.lua; do
+  test -f "$reference_dir/$required" || { echo "Missing $required; build Prosody first." >&2; exit 2; }
+done
+export PROSODY_REFERENCE_SOURCE="$reference_dir"
+export PROSODY_REFERENCE_LUA="${2:-lua5.4}"
+export RUSTUP_TOOLCHAIN="${RUSTUP_TOOLCHAIN:-1.99.1}"
+cd -- "$project_dir"
+"${CARGO:-cargo}" test --locked --test reference_prosody --test pubsub_native --test pep_item_order --test muc_native --test muc_state_native --test csi_native --test push --test preauth_native -- --ignored --nocapture