Skip to content

development.md

Latest commit

 

History

History
615 lines (529 loc) · 38.3 KB

development.md

File metadata and controls

615 lines (529 loc) · 38.3 KB
Raw

Developing Webjet

How the code is organized, how to build and test it, and the conventions and pitfalls that aren't obvious from the code. The design itself is in webjet-architecture.md; section numbers (§) in code comments refer to it.

Workspace

Crate Depends on Notes
webjet-proto FlatBuffers types, generated by planus.
webjet-core proto Pure, deterministic logic: no I/O, no clocks except where passed in.
webjet-net core, proto Transports. Feature anytls (default) pulls in BoringSSL.
webjet-release core, proto What webjetup, webjetd and webjetctl share about releases: targets, file names, version order, the remote upgrade spool (QueuedRequest, Trust, plan: the helper's own check of a signed UpgradeRequest, §19.8) and, with feature http, the release server client.
webjetd release, core, net, proto The agent (library + binary). Feature anytls forwards to webjet-net. upgrade.rs: remote upgrade requests (check, queue for the helper, pass on as a controller).
webjet-tools webjetd, release, core, net, proto webjetctl (network, local, root); config.rs is the TOML network/node configuration document (parse, scope checks, wire conversion, show, get/set/unset); also the shared account-key code (Account::unlock, sign); upgrade.rs pins a release, signs per-node UpgradeRequests and orders a rollout.
webjetup release Installation manager; remote.rs is the remote upgrade helper (webjetup apply-upgrade) and its systemd/launchd/scheduled-task triggers.
webjet-app webjetd, tools, core, proto GPUI desktop app. model, client, graph (network graph and layout) and openmetrics are GUI-free and unit-tested; ui is the GPUI front end, one file per tab, rendered headless in a test.

Edition 2024, rust-version = "1.88".

Building

cargo build --workspace                       # needs cmake and Go for BoringSSL
cargo build --workspace --no-default-features # without AnyTLS: no cmake or Go needed

A C compiler is always needed: aws-lc (rustls) and zstd (zstd-sys, control plane compression, architecture §9.8) are C libraries built by their crates.

BoringSSL makes the first build slow (several minutes). For quick iteration on code that doesn't touch AnyTLS, use --no-default-features.

Checks before committing

cargo fmt
cargo clippy --workspace --all-targets
cargo clippy --workspace --all-targets --no-default-features
cargo test --workspace
scripts/deny.sh                                   # dependencies (cargo-deny)
scripts/fuzz-lock.sh && cargo test --release --manifest-path fuzz/Cargo.toml --no-default-features  # fuzz targets, smoke

All of them must be clean: the workspace has no clippy warnings, including in tests.

  • No unwrap()/expect() outside tests. The workspace lints warn on clippy::unwrap_used and clippy::expect_used; clippy.toml allows them in tests. Parse untrusted bytes with the checked readers in webjet-core/src/bytes.rs, which return None on short input, and drop bad packets instead of panicking. Sort floats with total_cmp.
  • The generated FlatBuffers module is exempt (#[rustfmt::skip] and lint allows in webjet-proto/src/lib.rs). Don't edit generated.rs by hand.
  • Some code is compiled only on one OS (cfg(target_os = "linux"), cfg(windows)). Clippy on macOS doesn't see it, so check the other targets too when you touch platform code:
    • Linux: docker run --rm -v "$PWD":/src -w /src rust:1-bookworm cargo clippy --workspace --exclude webjet-app --all-targets (or build the Docker image). webjet-app is left out because GPUI links against libxkbcommon, libxkbcommon-x11 and libxcb, which the plain Rust image doesn't have (apt-get install libxkbcommon-dev libxkbcommon-x11-dev libxcb1-dev adds them; the CI image has them).
    • Windows: cargo xwin clippy --target x86_64-pc-windows-msvc -p webjetd -p webjet-tools --no-default-features.
    • FreeBSD (OPNsense): scripts/freebsd-cargo.sh clippy -p webjetd -p webjet-tools -p webjetup --all-targets -- -D warnings. The script runs cargo for x86_64-unknown-freebsd with Clang and lld against a FreeBSD 14.4 sysroot, which scripts/freebsd-sysroot.sh downloads to ~/.cache/freebsd-sysroot/x86_64 the first time (base.txz, checksum pinned). On a Mac it needs Homebrew's llvm and lld; elsewhere WEBJET_FREEBSD_LLVM names their directory if they aren't on PATH. scripts/freebsd-cargo.sh build --release -p webjetd -p webjet-tools -p webjetup builds the programs. They run on FreeBSD 14.0 and later 14.x (OPNsense 24.7 and later): scripts/release.sh refuses a build that needs a libc symbol newer than 14.0's (FBSD_1.7).

Dependencies

deny.toml configures cargo-deny, which scripts/deny.sh runs (it installs a pinned release on Linux if cargo-deny isn't on PATH; on a Mac use cargo install --locked cargo-deny). It fails on:

  • a RustSec advisory (vulnerable, unsound, yanked or unmaintained crate);
  • a licence that isn't in the allow list (permissive licences and MPL-2.0);
  • a crate from anywhere but crates.io.

An unmaintained crate that can't be replaced yet goes into [advisories] ignore with the reason (today: crates pinned by gpui 0.2.2 in webjet-app). For a vulnerability, update the crate (cargo update -p <crate>) instead; ignore it only if the vulnerable code is unreachable, and say why. Duplicate versions are reported as warnings (scripts/deny.sh bans lists them). The workspace crates are publish = false, so path dependencies without a version are fine.

gpui, gpui_media, gpui_http_client and reed-solomon-erasure come from patched copies in vendor/ ([patch.crates-io]; vendor/README.md says what changed and how to update them), so cargo update never moves them to a new release. reed-solomon-erasure is patched to pick its SIMD kernels at run time (AVX2, SSSE3 or the portable code on x86_64), so one release build runs fast on new CPUs and still runs on old ones. gpui_http_client is patched only so that it no longer pulls in xattr 0.2, which needs the Linux constant libc::ENOATTR that libc 0.2.190 removed; libc can be updated freely. scripts/gpui-vendor-check.sh asks crates.io whether there is a newer one (exit 1 if so). The pre-commit hook in .githooks/ runs it with --hook: a warning only, looked up at most once a day and skipped without network. Enable the hook once per clone with git config core.hooksPath .githooks (WEBJET_SKIP_GPUI_CHECK=1 or git commit --no-verify skips it).

Fuzzing

fuzz/ fuzzes every parser that reads bytes from the network or a local client (§20). It is its own workspace, outside cargo test --workspace. Its Cargo.lock is not committed: it is a copy of the main Cargo.lock (scripts/fuzz-lock.sh, which CI and scripts/fuzz.sh run first), so the targets build against the same dependency versions as the programs and the lock never needs updating by hand. Without the copy, cargo resolves the fuzz workspace on its own (newest versions). fuzz/src/lib.rs has one function per target:

Target What it feeds
proto Every FlatBuffers message type (decode, re-encode, decode again), signed envelopes, tree leaves
frames e2e frame headers, handshake and relay frames, AEAD open, the replay window; seal/open round trips
packet IP packets: parsing, NAT rewrite, ECN, ICMP replies, LZ4 round trip
decompress zstd and LZ4 decompression of hostile input: the size cap holds, compressed data round-trips
gate The UDP gate (UdpObfs::open) and seal/open round trips
kcp_session FEC shards and KCP segments after the gate, as a session processes them, and raw packets into KCP (kcp::fuzzing)
hop_record Hop record framing (TLS/AnyTLS/KCP hops), CONTROL_ZSTD payloads, HopControl
dns The local resolver: client queries and forwarded zones' answers (dns::fuzzing)
control_request Control-channel request lines (control::Request): parse, and parsed requests round-trip
sntp SNTP answers from time servers (sntp::parse); a well-formed answer gives the exact offset
multicast_report Multicast membership reports from peers (multicast::decode_report): decoded reports round-trip
padding Padding schemes from the TransportConfig (padding::parse_line, Scheme::from_wire): plans carry the whole payload

Two ways to run them:

# Stable, seconds: seeds plus random mutations (CI runs this).
scripts/fuzz-lock.sh
cargo test --release --manifest-path fuzz/Cargo.toml --no-default-features
WEBJET_FUZZ_ITERS=100000 WEBJET_FUZZ_SEED=1 cargo test ...   # longer / repeatable

# Coverage-guided with libFuzzer (nightly + cargo-fuzz):
rustup toolchain install nightly --profile minimal && cargo install cargo-fuzz
scripts/fuzz.sh 300            # every target for 300 s
scripts/fuzz.sh 0 dns          # one target until stopped (Ctrl-C)

fuzz.sh starts from the seeds in fuzz/src/seeds.rs (valid messages of every kind). Corpus and findings stay in fuzz/corpus/ and fuzz/artifacts/ (gitignored). For a finding: reproduce it with cargo +nightly fuzz run <target> fuzz/artifacts/<target>/<file>, fix the bug, and commit the input as fuzz/regressions/<target>/<name>; the smoke test replays every file there. A target fails by panicking, so new targets should assert the invariants they can (round trips, size caps), not only call the parser. When adding a parser for untrusted input, add a target and seeds for it.

KCP is our own adaptation of the kcp crate 0.6.0 (MIT, crates/webjet-net/src/kcp/proto.rs, architecture §17.5). Its sequence numbers and millisecond timestamps wrap by design: use wrapping_* and the diff helper, never plain +/-, and keep malformed input an Err, not a panic. The kcp-wrapping-sn and kcp-wrapping-ts regressions came from the crate's plain arithmetic.

Wire formats

Schemas are crates/webjet-proto/schema/webjet.fbs (signed objects, tree leaves) and wire.fbs (hop messages, RPC, Raft). After editing them, run crates/webjet-proto/regen.sh (needs cargo install planus-cli --version 1.3.0) and commit the regenerated src/generated.rs.

  • planus 1.3 has no fixed-size arrays in structs, so 32-byte keys, 64-byte signatures, 16-byte ids and CIDRs are structs of little-endian u64 words (b0..b3) with the same byte layout. Their JSON form (control channel, --json) is hand-written in webjet-proto/src/serde_impls.rs: ids, keys and signatures as hex, addresses as ip, ip:port, ip/prefix. regen.sh strips the derived serde impls of those structs; reading still accepts the old {"b0": …} form.
  • Only append fields to tables, never reorder or remove them: older nodes must still decode new messages (§15.7). New union variants go at the end.
  • Signed objects are envelopes: type || body, signed with Ed25519 over a context string per message type (webjet-proto message-type registry).
  • New hop record types (webjet-net hop::rec) and e2e frame types (webjet-core frames::ftype) are sent only to peers that advertise a feature for them in their EndpointRecord (gossip::features); older nodes drop types they don't know. Compression works this way: CONTROL_ZSTD, RPC_ZSTD, DATA_LZ4, PROXY_LZ4 behind the zstd/lz4 features (webjetd/src/compression.rs, webjet-core/src/compress.rs); broadcast and multicast (MCAST, GROUPS) behind multicast (webjetd/src/multicast.rs, webjet-core/src/multicast.rs).

The controller state machine

webjet-core/src/statemachine.rs applies committed log entries. It must be deterministic: every controller computes the same tree from the same log (§15.5).

  • No wall clock: use the entry's timestamp.
  • No randomness: use the entry's random fields, chosen by the leader when it prepares the entry.
  • Only ordered maps (BTreeMap/BTreeSet) where iteration order matters.

The leader prepares an entry: it fills in the timestamp, the random values and the resulting leaf bytes. Followers recompute the leaves and halt on any difference, so a bug that makes replicas disagree shows up as a halted follower, not as silent divergence.

Raft

webjetd/src/controller/raft.rs is a pure state machine: time is passed in, randomness comes from a seeded RNG, storage can be in memory. Pre-vote, check-quorum, single-server membership changes tracked from the log (classify_entry in controller/mod.rs).

Testing it:

cargo test -p webjetd --lib raft                                          # ~1 min
WEBJET_RAFT_SEEDS=2000 cargo test --release -p webjetd --lib randomized   # soak
WEBJET_RAFT_STATS=1 WEBJET_RAFT_TRACE=1 ...                               # per-seed stats / seed being run
scripts/raft-mutations.py                                                 # ~10 min

raft_tests.rs has targeted scenarios and a simulator. The simulator injects crashes, restarts, partitions, message loss, duplication, reordering, snapshots and membership changes, checks the Raft safety properties after every step, and checks liveness once faults stop.

scripts/raft-mutations.py re-introduces ten classic Raft bugs one at a time and fails if the tests miss one. Run it after changing raft.rs or the tests. Its patterns match the source with any whitespace; if a refactor changes the code it looks for, it reports a setup error (exit status 2) and the pattern must be updated.

Offline recovery (controller/recovery.rs, ops guide 4.1) writes into a stopped controller's log: force-new-cluster replays the log, and restore installs an exported ControllerSnapshot. Either way it appends one ForceNewClusterEntry in a new term this server voted itself in (Raft::append_offline), and classify_entry turns it into Set with one voter. When the log is below --min-version, it is replaced by a snapshot at that index first, so the new tree heads are above any version a lost controller signed. Machine::prepare refuses the entry, so a running leader never proposes one (prepare_offline is for this path only).

When a randomized run fails, rerun the failing test with WEBJET_RAFT_TRACE=1 to find the seed, then reproduce with that one seed. Every failing seed so far has been a real bug or a test-harness bug (for example, waiting for a value whose leader was deposed), never noise.

Locking

The agent shares state between many tasks; deadlocks have been the most common bug. Rules:

  • Never call a function that takes a lock while holding another lock it might take. Clone what you need, drop the guard, then call. Especially: Node::records, Node::hops, the store, a session's inner, and Node::circuits.
  • Node::store.read() is recursive (read_recursive), so nested reads are fine, but never take write() while holding a read.
  • Code that needs only the derived view, the overlay pool or the store's generation reads n.store.snapshot() instead: no lock, so packets never wait behind a writer rebuilding the view. The write guard republishes the snapshot when it drops, if the store's generation moved, so every change to the store must bump Store::generation. Don't hold a snapshot guard across .await (snapshot_arc() gives an owned one).
  • Never hold a parking_lot guard across .await.
  • Helpers that iterate over all sessions or hops (e.g. counting handshakes in progress) must not run while the caller holds one of them; take the count first.
  • In edition 2024 a temporary in an if let or a let-chain lives until the end of the whole if, including its body. So if let Some(c) = n.circuits.lock().first() && send(c) holds the circuit lock while send runs. Bind the value with let first, then test it. This deadlocked the e2e re-key (found by scripts/soak-test.sh).

sample <pid> (macOS) or gdb -p <pid> -batch -ex 'thread apply all bt' (Linux) on a hung agent shows which locks the threads wait on.

Agent tasks

The agent's long-running loops are owned by Node::tasks (webjetd/src/tasks.rs), not started with a bare tokio::spawn:

  • n.tasks.service(name, fut) (or service_result) for a loop that must run as long as the agent: gossip, e2e, paths, metrics, health, punch, connections, secrets, the controller, the hop listeners, the tun reader and writer, the data plane refresh. If one returns, fails or panics, the agent logs it and exits with an error, and the service manager restarts it (systemd Restart=on-failure, launchd KeepAlive, Docker's restart policy). Before, such a task died silently and left a running agent that no longer gossiped or routed.
  • n.tasks.oneshot(name, fut) for a task that may finish (starting the data plane, port mapping, the control socket and metrics endpoint, whose errors are logged but don't stop the overlay). Only a panic there is fatal.
  • At shutdown abort_all stops every task before the agent persists its state and removes routes and DNS settings.

Per-connection and per-request tasks stay plain tokio::spawn; a panic ends only that connection. Every panic is logged through tracing (the panic hook from tasks::install_panic_hook, installed by logging::init).

The control channel

Every local user can talk to the agent's control socket (control/), so every command is listed in control::COMMANDS with its access class: Read (shows state; changes nothing), Signed (only acts on material whose account, admin, root or emergency signature is checked) or Capability (needs a secret, the enrollment token). Requests are one typed enum, control::Request (control/request.rs), which webjetd parses and webjetctl, webjet-app and webjetctl local enroll send through control::client::call; its wire form is the plain JSON line {"cmd": "...", <arguments>} (binary arguments base64, node ids hex), so agents and clients of other versions still understand each other, and a new argument must be optional. handle refuses commands that aren't in the table, then parses the arguments and hands the Request to control/commands.rs (one function per command). Unit tests check that the table and the enum name the same commands, that the enum writes the same JSON as before, and that only the known state-changing commands are not Read. A new command needs a variant and an entry; a new state-changing one also has to be added to that test, together with its signature check. Requests are limited to 4 MiB per line and 64 connections at once. log and snapshot take admin-signed requests (controller::audit); a snapshot reply holds the PSKs and the camouflage key, so compression::secret_rpc keeps it uncompressed like Raft traffic.

The signer (controller/signer.rs) listens with ipc::Access::Private, or Group (mode 0660) with --group-access, which the systemd unit from webjetup install --signer uses: the signer runs as webjet-signer in the agent's group, so the agent can sign but never read the key. serve refuses Everyone.

Unit tests

cargo test --workspace --exclude webjet-app runs everything below in a few seconds; nothing needs root, a network or Docker.

  • A real network state in memory. webjet_core::testutil (built for this crate's tests, and for other crates with the testutil feature, as webjetd's dev-dependencies do) has net() (root, emergency and admin keys and a genesis) and Cluster: a controller state machine you enroll nodes into (data_node, enroll(NodeSpec { .. })) and configure (set_network, set_policy, set_node_config), which hands out stores that trust its tree (store(), sync()). testutil::packets builds checksum-correct IPv4 packets.
  • An agent without sockets. In webjetd, testnode::node(&mut cluster, id, key) gives a Node over such a store with a temporary data directory. Node::test_hop(peer, transport) registers a hop with no connection behind it; what the agent sends on it is read from the returned receivers (control() decodes the messages, relay() gives the relay frames). Dataplane::for_test(&node) is a data plane without a tun interface whose tun-bound packets arrive on a channel. The circuit, data plane, reverse proxy, path, metrics, compression, state and controller modules are tested this way.
  • Property tests (proptest) compare code with a simple model of what it should do: policy evaluation and port sets, path search (against brute force over every path), the replay window, frames, packet rewriting and ICMP errors, connection tracking counters and eviction order, and the compressors. A failure shrinks to a small case and is saved in crates/webjet-core/proptest-regressions/; commit that file, so the case runs first from then on, and add a named regression test.
  • The offline root commands run as the real webjetctl binary in crates/webjet-tools/tests/root.rs.
  • Coverage: cargo llvm-cov --workspace --exclude webjet-app --summary-only (cargo install cargo-llvm-cov, rustup component add llvm-tools-preview).

Tests that need more than cargo test

Script Needs What it checks
scripts/local-net.sh [N] nothing (no root) A full network on 127.0.0.1 by operations guide 2.1, tun disabled. scripts/local-net.sh stop ends it.
scripts/cluster-test.sh nothing Three controllers, leader failover, writes still commit.
scripts/soak-test.sh [s] nothing Runs the local network past the e2e re-key interval (120 s) and checks every agent still answers. Catches timer-driven deadlocks.
cargo test -p webjet-app --test live scripts/local-net.sh 2 The app's sign-in and operations against a live network (admin vs. normal account), and the data behind its tabs. Skipped without a network.
docker/e2e.sh Docker, image webjet:test Real tun interfaces: kernel ping, magic DNS, policy, subnet gateway.
docker/realip-test.sh Docker, image webjet:test Reverse proxy real client addresses (architecture §11.11): a target without accept_real_ip sees the ingress, then turns it on and restarts its agent; new TCP/UDP flows and flows that kept running (UDP moves, TCP is reset and reconnects) get the client's address; turned off again, back to the ingress. WAN_PREFIX=10.88 if 172.30.0.0/24 is taken.
docker/multicast-test.sh Docker, image webjet:test Broadcast and multicast on five nodes (architecture §9.5.1): Bedrock-style discovery pings with unicast pongs, the pool broadcast, IGMP membership, the echo guard, the per-node budget and its alarm, turning it off per node. docker/mcast.py sends and counts datagrams.
docker/capture-test.sh Docker, image webjet:test; Go (or the golang image) builds docker/raknet-lan Capture (architecture §9.5.2) on four nodes: Bedrock clients broadcasting from 0.0.0.0 out of eth0, as the real game does. No world found without capture; with [multicast] capture every world found within seconds and RakNet play works; a node's capture = [] override; a datagram sent both ways carried once. About 3 minutes.
docker/mdns-test.sh Docker, image webjet:test (adds Avahi, nss-mdns and D-Bus as webjet:mdns) mDNS/DNS-SD on five nodes with Avahi on the tun interface, IPv4 and IPv6 pools: service browsing, .local names, ping and connect, policy keeping a node out, a node that hasn't joined 224.0.0.251 receiving nothing, goodbyes, steady-state bandwidth. About 3 minutes.
docker/raknet-storm-test.sh Docker, image webjet:test; Go (or the golang image) builds docker/raknet-lan Minecraft Bedrock LAN discovery on five nodes with real RakNet (go-raknet): every world seen, 20 RakNet connections, and whether it storms: aggressive clients, naive LAN bridges rebroadcasting every ping (exact and changed). Measures wire bandwidth against an idle baseline. About 5 minutes.
docker/nat-test.sh Docker, image webjet:test Hole punching through two NAT routers (MODE=symmetric for a symmetric NAT), a public node without NAT, a fake wg0 VPN that must not be used as a direct path, and loopback pings.
docker/load-test.sh Docker, image webjet:test Path stability under load: two data nodes behind shaped access links (netem), two relays. A bulk transfer must keep its circuit (at most one path switch), the busy circuit's probes pause, the idle one keeps being probed at the backup rate (§11.2, §11.4). SECS=, RATE=, DELAY= adjust it.
docker/deploy-test.sh Docker The deployment image as documented in docs/docker.md.
docker/tpm-test.sh Docker TPM key sealing against a software TPM (swtpm), refusal by another TPM.
docker/windows-test.sh Docker, cargo-xwin, LLVM Windows unit tests under Wine.

Notes:

  • local-net.sh uses fixed passphrases from the environment (WEBJET_ROOT_PASSPHRASE, WEBJET_ACCOUNT_PASSPHRASE) and keys in /tmp/webjet-local/keys. Query a node with target/debug/webjetctl --socket /tmp/webjet-local/node-02/control.sock local status.
  • macOS has no timeout; use perl -e 'alarm shift; exec @ARGV' 5 <cmd>.
  • docker/e2e.sh, docker/realip-test.sh, docker/multicast-test.sh, docker/nat-test.sh and docker/load-test.sh use the test image: docker build --target test -t webjet:test .
  • When the host already uses the test subnets, move them: WAN_PREFIX=10.88 (first two octets) for e2e.sh, realip-test.sh, multicast-test.sh, capture-test.sh, mdns-test.sh and raknet-storm-test.sh; INET_PREFIX=10.89.0 for nat-test.sh; NET_PREFIX=10.89.0 for deploy-test.sh (first three octets).
  • docker/tpm-test.sh sets TPM2TOOLS_TCTI to the software TPM and runs the TPM unit test with WEBJET_TEST_TPM=1. Without a kernel resource manager, every TPM operation must flush transient objects or the TPM runs out of slots after a few operations; the keystore does this.
  • Windows: install with cargo install cargo-xwin and brew install llvm (for clang-cl/lld-link). windows-test.sh runs each test in its own Wine process. On Apple Silicon, Wine runs under x86 emulation, which sometimes crashes the emulator ("rosetta error"); those tests are counted as EMU, not as failures. Wintun, NRPT and the service manager need a real Windows machine.
  • Shell scripts that create a file and run it in the same step: chmod +x first, or start them with sh script.sh.

Data path cost

e2e::tests::data_path_cost (ignored by default) times one 1200-byte packet through an established session on a relayed circuit: send (seal, circuit and hop choice, relay header, hop record) and on_frame (open, replay window, data plane). Run it in release before and after touching the hot path:

cargo test --release -p webjetd --lib e2e::tests::data_path_cost -- --ignored --nocapture

WEBJET_BENCH_ROUNDS sets the packet count (default 200 000); with a large count, sample <pid> (macOS) or perf record shows where the time goes. On an M-series Mac it measured about 2.8/2.5 µs per packet (send/receive) with RustCrypto's ChaCha20-Poly1305 and 1.5/1.3 µs after frames::SessionKey moved to aws-lc's (the same library rustls uses); the AEAD is now most of the remaining cost. The KCP gate (webjet_net::gate) still uses RustCrypto's XChaCha20-Poly1305, which aws-lc-rs doesn't offer.

Probe traffic

What a node sends and receives only to measure links and circuits: hop keepalives (§11.2: from the dialer only, every 2 s for 30 s then every 5 s, each answered) and end-to-end probes (§11.4: from the circuit's source only, 1 s until measured, then 5 s on the circuit in use and 15 s on a backup; only while the session was used in the last minute). Unit tests check the cadence with the agent's own tasks: probe_traffic::keepalives_per_hop, probe_traffic::probe_record_sizes and e2e::handshake_tests::probe_traffic_on_an_idle_circuit. probe_traffic::wire::wire_cost (ignored by default) runs two agents (edge and relay: hop writer/reader/keepalive, path loop, session timers) over a real UDP+KCP+FEC+TLS and TCP+TLS hop on loopback through a counting proxy and prints bytes and packets per second:

cargo test -p webjetd --lib probe_traffic -- --ignored --nocapture

WEBJET_PROBE_SECS sets the measured window (default 30 s). An idle KCP hop measured ≈ 2.5 datagrams/s and ≈ 480 B/s each way (with IP/UDP headers) before 0.10.2, ≈ 0.4–0.7 datagrams/s and ≈ 130–150 B/s after.

Docker image

The root Dockerfile has a runtime target (default, the deployment image) and a test target (adds ping, dig, nc and the test scripts). Build caches are keyed by platform. docker/entrypoint.sh turns WEBJET_* variables into webjetd.toml and handles first-start enrollment; it writes values into TOML strings, so it rejects values containing quotes, backslashes or newlines. Everything an operator needs is in docs/docker.md.

Continuous integration

.forgejo/workflows/ runs on the project's Forgejo runner (linux/amd64, label ubuntu-latest), with every job in the builder image forgejo.wolf109909.top/webj8-internal/webjet-builder:5:

Workflow When What
ci.yml push to main, pull requests cargo fmt --check; clippy with -D warnings for Linux (with and without AnyTLS), Windows, FreeBSD and macOS (arm64); scripts/deny.sh; cargo test; the fuzz targets' smoke test and clippy. webjet-app is checked for every target; macOS and Windows cross-build through the patched gpui in vendor/ (vendor/README.md).
fuzz.yml manually only (Run workflow) Installs nightly Rust and cargo-fuzz, builds the targets, then runs scripts/fuzz.sh for 30 minutes in total, split evenly over the targets (input minutes); findings are uploaded as an artifact.
release.yml a pushed v* tag tests, then scripts/release.sh <tag> --publish for every target: Linux x86_64 natively; Linux arm64, Windows x86_64, FreeBSD x86_64 and macOS arm64 and x86_64 cross-compiled

Every job of ci.yml and release.yml first runs scripts/cargo-mirror.sh: the runners are in China, where crates.io is slow. It tries one cargo fetch --locked from each source in turn and keeps the first that has every crate in Cargo.lock: rsproxy.cn (index and crates both served in China; a cold fetch takes ~35 s on the runner, also with four jobs at once), then the sparse index of TUNA (a MirrorZ member; the MirrorZ endpoint mirrors.cernet.edu.cn itself doesn't answer from the runner), then USTC, then crates.io. TUNA mirrors only the index and syncs it periodically, so for a few hours after a cargo update it may lack the newest versions, and its crates download from static.crates.io (a cold fetch took 138 s in October 2026, beyond the 60 s limit below, which failed every job of v0.12.2's first runs). USTC proxies crates.io live but rate-limits (HTTP 429) or stops answering the runner, which is why it isn't earlier. Cargo retries each request up to 10 times (net.retry), and when no source works the script goes through the list again, up to three times a minute apart: the jobs of a run start together from one address. A source that doesn't answer its config.json within 10 s is skipped (USTC sometimes stops answering the runner instead of sending a 429, and cargo would wait on it for minutes), and a fetch is stopped after 60 s (FETCH_TIMEOUT); what it downloaded stays, so the next attempt continues. The choice is a source replacement in $CARGO_HOME/config.toml, so the later steps use it and the crates are already downloaded. Cargo.lock still names crates.io and cargo checks every crate against its checksum there, so --locked and cargo-deny's source check work as before. Other sources: scripts/cargo-mirror.sh <URL>... (crates.io for none). The runner's job environment (runner-config.yml) lists the mirrors in NO_PROXY. fuzz.yml doesn't use the mirror.

The builder image is Debian 11 (glibc 2.31, so the Linux binaries run on Debian 11+, Ubuntu 20.04+, RHEL 9+) with Rust, the aarch64 cross toolchain, LLVM 20 (libclang for bindgen, clang-cl/lld-link for Windows, Clang and lld for FreeBSD and macOS), NASM, CMake 3.31, cargo-xwin with the Microsoft CRT and SDK already downloaded, the FreeBSD target with a FreeBSD 14.4 sysroot (/opt/freebsd-sysroot), webjet-app's X11 libraries (libxkbcommon, libxkbcommon-x11, libxcb) for Linux amd64 and arm64 and in the FreeBSD sysroot, the macOS targets with a macOS SDK (/opt/macos-sdk/MacOSX.sdk), and Node.js for actions/checkout. Building it accepts Microsoft's license for the CRT and SDK, as cargo-xwin does on first use.

The image is built from its own repository, webj8-internal/webjet-builder (~/Projects/webjet-builder on the Mac), whose README covers building, pushing and the macOS SDK choice. It holds Apple's macOS SDK, which Apple's license allows using on Apple computers only and forbids redistributing, so the repository and its image live in the private organization webj8-internal, never under the public webj8.

Packages of a private organization can only be read by its members, so every job's container: logs in with the repository secrets BUILDER_USER (a member of webj8-internal) and BUILDER_TOKEN (that user's token with read:package). Pull requests from forks get no secrets, so their CI can't pull the image.

The runner pulls job images on every run (force_pull), so the image must be in the registry before a workflow names it. To change it: edit and push it from webjet-builder with a new tag (./build.sh --push), then bump the tag in ci.yml, release.yml and fuzz.yml here. scripts/freebsd-sysroot.sh exists in both repositories (the image runs its copy): keep them the same (FREEBSD_VERSION, BASE_SHA256 and the pinned packages in PKGS). Besides the base system it unpacks the FreeBSD packages webjet-app links against (libxkbcommon, libxcb and their dependencies) from the release's frozen package set (release_4 for 14.4), each checked against its SHA-256 from the repository catalogue (data.pkg); GNU tar needs zstd to unpack them.

RELEASE_TOKEN (repository secret, write:repository) is optional; without it, release.yml publishes with the workflow's automatic token.

Releases

  1. Set the version in the workspace Cargo.toml (version = "X.Y.Z"), update the version in the install commands in README.md, scripts/install.sh, scripts/install.ps1 and docs/webjet-operations-guide.md, commit, and tag the commit: git tag -a vX.Y.Z with release notes as the message (they become the release's description).
  2. Push the tag: git push origin vX.Y.Z. release.yml tests, builds Linux x86_64 and arm64, Windows, FreeBSD and macOS arm64 and x86_64, packages webjet-<tag>-<target>.tar.gz with webjetd, webjetctl and the desktop app webjet-app (Windows: .zip, also with wintun.dll and its license), the webjetup binaries, install.sh and install.ps1, each with a .sha256, and creates the Forgejo release.
  3. To rebuild one target and replace its files, from a checkout of the tag: 
    WEBJET_FORGEJO_TOKEN=… scripts/release.sh vX.Y.Z --publish \
  --targets "aarch64-apple-darwin x86_64-apple-darwin"
    --targets uploads into the existing release; files with the same name are replaced.
  4. Check the published release the way hosts install it: docker/release-test.sh vX.Y.Z runs install.sh and webjetup in fresh Debian 11, Ubuntu 20.04 and Debian 12 containers on amd64 and arm64, and RELEASE=vX.Y.Z docker/runbook-test.sh runs most of the operations guide (its "Testing this guide" section lists which sections) on systemd hosts installed from it.

scripts/release.sh without --targets still builds everything on a Mac (Linux in Docker with docker/release.Dockerfile, Windows with cargo-xwin, FreeBSD with scripts/freebsd-cargo.sh, macOS natively), and without --publish it only builds and packages into dist/<tag>.

Before tagging, docker/webjetup-test.sh tests webjetup and install.sh from the working tree end to end: it builds the Linux release files, serves them from a stand-in for the Forgejo release pages, starts a controller, and on a container that boots systemd runs curl … install.sh | sh with an enrollment string, checks the agent runs as webjet with its capabilities and pings the controller, then reinstall (an outdated unit is replaced, also by the upgrade helper), remove (identity kept), install again, remote upgrades (the release's webjetup sets the node up), a hand-over from an older webjetup, and nuke. install.ps1 has no automated test. docker/runbook-test.sh without RELEASE does the same for the operations guide.

Building Windows on a Mac needs cargo install cargo-xwin and Homebrew's llvm and lld. Without Homebrew's lld-link on PATH, cargo-xwin links rustup's rust-lld into its cache, which can't run on its own (it misses libLLVM.dylib), and CMake-built dependencies (BoringSSL, aws-lc) fail their compiler check. If that happened once, delete ~/Library/Caches/cargo-xwin/lld-link and the boring-sys-*/aws-lc-sys-* directories under target/x86_64-pc-windows-msvc/release/build, since CMake keeps the linker path in its cache. Wintun's version and checksum are pinned in scripts/release.sh; its license allows shipping wintun.dll with software that uses it through its API, with the license file alongside.

The Linux build uses Debian 11 so the binaries need only glibc 2.31. Debian 11 is archived, which docker/release.Dockerfile works around: CMake comes from Kitware (BoringSSL needs 3.22+), and on amd64 empty stand-in packages replace the 32-bit multilib libraries libclang depends on, because the archive no longer serves the matching glibc security update.

Platform code

webjetd/src/platform.rs does routes, split DNS and gateway NAT per OS by running the system tools (ip, route, resolvectl, netsh, PowerShell for NRPT, iptables). ipc.rs abstracts the control channel (Unix socket or named pipe), winsec.rs builds Windows ACLs, keystore.rs the key storage backends, service.rs the Windows service and signal handling.

Unix paths must also work on macOS, whose root volume is sealed read-only: there is no /run and nothing can create it. That's why the default sockets are under /var/run (ipc::default_control/default_signer, on Linux a symlink to /run). webjetctl and webjet-app take their default from ipc::default_control rather than spelling out a path. webjetd logs to stdout, so the launchd plist that webjetup writes sends both stdout and stderr to /var/log/webjetd.log.