Load Tests for Avni Sync

./gradlew gatlingRun

Just want the results? docs/load-test-summary.md gives the assumptions, how long a sync takes on the phone, and how many syncs an hour each server size handles, in plain language.

New to this? docs/what-a-simulated-sync-does.md walks one virtual user from feeder row to telemetry POST — the request sequence, what is modelled rather than performed, and the things people most often get wrong about it. The sections below document each knob; that one is the shape they fit into.

Lost track of the cases? docs/case-index.md maps every case to the runs made against it, what each varied, and the findings that came out of them.

Configurations

Users

src/gatling/resources/sync-users.csv holds the users the simulation drives. It is not tracked — copy the example and edit:

cp src/gatling/resources/sync-users-example.csv src/gatling/resources/sync-users.csv

The csv expects the following columns:

• userName
• lastModifiedDateTime
• password — only under AUTH_MODE=cognito
• token — only under AUTH_MODE=cognito; skips minting if supplied
• pushScale — optional, default 1. Multiplies this user's push volume, so one file can carry field workers creating twenty encounters a day alongside supervisors creating almost none. Default 1 is not neutral: the base distribution is a field worker's twenty a day, so leaving a supervisor at 1 makes them push like a field worker. Case 3 runs supervisors alone and needs this set
• deviceId — optional, defaults to -DDEVICE_ID (itself defaulting to avni-perf-simulation) for every user. Supply it per user to model one device each: the server keys identifier assignment on (user, device), and sync_telemetry records it, so a single shared id makes the rows from a run indistinguishable by device

Unknown columns are ignored, so role and organisationUUID can be carried for legibility.

The data generator writes this file — deployment.feeder_csv() emits one sync-users.csv spanning every tenant, with these columns. A test pins it against sync-users-example.csv, so the two cannot drift apart again.

co-tenant-users.csv has the same shape and is only read under CO_TENANTS=on — see Co-tenants. Also untracked.

Authentication

AUTH_MODE selects how the simulation identifies users. Default is none.

Mode	How	Use it for
none (default)	Sends only the USER-NAME header. No credentials, no AWS access, no expiry. Requires the target server to run with AVNI_IDP_TYPE=none	Everything, and the only option for runs longer than an hour
cognito	Mints a token per user via AdminInitiateAuth and sends AUTH-TOKEN. Needs AWS developer credentials on the machine running the simulation, plus COGNITO_CLIENT_ID and COGNITO_USER_POOL_ID	Short runs against a Cognito environment — staging, prerelease — and measuring what the none path omits

./gradlew gatlingRun -DAUTH_MODE=cognito \
  -DCOGNITO_CLIENT_ID=... -DCOGNITO_USER_POOL_ID=...

Both identifiers are required under cognito and have no defaults — a default would silently point a run at whichever environment was hardcoded rather than the one under test.

cognito has no token refresh. Tokens expire after an hour by default, so runs longer than that will fail partway. Use none for soak testing.

A server running AVNI_IDP_TYPE=none must not be publicly reachable. Anyone who can reach it is authenticated as whatever username they send.

Running both modes against the same environment gives the per-request cost of token verification — see the sync simulation plan, section B.

Sync window

SYNC_MODE sets how far back each entity's loadedSince reaches, which is what decides whether the server takes the full-sync path or the incremental one. Default is csv.

Mode	Window	Use it for
full	1900 for every entity	A first sync, or a device after a reset. The heaviest case
incremental	now − INCREMENTAL_SINCE_HOURS (default 24) for every entity	A simple, uniform incremental run
csv	Whatever the user file carries	Replaying a specific state
realistic	Per entity, spread as production's own gaps are	The faithful case

realistic is the one that matches production. Q2 measured the gap between a user's syncs at a median of 16 minutes but a 75th percentile of 12.5 hours — two behaviours, not one with spread. Every entity gets its own draw from that distribution, so a single user's entities span minutes to days, which is what a real device looks like: reference data last pulled when the configuration changed, transactional data a few minutes ago.

Uniform timestamps are the thing to avoid. loadedSince feeds the per-row queries behind syncDetails, so one value across every entity produces uniform selectivity — a query plan production never runs. Draws are derived from the user and entity name rather than randomised, so a run reproduces.

The window only bites once the bootstrap has run. The simulation asks the server what each user tracks — one POST /v2/syncDetails with [] per user, cached — because the server matches on entity name and type uuid. Without it every typed entity falls through to the server's 1900 default and full-syncs whatever the mode says, which is most of the sync volume. The run warns on the console if it ever builds a body without one.

Client-side storage cost

The simulation pauses after each page to stand in for the time a device spends parsing and persisting it. STORAGE_MODEL=weighted (default) computes that as

MS_PER_PAGE + records x per-record cost

The per-record cost is measured for the entities that matter and modelled for the rest. The five observation-bearing entities take rates measured from production's own server logs (StorageProfiles, 11 days, 24,667 confirmed full pages): under the default STORAGE_PROFILE=customer, Individual 3.28, ProgramEnrolment 2.84, Encounter 5.17, ProgramEncounter 4.76, GroupSubject 5.45 ms/record. Everything else uses entity tier x BASE_MS_PER_RECORD, with three tiers assigned by the entity generator: 0.2x for flat lookup rows, 1.0x for configuration and light transactional, 3.0x for observation-bearing.

Both terms are needed — they dominate in different places. On an encounter page they are equal at about 22 records. Below that the fixed term is most of the cost; above it the per-record term takes over, which is where heavy syncs live. Drop the per-record term and a full encounter page is understated about 47x; drop the per-page term and a 3-record page is understated about 8x.

Defaults are 112 ms a page, 2.78 ms a record for the tiered fallback, and a cap of MAX_STORAGE_PAUSE_MS, 30,000 ms. A full 1000-record page therefore costs about 0.7 s on a light entity, 2.9 s on a medium one and 5.3 s on encounters.

MS_PER_PAGE is measured, and net of network. 174 ms observed between pages, less this environment's 62 ms median request — which the simulation already pays for real, so charging it again would count the round trip twice.

BASE_MS_PER_RECORD is a fallback, and its history is a warning. Q1's 9.19 was first used unmodified, which double-counted server and network and modelled 27.7 seconds for one heavy page. Then 0.60, derived from the cap rather than measured. 2.78 survives because measurement endorsed it for the entities the logs reach thinly — the 1.0 tier's 2.78 against a measured 2.27–3.13. It now applies to metadata and lookup entities, a small share of any sync.

All of it is SQLite-shaped. D7 measured a real device: 2.22–2.53 ms/record on SQLite, 10.18–11.37 on Realm. Durations from this model are about right for a SQLite fleet and understate a Realm one by roughly three times.

The cap is a guard, not the mechanism. Once it binds, page size stops changing the pause, and that relationship is the whole point of the model. The defaults are chosen so it never binds; the banner warns if it does, and names the record count where it starts.

STORAGE_MODEL=zero is not a neutral fallback. Removing the pause lets a virtual user fire its 81 requests back to back, bounded only by the server — at a 20 ms response that is 9x a real device's request rate, and 17x at 10 ms. It is a deliberate over-drive for saturation runs. Used by accident it manufactures contention that cannot occur.

The per-page term is what shapes load: a real device issues one request every 174 ms, and on a LAN the round trip vanishes. Without it the simulation runs about four times too fast per user.

Timeouts

Set explicitly in src/gatling/resources/gatling.conf, not left at Gatling's defaults. Most are the same number Gatling would have used — the point is that they are chosen, since an unchosen default is a hidden parameter and a Gatling upgrade can move one.

Setting	Value	Why
http.requestTimeout	60 s	The client's own limit. openchs-android wraps every fetch in a 60-second race and rejects with syncTimeoutError, across pull, push and media signing. A response slower than this is a failed sync in the field however fast the server eventually answers
http.pooledConnectionIdleTimeout	300 s	Matches okhttp's keep-alive. At Gatling's 60 s the injector would reconnect after every media pause — handshakes a real device never pays
socket.connectTimeout	10 s	A slower connect means the accept queue is full, which is a finding, not something to wait out
ssl.handshakeTimeout	10 s	Separate from connect, so a TLS stall is distinguishable behind a load balancer
http.dns.queryTimeout	5 s	DNS should never be inside the measurement

The file is committed, so the commit in run-metadata.json pins these for a run. Changing one changes what failed means and makes runs either side incomparable.

A failed page ends the sync rather than being retried. Both paged loops carry exitHereIfFailed(); without it a failure left the continue-flag set and the virtual user reissued the same page forever. make smoke_closed_port guards it.

Environment variables

Can be overridden using ./gradlew gatlingRun -DBASE_URL= etc.

BASE_URL default https://loadtest.avniproject.org

USER_COUNT defaults to number of rows in resources/sync-users.csv

RAMP_PERIOD defaults to number of rows in resources/sync-users.csv * 20 — PROFILE=ramp only

PROFILE ramp (default), steady, burst, stress or smoke — see below

SYNC_WINDOW_HOURS default 12; hours over which a day's syncs arrive

SYNCS_PER_HOUR defaults to USER_COUNT / SYNC_WINDOW_HOURS

DURATION_MINUTES default 120 — steady and stress

BURST_SECONDS default 60 — burst only. Renamed from BURST_MINUTES, which is no longer read: passing it is silently ignored

STRESS_TO_SYNCS_PER_HOUR defaults to ten times the starting rate — stress only

PAGE_SIZE defaults to 1000, matching the client

NOW defaults to current time at start of simulation

SYNC_MODE one of full, incremental, csv, realistic — see above. Default csv

INCREMENTAL_SINCE_HOURS defaults to 24, used by SYNC_MODE=incremental

STRUCTURAL_CHECK true asserts zero failures, for H5's gate on a generated dataset

MAX_FAILED_PERCENT error budget for a load run, default 0.05 — the customer's figure

MAX_P95_MS asserts the 95th percentile when set. Production's light-band figure is 80,000

STORAGE_MODEL weighted (default) or zero — see above

BASE_MS_PER_RECORD defaults to 2.78, the fallback for entities StorageProfiles does not cover — see above

MAX_STORAGE_PAUSE_MS caps one page's modelled client cost, default 30000 — see above

MS_PER_PAGE defaults to 112, measured and net of network — see above

PUSH true includes the upload path. Default false — see below

PUSH_PROFILE customer (default) or production — see below

CUSTOMER_PUSH_DRAW v2 (default), v1 or v3 — the customer's encounters per sync. v2 is 10/20/30/45, mean 20, the customer's twenty a day; v1 is 8/20/45/150, mean 24, and reproduces every push run before 21 Oct 2026; v3 is 5/10/15/23, mean 10, the ten a day plan rows 24–26 run at. Recorded as push.customerDraw in run-metadata.json

PUSH_ENCOUNTER_MODEL general (default) or program — which table the customer's encounters land on. The programme design is out of scope for this exercise, so general matches what the datasets actually contain

CO_TENANTS true adds production's other organisations as a second syncing population — see below

CO_TENANT_SYNCS_PER_HOUR default 792, Q4's busiest recorded hour

CO_TENANT_SECONDS how long to sustain that rate, defaults to RAMP_PERIOD

CO_TENANT_USERS their user file, default co-tenant-users.csv

CO_TENANT_MEDIA_PER_ENCOUNTER default 0.2306, production-wide rather than the customer's bundle — uploadUrl calls per pushed encounter, measured from 30,736 calls over 133,259 encounters. An earlier 0.0214 was 10.8x low

PUSH_INDIVIDUALS / PUSH_ENROLMENTS / PUSH_PROGRAM_ENCOUNTERS / PUSH_ENCOUNTERS override one entity's distribution, as probability:[min:]p50:p95:max:mean

PUSH_OBSERVATION_MULTIPLE harvested observation sets per pushed record, default 1

PUSH_MEDIA_PER_ENCOUNTER media files each encounter queues, default 1.42; 0 disables media

MEDIA_MODEL pause (default) charges the upload time; none charges nothing — see below

MEDIA_FILE_KB default 500, matching the client's 1280x960 quality-1 capture

MEDIA_UPLOAD_KBPS device upload bandwidth, default 125 (about 1 Mbps)

UPLOAD_URL_BATCH how the device asks for upload URLs. 0 (default) is today's app, one GET /media/uploadUrl/{fileName} per file. 1–50 is the batched app of avniproject/avni-client#2218: one POST /media/uploadUrls per group of up to that many files, each group then uploaded file by file with the same pause. Above 50 is refused, matching the server's cap. Needs a server with avniproject/avni-server#1100 (not on 18.1). Recorded as media.uploadUrlBatch in run-metadata.json; in the Gatling report the request is Media uploadUrls rather than Media uploadUrl, and the server log says getting N media upload urls once per group, so count files from that line rather than from uploadUrl calls

PUSH_SEED_SIZE rows harvested per entity when seeding a device, default 20

PUSH_ALLOW_UNSAFE_TARGET true lifts the protected-host guard

PAGING page (default) or slice. Which form to request where the server offers both.

page is what the client calls — it reads page.totalPages from the first response and enumerates the rest — so it is the fidelity baseline and the default.

slice requests the server's /v2 variants instead. Spring's Page runs a count(*) over the whole matching set to report totalPages; Slice fetches size + 1 rows and reports hasNext, with no count. Run the same case twice, changing only this, and the delta is what that count costs. 22 of the 75 pulled entities have a slice endpoint — the transactional ones, where it is expensive — and the rest fall back to the paged path, which the banner reports.

The request pattern is identical either way: the client fetches pages strictly sequentially (ChainedRequests.fire() reduces over .then), which is what the simulation already does.

HTTP_LOG_LEVEL detail for failing HTTP requests, default WARN. DEBUG logs every failing request with its full request/response block, TRACE every request. Off by default because a stress run fails by design, and left on it is injector CPU spent formatting strings about the latency being measured.

Checks

make smoke_closed_port runs the smoke profile against a port with nothing behind it and requires that it fails fast: the run must terminate, issue only a handful of requests, and fail its assertions. CI runs it on every PR touching src/gatling/**.

This guards a real defect (D8.5). Both paged loops once cleared their continue-flag only from a check on a 200, so any failure left it set and the virtual user reissued the same page forever — 23 million log lines and 1.4 GB in two minutes, with no error and no termination. It matters most where the plan needs it most: PROFILE=stress ramps until the server stops answering, which is exactly when every user would otherwise start hammering it flat out.

make check_entities_current regenerates the entity table from the pinned openchs-models and fails if the committed copy has drifted.

make check_environment URL=… [SYNC_USER=…] [DB=…] [WAF=1] asks whether an environment is ready for a run — DNS, reachability, round-trip time, auth, syncDetails, whether the user resolves privileges at all, entity coverage, and optionally the database and the WAF's rate rule. It exits non-zero if not, and names the failures already seen so nobody diagnoses them twice.

SYNC_USER is optional — and is named that rather than USER because every login shell sets USER, which Make would inherit and pass silently. The users the scenarios run as come from the dataset generator, so requiring one would mean the check could only run after the thing it de-risks. make bootstrap_user ORG=… USERNAME=… emits idempotent SQL for one syncable user with no dataset behind it — one location and its type, a catchment, a group carrying privileges, and the membership row without which a user silently syncs none of the data.

make unit_test runs the simulation's unit tests — the push distribution's validation and maths, and page-metadata parsing. Only what can be tested without an injector; anything needing a running scenario is covered by the closed-port check above.

make test_data_generator runs the generator's suite. Among other things it pins the generated sync-users.csv to the columns the simulation actually reads, checked against sync-users-example.csv — the two had drifted apart once, silently costing per-user push volume.

Injection profiles

PROFILE sets the arrival shape. Arrival rate is derived, not configured — one sync per user per day over SYNC_WINDOW_HOURS — so 500 workers over twelve hours is 42 an hour.

Profile	Shape	For
ramp (default)	every user in the file syncs once, then exits	the H5 structural check
steady	constant arrival for DURATION_MINUTES	most test cases
burst	USER_COUNT devices arriving over BURST_SECONDS	a training cohort
stress	rate climbing to STRESS_TO_SYNCS_PER_HOUR	finding the knee
smoke	one sync	CI

The default is the wrong shape for a load run, deliberately. ramp gives each virtual user one sync and then exits, so it cannot express "42 syncs an hour for four hours" — which is how every test case is specified. It is kept as the default because it is what runs today and what the structural check needs: H5 has to touch every user in the file, and a rate-based profile syncs a sample, so an unreadable row belonging to a user it never reached would pass silently. The run warns if STRUCTURAL_CHECK is set with any other profile.

Injection is open in every profile: a sync is a short visit, not a session, so the server sees a rate. A closed model would hold in-flight syncs constant, which is the one thing that has to be free to move when the server slows down.

SYNC_WINDOW_HOURS=1 compresses a day's syncs into one hour — the single property separating the clustered test cases from the spread ones.

The push path

PUSH=on adds the upload half of a sync: the records a device queued since it last synced, posted before it asks the server what changed — the order dataServerSync uses.

There is no bulk endpoint. The client posts one record per request and waits for each, so a device with twenty queued encounters makes twenty sequential round trips, each through the full filter chain and its own transaction. That is the load: request count, not payload size.

Push writes, and that is why it is off by default. A run with PUSH=on changes the database it just measured. The next run is not the same experiment, and the dataset's H5 verdict no longer describes what is in the tables. Budget a restore between runs.

The cost of the default running the other way: a green run without PUSH=on is not evidence the write path holds, because write contention, lock waits and index maintenance cannot appear in it at all. The startup banner says which of the two you are getting.

A short deny list refuses the protected hosts outright. Pushed subjects and encounters cannot be told apart from field data afterwards.

References are harvested, not configured. Four page-0 reads per user collect the subject type, address, enrolments and real observation sets that pushed records point at — so the push path works against a generated dataset, a restored dump or a hand-built org without a shared list of UUIDs. Observations are reused verbatim from real rows, because insert cost here is dominated by GIN maintenance over the observations jsonb.

A device with nothing to reference pushes nothing and fails nothing, so the run prints how many devices were only partially seeded and what they lacked. Read a run with a high count as a floor.

Each sync draws twice per entity: whether it pushes that entity at all, and if so how many. PUSH_PROFILE picks which workload those draws describe.

customer (the default) — the deployment this exercise exists to size, from the customer's stated 20 encounters per field worker per day.

Entity	Pushed in	min	p50	p95	max	Per sync
ProgramEncounter	every sync	8	20	45	150	24.0
Individual	59.6%	1	1	13	174	1.98
ProgramEnrolment	33.2%	1	1	15	349	1.33

The level is the customer's; the spread is a modelling choice — their one number read as the median, p95 at 45, floor at 8. -DPUSH_PROGRAM_ENCOUNTERS=1.0:20:20:20:20 restores a flat twenty if the literal reading is preferred. No customer figure exists for registrations or enrolments, so those borrow production's shape.

PUSH_ENCOUNTER_MODEL decides which table the twenty a day land on. The programme design is out of scope, so the default is general. The bundle the customer exports today has no live programs — every mapping is a general Encounter on one Patient subject type — and the designed NCD programme is still being built. Separate runs later if it lands.

general (default) follows the current export and matches what the generated datasets contain: the volume moves to Encounter and there are no enrolments. program follows the design and would push a volume these datasets cannot receive. Same volume either way; what changes is whether the write path touches program_encounter behind a program_enrolment parent or encounter hanging straight off the subject — different sync strategies, indexes and join depth.

production — Q17's measurement of the platform as it stands, over 105,718 syncs.

Entity	Pushed in	p50	p95	max	Per sync
ProgramEncounter	28.7% of syncs	3	33	323	2.38
Individual	59.6%	1	13	174	1.98
ProgramEnrolment	33.2%	1	15	349	1.33
Encounter	21.1%	1	13	479	0.83

35% of real syncs push nothing at all, which is what the probabilities encode. A sync that pushes anything averages 14.3 records; over all syncs it is 9.3 — against the customer profile's 27.3.

Reference, and the co-tenant setting. Use production for the co-tenant traffic in cases 7 and 13, where the platform's existing organisations load the server alongside the customer's. Never for the customer's own cases: it understates them roughly tenfold.

Override either with probability:p50:p95:max:mean or probability:min:p50:p95:max:mean, e.g. -DPUSH_PROGRAM_ENCOUNTERS=1.0:8:20:45:150:24. A probability of 0 disables the entity.

Per-user variation comes from an optional pushScale column in the user file: a supervisor pulls a wide catchment but creates few records, so case 3 wants a value well under 1.

ChecklistItem and AttendanceRecord are deliberately not modelled. They are 20% of what production pushes and by far the burstiest — maxima of 4,790 and 750 records in one sync, minutes of uninterrupted POSTing from a single device — but the organisations in scope do not use them. A future extension if this is ever pointed at one that does.

Media

Every queued media file costs one GET /media/uploadUrl against the server and one PUT straight to S3. MediaQueueService sends them one at a time, and drains the whole queue before the first record is pushed — PARALLEL_UPLOAD_COUNT is 1, despite the chunking around it.

Set the rate from the deployment's bundle, not from the default. survey.py BUNDLE --media-repeats N reports files per filled form, per form type. A platform-wide average is the wrong base rate for any one implementation: production sits at 0.23 files per pushed encounter, while the customer's screening bundle reaches 16 on a single encounter.

Two structural details drive that, and a flat count of media form elements misses both. Images sit inside repeatable question groups — one is named "Take photos of all lesions and 1 photo without lesion" — so a single form element produces one file per repeat. And elements carrying editable: false are the app's own display copies of images another element already uploaded; counting them doubles every image.

The bytes are not transferred, but the time is charged. S3 serves the objects directly, so a PUT from the injector would measure the injector's own network — an in-region pipe that moves 500 KB in tens of milliseconds where a field device on rural 3G takes four seconds. Neither the server's load nor the device's timing. So MEDIA_MODEL=pause spends the elapsed time and skips the request, the same trade STORAGE_MODEL=weighted makes for parse-and-persist.

MEDIA_UPLOAD_KBPS	~	31 files	Sync duration
40	0.3 Mbps	388 s	402 s
125 (default)	1 Mbps	125 s	139 s
375	3 Mbps	42 s	56 s
1250	10 Mbps	12 s	26 s

That is the conservative row of the encounter mix — a uniform one. Where the image-heavy encounter type is the common one, multiply by up to eleven.

This inflates sync duration without adding server load. During the transfer the device asks avni-server for nothing, so syncs in progress rises several-fold while server requests in flight does not move. Sweep MEDIA_UPLOAD_KBPS rather than trusting one value — it is the largest single lever on how long a media-bearing sync takes, and it says nothing about the server.

MEDIA_MODEL=none restores the old behaviour of charging nothing. The startup banner says which you have, because a run that skips the time starts its data push sooner than any real device could.

Co-tenants

CO_TENANTS=on runs a second population alongside the customer's: production's other organisations, syncing rather than merely present. That distinction is the whole difference between test cases 6 and 7, and it separates a structural cost from a contention one.

• Case 6 — the co-tenant dataset loaded, CO_TENANTS off. What their presence costs: RLS selectivity, planner statistics, table and index size.
• Case 7 — the same dataset, CO_TENANTS=on. What their activity costs on top: connection pool, CPU, IO.

The difference between those two runs is the pair of numbers the hosting decision needs.

They arrive at a fixed rate rather than as a user count, because co-tenant load exists to occupy the pool and the CPU — how many distinct accounts produce it does not change what the server feels. They also push on the production profile and production's media rate, not the customer's: a co-tenant pushes 6.5 records across a third of its syncs and almost never photographs anything.

Every request is named with its population's prefix, so Customer · Individual and Co-tenant · Individual are separate rows in the report. Case 7 asks what the customer's sync costs while the platform is busy; pooling both into one distribution answers a different question, and the answer would move with the mix rather than with the server.

forAll() asserts the error budget per request name, so the two are independent there. MAX_P95_MS is not — it uses global() and pools them. With co-tenants running, read the customer's p95 off the Customer · rows. The run warns about this at startup.

Their user file is separate and untracked, the same as sync-users.csv:

cp src/gatling/resources/co-tenant-users-example.csv src/gatling/resources/co-tenant-users.csv

Run archiving

Every gatlingRun writes run-metadata.json next to the report's simulation.log. It has two halves: what produced the run — simulation commit and whether the tree was dirty, target, server build, dataset, entity table source — and what it ran with, which the simulation writes itself.

That second half is read from the simulation rather than restated in the build file, because the first version restated it and drifted: it recorded a property that had been removed and none of push, media, co-tenants or injection profiles. Values are the ones that actually applied, so a default nobody passed is recorded as its number rather than as a blank.

It also records the injector — label, OS, JVM, CPU count, heap and round-trip time to the target. Pass -DINJECTOR=<label>; it falls back to the hostname.

RTT is measured before the run as a handful of bare TCP connects, so no server behaviour contaminates it. Both the minimum and the median are kept — the minimum is the network, the median is what the run actually pays — alongside estimatedSyncOverheadSeconds, which multiplies the median by a sync's ~109 requests to make the number legible. RTT_SAMPLES and RTT_TIMEOUT_MS tune it; an unreachable target records why rather than a zero, and never fails a run.

Runs from different injector positions are not comparable. A sync is about 109 requests, so 25 ms of extra round trip adds 2.7 s to a 14.1-second median — a constant offset on every sync, not noise that averages out. The recorded RTT is what makes that checkable rather than inferred from a label: two runs whose estimatedSyncOverheadSeconds differ by seconds are not measuring the same thing. CPU and heap are recorded because they are what tells an injector that saturated from a server that did.

-DDATASET_ID and -DSERVER_BUILD are not discoverable — the server exposes only /ping — so they are written as unrecorded unless passed, and the task says so.

Two of those cannot be discovered and must be passed, or they are recorded as unrecorded:

./gradlew gatlingRun -DDATASET_ID=... -DSERVER_BUILD=...

A simulation.log with no provenance cannot be compared with anything, which is the whole point of keeping it.

Server-side instrumentation

The Gatling side sees a duration and a status code. It cannot tell a request that waited on the database from one that spent its time in the JVM — that comes from the New Relic agent on the server, and it is where the database-per-call measure behind the co-tenant and soak findings comes from.

Most of it expires in eight days, and the environment that produced a run is destroyed after it, so an expired hour is not a measurement that can be retaken. make newrelic_export copies it out before that happens; make newrelic_query RUN=case3 reads it back with no account and no key.

See docs/newrelic.md for what is attached, what each retention tier holds, the measures worth taking and the two ways to compare the wrong things.