Repository navigation
feat: tracel-runtime and the tracel-job protocol crate - #202
Merged
Merged
Conversation
This was referenced Oct 14, 2026
Closed
jwric
marked this pull request as ready for review
October 14, 2026 20:09
jwric
force-pushed
the
feat/tracel-runtime
branch
from
October 14, 2026 20:58
fdfd0a3 to
cc2287a
Compare
- Building a `Console` or `ProjectHandle` does no network I/O. The first request that needs the server verifies the credential and warns when it ends soon; a failed check is retried on the next request. - `Console::connect(env, credentials)` takes everything explicitly. `Console::from_env()` and `ProjectHandle::from_env()` read TRACEL_ENV, TRACEL_API_KEY or the stored login, TRACEL_NAMESPACE, TRACEL_PROJECT and tracel.toml, replacing tracel-core's own copy of that discovery. - An unrecognized TRACEL_ENV is an error naming the accepted forms instead of silently meaning Production; lowercase `development` and `staging-n` are accepted, as in the tracel CLI. - `ProjectHandle::console()` reaches the console a project came from.
- `tracel::Target` picks the backend from TRACEL_CONNECTION (`offline`, the default, or `console`; `station` behind its feature) without network I/O, and hands out experiments, the model and dataset registries, and inference. Offline targets report the registries as unsupported instead of returning `None`. - tracel-core is removed. The offline recorder moves to tracel-experiment as `LocalExperiments` behind a `local` feature, the no-op inference provider to tracel-inference as `NoopInferenceProvider`, and console discovery is public in tracel-console as `CredentialSource`, `ProjectRef` and `env_from_environment`. - `ExperimentModule`, `Models` and `Datasets` are renamed `Experiments`, `ModelRegistry` and `DatasetRegistry`. - Examples use `Target::from_env()` directly; the shared `common` crate is gone. Stale README sections and doc examples are rewritten against the current API and compiled as doc tests.
- `JobRegistry`, `JobDefinition` and the `Job`/`IntoJob` traits in tracel-app replace the separate registries and job adapters of `Cli`, `Server` and `StationRunner`. One `Mapper<I>` over JSON input replaces the three mapper families; `JsonMapper` merge-patches input onto its default and can report a JSON schema (`schema` feature). - With `TRACEL_DESCRIBE=<path>`, every runner writes its job definitions (name, kind, description, input schema and example) atomically to that path and returns without serving. - While describing, `ExperimentJob::run` returns `ExperimentErrorKind::Describing` instead of creating an experiment, so a binary that runs directly never trains. - `StationRunner::register` accepts experiment jobs only, checked at compile time.
- With `TRACEL_REPORT_FILE=<path>`, an experiment run writes a report (job, experiment number and Console URL or offline directory, status, times, error) atomically when it starts and when it ends. - The resolved job input is recorded as the experiment's arguments without a call from the job; `TRACEL_JOB_NUM` becomes the `tracel.job_num` attribute. - Offline runs write `events.jsonl` with the Console API's field names and `status.json` in the report format, replacing the Debug text and `status.txt`. - `Cli::run` returns an exit code: 0 completed, 1 failed, 2 unknown job or bad input, 130 cancelled. SIGTERM or Ctrl-C cancels the run's token; launchers send SIGKILL after a 30-second grace period. - `TRACEL_CONNECTION` is renamed `TRACEL_TARGET`, and `TargetError::UnknownConnection` becomes `UnknownTarget`.
- The Console and Station artifact lookups took the first result of the server's name filter, which matches substrings, so asking for `model` could return `model-2.bpk`. They now select the artifact whose name is exactly the one asked for, and report an ambiguous name instead of picking one. - Downloading lists the artifacts once instead of twice.
- `Cli` builds a typed command line from its job definitions at run
time: one subcommand per job, a flag per input field typed from the
JSON schema (with doc-comment help, allowed values and required
fields) or from the example input, dotted kebab-case names for nested
fields, and JSON literals for arrays and objects.
- Precedence is the example input, then `--config <FILE>`, then the
positional JSON, then flags, so `<bin> <job> <input-json>` keeps
working unchanged.
- `--completions <SHELL>` prints a completion script, and
`Cli::version(..)` adds `--version`. Usage errors exit 2.
- `tracel_app::cli::{command, job_command, job_input}` build the same
command from a definitions file.
- The `owner` and `name` aliases are gone. Missing-project errors name the expected keys `namespace` and `project`. - The mnist example's tracel.toml uses those keys.
- `rust-version` is 1.97.0, the lowest toolchain that builds and tests the workspace; the dependency graph needs at least 1.95, which 1.87 could not build. - Nested `if` blocks that clippy now collapses into let-chains are rewritten.
- The new `tracel-job` crate, exposed as `tracel::job`, holds what describes a job from outside the program that runs it: the definitions file written under TRACEL_DESCRIBE, the run report written to TRACEL_REPORT_FILE, and the generated command line. It depends only on clap, serde and serde_json. - tracel-app, tracel-experiment and tracel-runner use it instead of their own copies, with one atomic JSON writer and one protocol constant. File formats and command lines are unchanged.
- `ExperimentCompletion` and the run report's `RunStatus` lose `Cancelled`. A run's outcome is what its function returns: an error or a panic fails it with its reason, anything else completes it, whether or not its cancel token was cancelled. Being stopped is known to whoever stopped the run. - `ExperimentRunControl::is_run_cancelled` is gone; the cancel token, its signal handling, and `ExperimentErrorKind::Cancelled` for a job cancelled before it starts are unchanged. - A program asked to stop still exits with code 130. - The job binary says the job was asked to stop, not that it was cancelled: `CliError::Stopped`, and the signal message reads "Asking the job to stop".
- When a run's cancel token is cancelled while the run is active, the run logs one warning, "Stop requested; the run ends when the job returns", to the experiment's log, offline and on the console. It is an ordinary log entry, logged once per run on the thread that cancels the token, and not once the run has finished.
- The run report describes the job: `status` is how the job is going and how it ended, by what the job returned, whether or not it was asked to stop, and `experiment` links the experiment the job recorded. It is `null` until the job records one, and for a job that records none, such as an inference. `RunReport::new` takes only the job and when it started. - `Cli` writes the report when `TRACEL_REPORT_FILE` names a path, for every kind of job: when the job starts, once its input decodes, again when the job records an experiment, and again when it ends. A rejected command line or input writes none; a report that cannot be written when the job starts fails the job, with exit code 1, before it runs. - A prepared job runs in a `JobContext`: its cancel token and an `ExperimentReporter`, to which the experiment adapter hands the experiment once it is created. `ExperimentJob::run_with` takes an `on_created` hook, called with the run before the job's function runs; tracel-experiment no longer reads `TRACEL_REPORT_FILE` or writes the report. - An offline run's `status.json` is the experiment's own record: its name, run number, status, start, end and error.
- When the local backend creates the directory runs are recorded under (`./runs` by default), it writes a `.gitignore` there that ignores everything in it. Git and code packaging, which reads ignore files, then leave the records out, so recording a run no longer changes the code version digest. A directory that already exists is left as it is.
- The job binary prints "warning: the job was asked to stop" when it exits 130 after a stop request: the job may have completed, so the stop is not an error of its own. The exit code is unchanged.
- A job definition is its `name`, `description`, `input_schema` and
`input_example`: `JobKind` and the definitions file's `kind` are gone.
Recording an experiment, or answering inputs with outputs, is what a
job does as it runs.
- Every job takes a `JobInput` in either form. A stream's items are
`Result`s, so a document that cannot be read reaches the job.
`JobInput::one` gives the input of a job that takes one: the
document, or the stream's one document, `null` when it has none, and
an error for a second. The experiment adapter takes its input that
way; the inference adapter reports an unreadable input as an error
and takes no more.
- The server runs every job the same way at `POST /{job_name}`. The
body is one JSON document per line (NDJSON), each handed over once
its line arrives, or one document over several lines, handed over
once the body ends; an empty body is a `null` input. The response is
Server-Sent Events: each output as an unnamed event, each error as an
`error` event, the experiment the job records as an `experiment`
event, and last a `done` event with `status`, `error` and
`experiment`. An input rejected before the job runs is answered with
400, and a client that disconnects asks the job to stop through its
cancel token. The body limit now applies to the streamed body.
- `JobRegistry::get` returns the job as an `Arc`, so a runner can run
it on another thread. The command line and the server share how a
panicking job fails.
…ence jobs - `tracel_app::server::Server`, the `server` features and the basics `serve` and `infer-client` examples are gone, with the workspace's axum and tokio dependencies. - A job takes one JSON document: `JobInput` is gone and `Job::prepare` takes a `Value`. Jobs write no outputs: `JobOutput` and `DiscardOutput` are gone, and `PreparedJob::run` takes only the job's context. The Station runner hands its jobs their dispatched input that way. - An `InferenceJob` is no longer a job: its adapter and its runner-only description are gone. `InferenceModule` and `Target::inference` are unchanged.
- `RunReport::experiment` becomes `experiments`, a list in the order the job records them, always written: `[]` until the job records one. - The command line adds each experiment the job hands its reporter and rewrites the report, as before for the one experiment.
- An `ExperimentRun` ends as completed only through `finish()`. One dropped before `finish()` or `fail()` ends as failed, with the panic's message while a panic unwinds it, and otherwise with "the run was dropped without finish() or fail()", as when an error returns early past it. A run that completed did so because the code said so.
- `Experiments::run(name, |run| ...)` runs code in a new experiment and ends it by what the code returns: completed on `Ok`, failed with the error's message on `Err` or with the panic's message on a panic. The run is the current experiment while the code runs, and a panic on any thread is logged to it. - `Experiments::start(name)` starts an experiment and returns its run, for code that ends it itself. - `Experiments::attribute` and `attributes` set the attributes of each experiment started from the client. The basics `train` example uses `Experiments::run`.
- The `tracel-runner` crate becomes `tracel_app::station`, behind the `station` feature of tracel-app, which the `station` feature of `tracel` enables. The `runner` feature and `tracel::runner` are gone; the runner's behaviour is unchanged.
- `tracel_app::Jobs` registers a program's jobs by name:
`Jobs::new().job("train", train)`. A job is a function
`Fn(I) -> Result<(), E> + Send + Sync` of an input decoded from JSON;
`Job` adds its description, its default input, which the input is
merged onto, and, with the `schema` feature, its input schema.
`cli::Cli`, the `Job` trait, `JobContext`, the mappers and the
experiment adapter are gone, and `tracel-app` no longer depends on
`tracel-experiment`.
- The core runs a job by name; adapters ask it to. The process adapter
is `Jobs::run()`: the command line, definitions file, run report,
signals and exit codes as before. The Station adapter is
`station::StationRunner::new(url).name(..).run(jobs)`; it no longer
writes the definitions file.
- While an adapter runs a job, the job is the process's:
`job_running()`, `stop_requested()`, `on_stop(action)`, `job_num()`
and `report_experiment()`. A Station job's number is its station id,
and the experiments it reports are logged.
- `tracel-experiment` knows no jobs: `ExperimentJob`,
`Experiments::create` and the `TRACEL_DESCRIBE` guard are gone, with
its `tracel-job` dependency. `Experiments::on_start` hooks get each
run started, and `Experiments::cancel_token` stops those runs and
refuses new ones.
- `Target::experiments` links the experiments to the job that runs, if
any: `tracel.job_num`, a stop when the job is asked to, and each run
reported.
- The basics `cli`, `runner` and mnist examples are plain jobs
binaries.
- Each execution of a job has its own `JobScope`: its number or id, the request that it stop, and where the experiments it starts are reported. The scope is current on the thread that runs the job's function; `job_running()`, `stop_requested()`, `on_stop()`, `job_num()` and `report_experiment()` read it, and `JobScope::current()` holds it for other threads. Nothing about the running job is process-wide any more. - The Station runner runs each dispatched job on a thread of its own, so jobs dispatched at once run at once, each in its own scope. - `Target::experiments` links the experiments to the scope of the job the current thread runs, so runs started from them on any thread belong to that job.
- The crate is `tracel-runtime`, in `crates/tracel-runtime`, and the `tracel` facade re-exports it as `tracel::runtime`. Code, docs, examples and the publish workflow use the new name.
- A job's scope gives the input the job runs with, as JSON: `JobScope::input()` and `tracel_runtime::job_input()`. It is the input its adapter gave merged onto the job's default, which the job's function receives decoded. - `Experiments::arguments(json)` gives each experiment started from the client arguments it records when it starts. `ExperimentRun::log_args` replaces them: the arguments logged last are the run's. - `Target::experiments` gives the job's input, unless it is `null`, as those arguments when it is built in a job. Outside a job nothing is recorded. - The basics `cli` and `runner` and the mnist jobs no longer log their input.
- The Station runner decodes `cancel` events, `{"job_id": ...}`, and
requests the stop of the job they name: its `stop_requested()` turns
`true` and what it registered with `on_stop` runs, which cancels the
experiments it started from a `Target`. A cancel for a job that does
not run here is logged and changes nothing.
- A job that returns `Ok` after it was asked to stop is reported to the
station as `cancelled`; one that returns an error is reported as
`failed` with it.
- `DefinitionsFile` drops `runner`, which every program wrote as `cli`: the file gives `protocol`, `sdk_version` and `jobs`, and `DefinitionsFile::new` takes only the jobs. The protocol stays 1.
- The crate is `tracel-job-proto`: the contract between the runtime's process adapter and the programs that launch job binaries. It keeps the protocol's shapes and names and does no I/O. - `DefinitionsFile` and `RunReport` lose `write` and `read`. `to_json` gives a file's exact text, pretty-printed JSON ending with a newline, and `from_json` reads it back, failing with `ParseError::Protocol` for another protocol version before looking at its other fields. - `job_input` reads the `--config` file through the function its caller gives and fails with a `ConfigError`; an unreadable or non-JSON config file still exits with code 2. - The process adapter writes the definitions file and the run report itself, atomically through `<path>.tmp`, with the tests of that write.
- Drop the tracel::job re-export; the facade uses tracel-job-proto internally - Move the launch environment, definitions file and run report docs into tracel-job-proto - User docs keep Jobs::run, the command line, tracel run, exit codes and configuration - Re-export ReportedExperiment from tracel-runtime, the type report_experiment takes
- tracel-job-proto drops RunReport, RunStatus, ReportedExperiment and TRACEL_REPORT_FILE - The runtime drops report_experiment and the job scope's experiment sink - The process adapter answers through its exit code and the definitions file only - The Station runner no longer logs reported experiments - An experiment started in a job prints where it records to stderr as it starts
- Jobs::prepare gives a PreparedJob, or a PrepareError naming an unknown job or an invalid input - PreparedJob::run runs it in its own job scope, as an Execution says: its number and its Stop - Outcome gives the job's status and error, and whether it was asked to stop - Stop is public, and jobs, prepared jobs and executions move between threads - The process adapter and the Station runner run their jobs through this API
- process and station are modules beside the core, not inside it - Jobs::get_version and Jobs::get_default_job give the process adapter what it reads of the builder - The process adapter is functions of the Jobs it runs; its tests stop jobs through a Stop
- Once the station asked a job to stop, its finish is cancelled whatever the job returned - The finish keeps the job's error when it failed
- Its messages are tracing events with the job and runner ids as fields - It installs no subscriber, so the program decides where they go - tracing is a dependency of the station feature only
- The new tracel-job-args crate builds a program's command line from each job's name, description, input schema and default input, and reads the job's input back from it - A job has typed flags only when its input schema describes an object; a job without one takes its input as JSON or a --config file - It reads the schema in the form schemars writes it: object properties, $ref resolved with the referencing crate, enum and const choices, nullable fields; any other field takes JSON - The process adapter builds its command line with it; tracel-job-proto no longer depends on clap - The docs and the basics and mnist examples show flags on jobs with a schema
- ExitCode names the binary's exit codes, and STOP_GRACE_PERIOD how long a launcher gives a job it asked to stop - The runtime core has its own JobDefinition: the process adapter maps it to the definitions file and its exit codes to the protocol's, and the Station adapter maps it to its registration - The tracel crate defines the tracel.job_num experiment attribute it records, and no longer depends on tracel-job-proto - The crate docs describe the protocol of jobs run as processes
- tracel-runtime: `Stop`, `Execution::with_stop`, `Outcome::stop_requested`,
`JobScope::{stop_requested, on_stop}` and the free functions
`stop_requested` and `on_stop` are gone. An `Execution` gives the job its
number only.
- The process adapter installs no signal handlers: SIGINT and SIGTERM end
the job binary by their default action. It exits with 0 when the job
completed, 1 when it failed and 2 when the job or its input is rejected;
130 and the "the job was asked to stop" warning are gone, with the
`ctrlc` dependency.
- The Station adapter logs a `cancel` event and lets the job it names run
to completion. Jobs finish as completed or failed only; the `cancelled`
finish status is gone.
- tracel-job-proto: `ExitCode::Stopped` and `STOP_GRACE_PERIOD` are gone.
- The `tracel` facade no longer cancels the experiments a job started when
the job is asked to stop. Experiments started in a job still record its
number and input and print where they record.
- tracel-experiment: `Experiments::cancel_token` is gone, and with it the
link to each run started and the refusal to start once it is cancelled;
`ExperimentErrorKind::Cancelled`, then unused, is gone too. The warning a
run logs when its cancel token is cancelled no longer mentions jobs.
- Runs keep their own cancellation: `ExperimentRun::cancel_token`,
`ExperimentRun::cancel`, the console's cancel requests and the training
interrupter are unchanged.
- The README, crate docs and the mnist example no longer describe stopping
a job, its grace period or exit code 130.
One crate for jobs run as processes. By default it is the protocol, which needs only serde; the `cli` feature adds the command line generated from a job's input schema. The command line takes the protocol's JobDefinition, so the separate Job type is gone.
The rebase onto main took this branch's README text where both changed it, which dropped main's links to the Tracel console and its "Tracel account" naming.
0.22.0 is the release with the job reads and the removal of the presigned URL transfers, which tracel-artifact replaces. Only the requirement and its Cargo.lock entry change.
Marc-AnthonyG
force-pushed
the
feat/tracel-runtime
branch
from
October 14, 2026 23:50
84d5b9a to
7882f08
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reworks the job layer from #200, which this is stacked on, into
tracel-runtimeand one protocol crate,tracel-job.tracel-runtime(formerlytracel-app)Fn(I) -> Result<(), E>, registered by name inJobs.Jobadds a description, a default input, which the input is merged onto, and, with theschemafeature, the input's JSON Schema.Experiments::runandExperiments::startstart experiments from the client.Jobs::prepareandPreparedJob::runare the one API every transport uses. Two adapters ship:mainreturnsJobs::run();stationfeature, which logs throughtracing.ExperimentRun::cancel_token.tracel-jobTRACEL_DESCRIBE;TRACEL_JOB_NUM;clifeature, off by default, generates a job's command line from its input schema: a typed flag per field, with nested fields dotted, plus--config, the JSON input and shell completions. It also reads the input back. The runtime parses its own command line with it, andtracel runparses the same flags before it launches the binary.tracel-ai/tracel-cli#41 builds against this branch.