feat: tracel run, tracel jobs and remote runs by jwric · Pull Request #41 · tracel-ai/tracel-cli

jwric · 2026-10-07T14:03:09Z

tracel run runs a workspace's jobs locally or on a compute provider, tracel jobs follows them, and experiments compare diffs two experiments. Stacked on #40.

tracel run

run <job> builds, in release, the workspace's binary that depends on tracel, reads its jobs' definitions, and runs <binary> <job> <input>. The definitions come from TRACEL_DESCRIBE and are cached in target/tracel/jobs.json until the binary changes.
run --list lists the jobs, run <job> --help shows a job's flags, and run -- <args> is cargo run --release -- <args>.
The input is built in this order, each step winning over the one before:
1. the job's example input, or the input of the experiment --like names;
2. each -c <file>, as a JSON merge patch;
3. the job's typed flags, such as --epochs 5 --optimizer.lr 0.01, generated from its input schema by the SDK's tracel-job;
4. each --set <path>=<value>.
An input the schema rejects fails with USAGE.
The run records on the Console when a project is linked and a credential is available, and offline otherwise or with --offline.
The exit code says how the job ended: 0 completed, 2 rejected (USAGE), and anything else or a signal is JOB_FAILED. Ctrl-C kills the job and every process it started.

Remote runs

run <job> --remote <group> packages the workspace (--mode binary|source) and queues it with the resolved input.
A job may incur costs, so it asks before uploading; --yes answers.
--follow follows the job's logs and exits as jobs wait does. --dry-run shows the job without uploading.

tracel jobs

jobs list|get|logs|cancel|wait. logs reads byte-offset pages, and --follow reads until the job finishes. wait fails with JOB_FAILED or TIMEOUT.

experiments compare

experiments compare <base> <other> diffs config and attributes by path, and compares metric summaries by group, with deltas.

Packaging

A code version is named by a digest of its source. Source and binary packages of the same source share one code version, and only missing files are uploaded.
Source packages are zips of what the ignore files leave, with or without git. init no longer needs a commit, and tracel.toml holds only namespace and project.

Dependencies, pinned by git revision for review, to become crates.io releases before merging:

feat: job reads, cancel and the queued job number tracel-client#55, the job API;
tracel-job with its cli feature, from feat: tracel-runtime and the tracel-job protocol crate tracel#202.

Remote runs need tracel-ai/tracel-app#1752, the job number on queue, and tracel-ai/tracel-app#1751, the compute provider on the runner protocol.

- `tracel-client` is pinned to the client's jobs branch, which adds job reads, cancellation and the job number a queued job gets.

- `tracel jobs list|get|logs|cancel|wait` for the selected project. - `logs --follow` streams NDJSON until the job finishes; `--start` reads from a byte offset. - `cancel` asks for confirmation; without a terminal it needs `--yes` and otherwise exits 7. Cancelling a finished job is a conflict. - `wait` exits 0 when the job completes, 9 (JOB_FAILED) when it fails or is cancelled, and 10 when `--timeout` runs out.

- `tracel run` runs the project through `cargo run`, as `train` did, with the same environment and exit code. - Arguments reach the program only after `--`, so `tracel run` stays free for a job name.

- The code version digest is a SHA-256 over the sorted `name:sha256` lines of what is packaged: archived files in source mode, uploaded binaries by target in binary mode. Uncommitted edits now produce a new code version instead of being skipped as already packaged. - `package` no longer needs a commit: `--commit`, `--allow-dirty`, the commit prompt and the uncommitted-changes warnings are gone. - Source archives list files in path order, and the packager reuses the workspace metadata and file hashing helpers instead of recomputing them.

- Code versions are keyed by their packaged content, so `init` drops the commit requirement along with `--commit`, `--allow-dirty`, the commit prompt and the git helpers that only served them. - gix no longer needs its `status` feature.

- Source packages are a real zip archive, which is what the compute provider opens; they were a gzipped tar uploaded as `source.zip`, so every remote source job failed at extraction. Entries are in path order with fixed timestamps and modes, files sit at the archive root, and the code version digest is unchanged. - A workspace in a subdirectory of a git repository, or containing a nested repository, now packages its files; repository paths were joined onto the workspace root, so nothing was found. - `zip` replaces the `tar` and `flate2` dependencies.

- `run --remote <GROUP>` packages the workspace like `tracel package`, confirms, uploads and queues a job with the arguments after `--`, quoted so the compute provider receives them unchanged. - Without a terminal it needs `--yes` (exit 7), checked before building. `--dry-run` builds the package and reports the plan without uploading or queuing. - `--follow` streams the job's logs (NDJSON in JSON mode, starting with a `queued` item) and exits like `tracel jobs wait`. - `package` and `run --remote` share one packaging module, and confirmation and project resolution helpers are factored out.

- `tracel experiments compare <BASE> <OTHER> [--metric NAME]...` lists the config and attribute differences by dotted path and, for each metric and group, both best values from the run summaries and their delta. The server does not report which direction is better, so `compare` does not judge. - Every experiments subcommand resolves the project before logging in and names a missing experiment with a hint to list them.

- `tracel run --list` builds the workspace binary that depends on `tracel`, asks it for its job definitions with TRACEL_DESCRIBE, and caches them in target/tracel/jobs.json by binary hash. - `tracel run <JOB> [-c FILE]... [--set PATH=VALUE]... [--like NUM]` resolves the input from the job's example (or a past experiment's recorded config) with JSON merge patch and dotted paths, validates it against the job's JSON schema, and runs the binary directly. - The run records on Console when a project is linked and a login is available, offline otherwise or with `--offline`. The CLI reads the program's run report to announce the experiment and decide the exit code; signals are forwarded and the program is killed after a 30-second grace period. - `tracel run <JOB> --remote <GROUP>` sends the job and its validated input to the compute provider. - Cargo builds for `package` and `run` share one helper that keeps compiler diagnostics visible.

- Packaging lists files with one `ignore` walk that honors .gitignore, .ignore and git's exclude files whether or not the workspace is a git repository; `.git` and `target` are always left out. Dotfiles are packaged in a repository unless ignored, and skipped outside one. - `tracel init` no longer runs `git init`, and the gix dependency is gone, which drops about a third of the dependency tree. - Packaging nothing is a usage error instead of an empty upload.

- The `owner` and `name` aliases are gone. Any other key in tracel.toml is a usage error that names the expected keys. - One strict reader replaces the two parsers that read the file.

- `rust-version` is 1.97.0, the lowest toolchain that builds and tests the workspace; the client dependency needs at least 1.89, which 1.87 could not build. - Nested `if` blocks that clippy now collapses into let-chains are rewritten.

The digest of a binary package hashed the uploaded binaries and their targets, so each target set and each mode made its own code version and Console never merged them. Both modes now use the digest of the files a source package holds: binaries for any target and a source package of the same code land in one code version, which Console extends. Binary mode still uploads only the binaries.

- The definitions file and the run report come from the shared `tracel-job` crate, which the SDK writes them with, instead of copies of their types. - `tracel run <JOB>` takes the flags the job's binary has, such as `--epochs 5 --optimizer.lr 0.01`, typed by the job's input schema or example input; `tracel run <JOB> --help` lists them beside its own. - The input resolves from the example (or `--like`), then `-c` files, then the job's flags, then `--set`. `tracel run` keeps its own flags, so a field whose flag has one of their names is set with `--set`. - A bad value or unknown job flag is a usage error naming the flag.

- The run report's status is `running`, `completed` or `failed`; a run asked to stop still ends as completed or failed. `tracel run` knows a job was stopped from its own facts: it asked the program to stop, or the program exited 130. - A stopped job fails with `JOB_FAILED` as before, and says what its experiment ended as when the report gives it, such as "Job 'mnist' was stopped (experiment 12 completed)." Other endings are unchanged.

- The run report describes the job, and the SDK writes it for every job, an inference too: `status` is how the job is going and how it ended, and `experiment` links the experiment the job recorded, `null` until it records one. `tracel run` announces and names an experiment only when the report has one, and points at Console logs only for a Console experiment. - `ExperimentOutcome` is `JobOutcome`. A stopped job says how it ended when the report says, with its experiment, if any, such as "Job 'mnist' was stopped (it completed; experiment 42)." or "Job 'mnist' was stopped (it failed): loss is NaN". Exit codes are unchanged.

- The job definitions have no `kind`: a job is something `tracel run` and the runners run, whatever it records or outputs. `tracel run --list` lists each job's name and description. - `tracel-job` is pinned to the SDK commit that removes the kind.

- `tracel run` builds against the SDK's new `tracel-job`: the protocol of jobs run as processes, and, with its `cli` feature, the command line generated from a job's input schema. It is pinned to the SDK's `feat/tracel-runtime` branch. - There is no run report: the program gets no `TRACEL_REPORT_FILE`, and `tracel run <job>` doesn't announce or name an experiment. The SDK prints where it records into the job's output. The JSON result of `run <job>` has no `report`. - The program's exit code says how the job ended: 0 completed, 2 a job or input the program rejected (`USAGE`), and any other code or a signal failed (`JOB_FAILED`), such as "Job 'mnist' failed (exit code 1)." A job `tracel` asked to stop fails as "Stopped job 'mnist'." - Only a job whose input has a JSON schema has flags, as the SDK's generated command line has; a flag for any other job is a usage error that says so. - The definitions file has no `runner`, and the hints name `tracel::runtime::Jobs`.

- The job's program has no stop protocol, so the first Ctrl-C, SIGTERM or SIGHUP `tracel run <job>` gets kills it at once, with SIGKILL to the process group it leads: the processes it started, such as data loaders, end with it. There is no SIGTERM and no 30-second grace period, and no "Stopping" or "Killed" lines. - A stopped job fails with `JOB_FAILED`, as "Stopped job 'mnist'.", on a line of its own after the `^C` the terminal echoed. - Listing a binary's jobs shares the same wait, and still kills the binary when it hasn't listed them within 10 seconds.

- `tracel run <job>` and `tracel run --list` build the job binary with `cargo build --release --package <package> --bin <binary>`, under a spinner, and show cargo's output only when the build fails. `tracel run -- <args>` is `cargo run --release -- <args>`. - `tracel package --mode binary --bin <name>` and `tracel run <job> --remote` build only that binary, with its package, so a binary of the same name in another package can't stand in for it, and the remote run uploads the build `tracel run` made for this machine. - One build helper in `tools::cargo` makes every release build, and finds a `--bin` among the workspace's binaries.

- `tracel run <job>` is a flow, "Run job 'train'", or "Run job 'train' on gpu" with `--remote`, and the program writes among its lines. `--remote` checks its mode and confirmation before building anything, with or without a job. - `run --remote` asks to queue the job on the group with its short code version, says "Queued job 14 with code version 67c31f69845e.", and ends on the commands that follow its logs and wait for it. With `--follow`, the logs come after that line. `tracel init` ends on `tracel run --list`. - `jobs list` and `jobs get` name the job a `run <job> --remote` queued instead of its quoted command, show times to the minute and the short code version, and `jobs get` shows when the job ran and for how long, and its input. A job's state reads as "Job 14 completed.", "failed.", "was cancelled." or "is stopping.". - `jobs logs` shows the page as it is, then the command that reads the next one. `jobs wait` says what the job is doing while it waits, and `jobs cancel` ends on "Cancelled job 14." or "Job 14 is stopping.". - `experiments compare` writes each part under its heading, with its empty line in the table, and sums up the differences. - A value that is a web address shows as a link. `tracel run --help` into a reader that closes early is not an error. - Jobs' command quoting, polling and log following live in one helpers module that `jobs` and `run --remote` share. Messages, hints and doc comments follow the CLI's style.

Tests follow the rest of the CLI: given/when/then names, and named rstest cases for inputs that share a behavior. - tracel run: which flags are the job's, how they set its input, the order the input's sources win in, schema checks, the choice of the job binary, the definitions cache, and how a run ends. - Jobs: the command a queued job runs and reading it back, reading logs on, waiting, and how long a job ran. - Packaging: the digest, the files packaged and the archive, each in a test of its own; binary packages share the source package's digest. - No tests of clap's argument conflicts or of the JSON shape of events.

- The README starts with listing and running a job, says the run is recorded on the Tracel Console once the workspace is linked and you're logged in, that Ctrl-C stops the job, and that `tracel run` without a job is `cargo run --release`. - The reference drops the run report and `runner`, says how the exit code ends a job and that a stop kills it and what it started, which release build runs and is uploaded, that only a job with an input schema has flags, and how `jobs logs` and `run --remote` end. - Packaging uploads only the files a code version doesn't have yet.

…tracel-job This branch still moves bytes with the client's presigned URL transfers, which tracel-client 0.22.0 removes, so it builds against the tracel-client commit that merged the job reads and still has them. tracel-job is not released yet, so it stays pinned to the head of the SDK pull request that adds it.

* chore: build against the client's job API - `tracel-client` is pinned to the client's jobs branch, which adds job reads, cancellation and the job number a queued job gets. * feat: add tracel jobs - `tracel jobs list|get|logs|cancel|wait` for the selected project. - `logs --follow` streams NDJSON until the job finishes; `--start` reads from a byte offset. - `cancel` asks for confirmation; without a terminal it needs `--yes` and otherwise exits 7. Cancelling a finished job is a conflict. - `wait` exits 0 when the job completes, 9 (JOB_FAILED) when it fails or is cancelled, and 10 when `--timeout` runs out. * feat: tracel run replaces tracel train - `tracel run` runs the project through `cargo run`, as `train` did, with the same environment and exit code. - Arguments reach the program only after `--`, so `tracel run` stays free for a job name. * fix: key code versions by their packaged content - The code version digest is a SHA-256 over the sorted `name:sha256` lines of what is packaged: archived files in source mode, uploaded binaries by target in binary mode. Uncommitted edits now produce a new code version instead of being skipped as already packaged. - `package` no longer needs a commit: `--commit`, `--allow-dirty`, the commit prompt and the uncommitted-changes warnings are gone. - Source archives list files in path order, and the packager reuses the workspace metadata and file hashing helpers instead of recomputing them. * fix: init no longer needs a git commit - Code versions are keyed by their packaged content, so `init` drops the commit requirement along with `--commit`, `--allow-dirty`, the commit prompt and the git helpers that only served them. - gix no longer needs its `status` feature. * fix: package source as a zip, from any workspace location - Source packages are a real zip archive, which is what the compute provider opens; they were a gzipped tar uploaded as `source.zip`, so every remote source job failed at extraction. Entries are in path order with fixed timestamps and modes, files sit at the archive root, and the code version digest is unchanged. - A workspace in a subdirectory of a git repository, or containing a nested repository, now packages its files; repository paths were joined onto the workspace root, so nothing was found. - `zip` replaces the `tar` and `flate2` dependencies. * feat: tracel run --remote queues the project as a job - `run --remote <GROUP>` packages the workspace like `tracel package`, confirms, uploads and queues a job with the arguments after `--`, quoted so the compute provider receives them unchanged. - Without a terminal it needs `--yes` (exit 7), checked before building. `--dry-run` builds the package and reports the plan without uploading or queuing. - `--follow` streams the job's logs (NDJSON in JSON mode, starting with a `queued` item) and exits like `tracel jobs wait`. - `package` and `run --remote` share one packaging module, and confirmation and project resolution helpers are factored out. * feat: compare two experiments - `tracel experiments compare <BASE> <OTHER> [--metric NAME]...` lists the config and attribute differences by dotted path and, for each metric and group, both best values from the run summaries and their delta. The server does not report which direction is better, so `compare` does not judge. - Every experiments subcommand resolves the project before logging in and names a missing experiment with a hint to list them. * feat: run jobs by name with tracel run - `tracel run --list` builds the workspace binary that depends on `tracel`, asks it for its job definitions with TRACEL_DESCRIBE, and caches them in target/tracel/jobs.json by binary hash. - `tracel run <JOB> [-c FILE]... [--set PATH=VALUE]... [--like NUM]` resolves the input from the job's example (or a past experiment's recorded config) with JSON merge patch and dotted paths, validates it against the job's JSON schema, and runs the binary directly. - The run records on Console when a project is linked and a login is available, offline otherwise or with `--offline`. The CLI reads the program's run report to announce the experiment and decide the exit code; signals are forwarded and the program is killed after a 30-second grace period. - `tracel run <JOB> --remote <GROUP>` sends the job and its validated input to the compute provider. - Cargo builds for `package` and `run` share one helper that keeps compiler diagnostics visible. * refactor: package with ignore rules alone, without git - Packaging lists files with one `ignore` walk that honors .gitignore, .ignore and git's exclude files whether or not the workspace is a git repository; `.git` and `target` are always left out. Dotfiles are packaged in a repository unless ignored, and skipped outside one. - `tracel init` no longer runs `git init`, and the gix dependency is gone, which drops about a third of the dependency tree. - Packaging nothing is a usage error instead of an empty upload. * fix: read only namespace and project from tracel.toml - The `owner` and `name` aliases are gone. Any other key in tracel.toml is a usage error that names the expected keys. - One strict reader replaces the two parsers that read the file. * chore: declare the Rust version the dependencies need - `rust-version` is 1.97.0, the lowest toolchain that builds and tests the workspace; the client dependency needs at least 1.89, which 1.87 could not build. - Nested `if` blocks that clippy now collapses into let-chains are rewritten. * fix: binary packages use the source digest The digest of a binary package hashed the uploaded binaries and their targets, so each target set and each mode made its own code version and Console never merged them. Both modes now use the digest of the files a source package holds: binaries for any target and a source package of the same code land in one code version, which Console extends. Binary mode still uploads only the binaries. * feat: tracel run takes the job's own flags - The definitions file and the run report come from the shared `tracel-job` crate, which the SDK writes them with, instead of copies of their types. - `tracel run <JOB>` takes the flags the job's binary has, such as `--epochs 5 --optimizer.lr 0.01`, typed by the job's input schema or example input; `tracel run <JOB> --help` lists them beside its own. - The input resolves from the example (or `--like`), then `-c` files, then the job's flags, then `--set`. `tracel run` keeps its own flags, so a field whose flag has one of their names is set with `--set`. - A bad value or unknown job flag is a usage error naming the flag. * refactor: the launcher decides when a job was stopped - The run report's status is `running`, `completed` or `failed`; a run asked to stop still ends as completed or failed. `tracel run` knows a job was stopped from its own facts: it asked the program to stop, or the program exited 130. - A stopped job fails with `JOB_FAILED` as before, and says what its experiment ended as when the report gives it, such as "Job 'mnist' was stopped (experiment 12 completed)." Other endings are unchanged. * refactor: the run report is the job's - The run report describes the job, and the SDK writes it for every job, an inference too: `status` is how the job is going and how it ended, and `experiment` links the experiment the job recorded, `null` until it records one. `tracel run` announces and names an experiment only when the report has one, and points at Console logs only for a Console experiment. - `ExperimentOutcome` is `JobOutcome`. A stopped job says how it ended when the report says, with its experiment, if any, such as "Job 'mnist' was stopped (it completed; experiment 42)." or "Job 'mnist' was stopped (it failed): loss is NaN". Exit codes are unchanged. * refactor: a job has no kind - The job definitions have no `kind`: a job is something `tracel run` and the runners run, whatever it records or outputs. `tracel run --list` lists each job's name and description. - `tracel-job` is pinned to the SDK commit that removes the kind. * refactor: a job ends on its exit code, with the SDK's tracel-job - `tracel run` builds against the SDK's new `tracel-job`: the protocol of jobs run as processes, and, with its `cli` feature, the command line generated from a job's input schema. It is pinned to the SDK's `feat/tracel-runtime` branch. - There is no run report: the program gets no `TRACEL_REPORT_FILE`, and `tracel run <job>` doesn't announce or name an experiment. The SDK prints where it records into the job's output. The JSON result of `run <job>` has no `report`. - The program's exit code says how the job ended: 0 completed, 2 a job or input the program rejected (`USAGE`), and any other code or a signal failed (`JOB_FAILED`), such as "Job 'mnist' failed (exit code 1)." A job `tracel` asked to stop fails as "Stopped job 'mnist'." - Only a job whose input has a JSON schema has flags, as the SDK's generated command line has; a flag for any other job is a usage error that says so. - The definitions file has no `runner`, and the hints name `tracel::runtime::Jobs`. * feat: a stop kills the job and what it started - The job's program has no stop protocol, so the first Ctrl-C, SIGTERM or SIGHUP `tracel run <job>` gets kills it at once, with SIGKILL to the process group it leads: the processes it started, such as data loaders, end with it. There is no SIGTERM and no 30-second grace period, and no "Stopping" or "Killed" lines. - A stopped job fails with `JOB_FAILED`, as "Stopped job 'mnist'.", on a line of its own after the `^C` the terminal echoed. - Listing a binary's jobs shares the same wait, and still kills the binary when it hasn't listed them within 10 seconds. * feat: tracel always builds in release - `tracel run <job>` and `tracel run --list` build the job binary with `cargo build --release --package <package> --bin <binary>`, under a spinner, and show cargo's output only when the build fails. `tracel run -- <args>` is `cargo run --release -- <args>`. - `tracel package --mode binary --bin <name>` and `tracel run <job> --remote` build only that binary, with its package, so a binary of the same name in another package can't stand in for it, and the remote run uploads the build `tracel run` made for this machine. - One build helper in `tools::cargo` makes every release build, and finds a `--bin` among the workspace's binaries. * feat: run and jobs follow the CLI's message style and layout - `tracel run <job>` is a flow, "Run job 'train'", or "Run job 'train' on gpu" with `--remote`, and the program writes among its lines. `--remote` checks its mode and confirmation before building anything, with or without a job. - `run --remote` asks to queue the job on the group with its short code version, says "Queued job 14 with code version 67c31f69845e.", and ends on the commands that follow its logs and wait for it. With `--follow`, the logs come after that line. `tracel init` ends on `tracel run --list`. - `jobs list` and `jobs get` name the job a `run <job> --remote` queued instead of its quoted command, show times to the minute and the short code version, and `jobs get` shows when the job ran and for how long, and its input. A job's state reads as "Job 14 completed.", "failed.", "was cancelled." or "is stopping.". - `jobs logs` shows the page as it is, then the command that reads the next one. `jobs wait` says what the job is doing while it waits, and `jobs cancel` ends on "Cancelled job 14." or "Job 14 is stopping.". - `experiments compare` writes each part under its heading, with its empty line in the table, and sums up the differences. - A value that is a web address shows as a link. `tracel run --help` into a reader that closes early is not an error. - Jobs' command quoting, polling and log following live in one helpers module that `jobs` and `run --remote` share. Messages, hints and doc comments follow the CLI's style. * test: one readable test per behavior for runs and jobs Tests follow the rest of the CLI: given/when/then names, and named rstest cases for inputs that share a behavior. - tracel run: which flags are the job's, how they set its input, the order the input's sources win in, schema checks, the choice of the job binary, the definitions cache, and how a run ends. - Jobs: the command a queued job runs and reading it back, reading logs on, waiting, and how long a job ran. - Packaging: the digest, the files packaged and the archive, each in a test of its own; binary packages share the source package's digest. - No tests of clap's argument conflicts or of the JSON shape of events. * docs: the README and the reference describe runs as they are now - The README starts with listing and running a job, says the run is recorded on the Tracel Console once the workspace is linked and you're logged in, that Ctrl-C stops the job, and that `tracel run` without a job is `cargo run --release`. - The reference drops the run report and `runner`, says how the exit code ends a job and that a stop kills it and what it started, which release build runs and is uploaded, that only a job with an input schema has flags, and how `jobs logs` and `run --remote` end. - Packaging uploads only the files a code version doesn't have yet. * chore: build against tracel-client after its job reads and the SDK's tracel-job This branch still moves bytes with the client's presigned URL transfers, which tracel-client 0.22.0 removes, so it builds against the tracel-client commit that merged the job reads and still has them. tracel-job is not released yet, so it stays pinned to the head of the SDK pull request that adds it. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Ex4o1LntLPL1RrM35YpRjE

* chore: build against the client's job API - `tracel-client` is pinned to the client's jobs branch, which adds job reads, cancellation and the job number a queued job gets. * feat: add tracel jobs - `tracel jobs list|get|logs|cancel|wait` for the selected project. - `logs --follow` streams NDJSON until the job finishes; `--start` reads from a byte offset. - `cancel` asks for confirmation; without a terminal it needs `--yes` and otherwise exits 7. Cancelling a finished job is a conflict. - `wait` exits 0 when the job completes, 9 (JOB_FAILED) when it fails or is cancelled, and 10 when `--timeout` runs out. * feat: tracel run replaces tracel train - `tracel run` runs the project through `cargo run`, as `train` did, with the same environment and exit code. - Arguments reach the program only after `--`, so `tracel run` stays free for a job name. * fix: key code versions by their packaged content - The code version digest is a SHA-256 over the sorted `name:sha256` lines of what is packaged: archived files in source mode, uploaded binaries by target in binary mode. Uncommitted edits now produce a new code version instead of being skipped as already packaged. - `package` no longer needs a commit: `--commit`, `--allow-dirty`, the commit prompt and the uncommitted-changes warnings are gone. - Source archives list files in path order, and the packager reuses the workspace metadata and file hashing helpers instead of recomputing them. * fix: init no longer needs a git commit - Code versions are keyed by their packaged content, so `init` drops the commit requirement along with `--commit`, `--allow-dirty`, the commit prompt and the git helpers that only served them. - gix no longer needs its `status` feature. * fix: package source as a zip, from any workspace location - Source packages are a real zip archive, which is what the compute provider opens; they were a gzipped tar uploaded as `source.zip`, so every remote source job failed at extraction. Entries are in path order with fixed timestamps and modes, files sit at the archive root, and the code version digest is unchanged. - A workspace in a subdirectory of a git repository, or containing a nested repository, now packages its files; repository paths were joined onto the workspace root, so nothing was found. - `zip` replaces the `tar` and `flate2` dependencies. * feat: tracel run --remote queues the project as a job - `run --remote <GROUP>` packages the workspace like `tracel package`, confirms, uploads and queues a job with the arguments after `--`, quoted so the compute provider receives them unchanged. - Without a terminal it needs `--yes` (exit 7), checked before building. `--dry-run` builds the package and reports the plan without uploading or queuing. - `--follow` streams the job's logs (NDJSON in JSON mode, starting with a `queued` item) and exits like `tracel jobs wait`. - `package` and `run --remote` share one packaging module, and confirmation and project resolution helpers are factored out. * feat: compare two experiments - `tracel experiments compare <BASE> <OTHER> [--metric NAME]...` lists the config and attribute differences by dotted path and, for each metric and group, both best values from the run summaries and their delta. The server does not report which direction is better, so `compare` does not judge. - Every experiments subcommand resolves the project before logging in and names a missing experiment with a hint to list them. * feat: run jobs by name with tracel run - `tracel run --list` builds the workspace binary that depends on `tracel`, asks it for its job definitions with TRACEL_DESCRIBE, and caches them in target/tracel/jobs.json by binary hash. - `tracel run <JOB> [-c FILE]... [--set PATH=VALUE]... [--like NUM]` resolves the input from the job's example (or a past experiment's recorded config) with JSON merge patch and dotted paths, validates it against the job's JSON schema, and runs the binary directly. - The run records on Console when a project is linked and a login is available, offline otherwise or with `--offline`. The CLI reads the program's run report to announce the experiment and decide the exit code; signals are forwarded and the program is killed after a 30-second grace period. - `tracel run <JOB> --remote <GROUP>` sends the job and its validated input to the compute provider. - Cargo builds for `package` and `run` share one helper that keeps compiler diagnostics visible. * refactor: package with ignore rules alone, without git - Packaging lists files with one `ignore` walk that honors .gitignore, .ignore and git's exclude files whether or not the workspace is a git repository; `.git` and `target` are always left out. Dotfiles are packaged in a repository unless ignored, and skipped outside one. - `tracel init` no longer runs `git init`, and the gix dependency is gone, which drops about a third of the dependency tree. - Packaging nothing is a usage error instead of an empty upload. * fix: read only namespace and project from tracel.toml - The `owner` and `name` aliases are gone. Any other key in tracel.toml is a usage error that names the expected keys. - One strict reader replaces the two parsers that read the file. * chore: declare the Rust version the dependencies need - `rust-version` is 1.97.0, the lowest toolchain that builds and tests the workspace; the client dependency needs at least 1.89, which 1.87 could not build. - Nested `if` blocks that clippy now collapses into let-chains are rewritten. * fix: binary packages use the source digest The digest of a binary package hashed the uploaded binaries and their targets, so each target set and each mode made its own code version and Console never merged them. Both modes now use the digest of the files a source package holds: binaries for any target and a source package of the same code land in one code version, which Console extends. Binary mode still uploads only the binaries. * feat: tracel run takes the job's own flags - The definitions file and the run report come from the shared `tracel-job` crate, which the SDK writes them with, instead of copies of their types. - `tracel run <JOB>` takes the flags the job's binary has, such as `--epochs 5 --optimizer.lr 0.01`, typed by the job's input schema or example input; `tracel run <JOB> --help` lists them beside its own. - The input resolves from the example (or `--like`), then `-c` files, then the job's flags, then `--set`. `tracel run` keeps its own flags, so a field whose flag has one of their names is set with `--set`. - A bad value or unknown job flag is a usage error naming the flag. * refactor: the launcher decides when a job was stopped - The run report's status is `running`, `completed` or `failed`; a run asked to stop still ends as completed or failed. `tracel run` knows a job was stopped from its own facts: it asked the program to stop, or the program exited 130. - A stopped job fails with `JOB_FAILED` as before, and says what its experiment ended as when the report gives it, such as "Job 'mnist' was stopped (experiment 12 completed)." Other endings are unchanged. * refactor: the run report is the job's - The run report describes the job, and the SDK writes it for every job, an inference too: `status` is how the job is going and how it ended, and `experiment` links the experiment the job recorded, `null` until it records one. `tracel run` announces and names an experiment only when the report has one, and points at Console logs only for a Console experiment. - `ExperimentOutcome` is `JobOutcome`. A stopped job says how it ended when the report says, with its experiment, if any, such as "Job 'mnist' was stopped (it completed; experiment 42)." or "Job 'mnist' was stopped (it failed): loss is NaN". Exit codes are unchanged. * refactor: a job has no kind - The job definitions have no `kind`: a job is something `tracel run` and the runners run, whatever it records or outputs. `tracel run --list` lists each job's name and description. - `tracel-job` is pinned to the SDK commit that removes the kind. * refactor: a job ends on its exit code, with the SDK's tracel-job - `tracel run` builds against the SDK's new `tracel-job`: the protocol of jobs run as processes, and, with its `cli` feature, the command line generated from a job's input schema. It is pinned to the SDK's `feat/tracel-runtime` branch. - There is no run report: the program gets no `TRACEL_REPORT_FILE`, and `tracel run <job>` doesn't announce or name an experiment. The SDK prints where it records into the job's output. The JSON result of `run <job>` has no `report`. - The program's exit code says how the job ended: 0 completed, 2 a job or input the program rejected (`USAGE`), and any other code or a signal failed (`JOB_FAILED`), such as "Job 'mnist' failed (exit code 1)." A job `tracel` asked to stop fails as "Stopped job 'mnist'." - Only a job whose input has a JSON schema has flags, as the SDK's generated command line has; a flag for any other job is a usage error that says so. - The definitions file has no `runner`, and the hints name `tracel::runtime::Jobs`. * feat: a stop kills the job and what it started - The job's program has no stop protocol, so the first Ctrl-C, SIGTERM or SIGHUP `tracel run <job>` gets kills it at once, with SIGKILL to the process group it leads: the processes it started, such as data loaders, end with it. There is no SIGTERM and no 30-second grace period, and no "Stopping" or "Killed" lines. - A stopped job fails with `JOB_FAILED`, as "Stopped job 'mnist'.", on a line of its own after the `^C` the terminal echoed. - Listing a binary's jobs shares the same wait, and still kills the binary when it hasn't listed them within 10 seconds. * feat: tracel always builds in release - `tracel run <job>` and `tracel run --list` build the job binary with `cargo build --release --package <package> --bin <binary>`, under a spinner, and show cargo's output only when the build fails. `tracel run -- <args>` is `cargo run --release -- <args>`. - `tracel package --mode binary --bin <name>` and `tracel run <job> --remote` build only that binary, with its package, so a binary of the same name in another package can't stand in for it, and the remote run uploads the build `tracel run` made for this machine. - One build helper in `tools::cargo` makes every release build, and finds a `--bin` among the workspace's binaries. * feat: run and jobs follow the CLI's message style and layout - `tracel run <job>` is a flow, "Run job 'train'", or "Run job 'train' on gpu" with `--remote`, and the program writes among its lines. `--remote` checks its mode and confirmation before building anything, with or without a job. - `run --remote` asks to queue the job on the group with its short code version, says "Queued job 14 with code version 67c31f69845e.", and ends on the commands that follow its logs and wait for it. With `--follow`, the logs come after that line. `tracel init` ends on `tracel run --list`. - `jobs list` and `jobs get` name the job a `run <job> --remote` queued instead of its quoted command, show times to the minute and the short code version, and `jobs get` shows when the job ran and for how long, and its input. A job's state reads as "Job 14 completed.", "failed.", "was cancelled." or "is stopping.". - `jobs logs` shows the page as it is, then the command that reads the next one. `jobs wait` says what the job is doing while it waits, and `jobs cancel` ends on "Cancelled job 14." or "Job 14 is stopping.". - `experiments compare` writes each part under its heading, with its empty line in the table, and sums up the differences. - A value that is a web address shows as a link. `tracel run --help` into a reader that closes early is not an error. - Jobs' command quoting, polling and log following live in one helpers module that `jobs` and `run --remote` share. Messages, hints and doc comments follow the CLI's style. * test: one readable test per behavior for runs and jobs Tests follow the rest of the CLI: given/when/then names, and named rstest cases for inputs that share a behavior. - tracel run: which flags are the job's, how they set its input, the order the input's sources win in, schema checks, the choice of the job binary, the definitions cache, and how a run ends. - Jobs: the command a queued job runs and reading it back, reading logs on, waiting, and how long a job ran. - Packaging: the digest, the files packaged and the archive, each in a test of its own; binary packages share the source package's digest. - No tests of clap's argument conflicts or of the JSON shape of events. * docs: the README and the reference describe runs as they are now - The README starts with listing and running a job, says the run is recorded on the Tracel Console once the workspace is linked and you're logged in, that Ctrl-C stops the job, and that `tracel run` without a job is `cargo run --release`. - The reference drops the run report and `runner`, says how the exit code ends a job and that a stop kills it and what it started, which release build runs and is uploaded, that only a job with an input schema has flags, and how `jobs logs` and `run --remote` end. - Packaging uploads only the files a code version doesn't have yet. * chore: build against tracel-client after its job reads and the SDK's tracel-job This branch still moves bytes with the client's presigned URL transfers, which tracel-client 0.22.0 removes, so it builds against the tracel-client commit that merged the job reads and still has them. tracel-job is not released yet, so it stays pinned to the head of the SDK pull request that adds it.

jwric added this pull request to stack #42 October 7, 2026 14:03

jwric force-pushed the feat/jobs-and-remote-runs branch 2 times, most recently from a42a48d to 54585a2 Compare October 7, 2026 20:50

This was referenced Oct 7, 2026

feat: job reads, cancel and the queued job number tracel-ai/tracel-client#55

Merged

feat: runner protocol tracel-ai/tracel#200

Closed

jwric force-pushed the feat/jobs-and-remote-runs branch 2 times, most recently from 065fd7d to 770023d Compare October 14, 2026 16:33

This was referenced Oct 14, 2026

feat: tracel-runtime and the tracel-job protocol crate tracel-ai/tracel#202

Merged

feat: a scriptable CLI with experiment, model and artifact commands #40

Merged

jwric changed the title ~~Feat/jobs and remote runs~~ feat: tracel run, tracel jobs and remote runs Oct 14, 2026

jwric force-pushed the feat/jobs-and-remote-runs branch from 770023d to 43c8f1c Compare October 14, 2026 17:03

jwric mentioned this pull request Oct 14, 2026

feat: transfers through tracel-artifact, concurrent and with byte progress #43

Merged

jwric marked this pull request as ready for review October 14, 2026 21:24

jwric force-pushed the feat/jobs-and-remote-runs branch from 43c8f1c to 541a649 Compare October 14, 2026 21:26

jwric added 15 commits October 14, 2026 18:01

chore: build against the client's job API

83876dc

- `tracel-client` is pinned to the client's jobs branch, which adds job reads, cancellation and the job number a queued job gets.

feat: tracel run replaces tracel train

1282992

- `tracel run` runs the project through `cargo run`, as `train` did, with the same environment and exit code. - Arguments reach the program only after `--`, so `tracel run` stays free for a job name.

fix: init no longer needs a git commit

0008da9

- Code versions are keyed by their packaged content, so `init` drops the commit requirement along with `--commit`, `--allow-dirty`, the commit prompt and the git helpers that only served them. - gix no longer needs its `status` feature.

fix: read only namespace and project from tracel.toml

f1a5fb3

- The `owner` and `name` aliases are gone. Any other key in tracel.toml is a usage error that names the expected keys. - One strict reader replaces the two parsers that read the file.

jwric added 9 commits October 14, 2026 18:01

refactor: a job has no kind

6e0b4ed

- The job definitions have no `kind`: a job is something `tracel run` and the runners run, whatever it records or outputs. `tracel run --list` lists each job's name and description. - `tracel-job` is pinned to the SDK commit that removes the kind.

jwric force-pushed the feat/jobs-and-remote-runs branch from 541a649 to 9e9f5af Compare October 14, 2026 22:02

Marc-AnthonyG approved these changes Oct 14, 2026

View reviewed changes

Marc-AnthonyG merged commit 2ac3f9d into main Oct 14, 2026
2 checks passed

Marc-AnthonyG deleted the feat/jobs-and-remote-runs branch October 29, 2026 19:07

Navigation Menu

Repository navigation

feat: tracel run, tracel jobs and remote runs - #41

feat: tracel run, tracel jobs and remote runs#41
Marc-AnthonyG merged 24 commits into
feat/scriptable-cli-and-readsfrom
feat/jobs-and-remote-runs

jwric commented Oct 7, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

jwric commented Oct 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jwric commented Oct 7, 2026 •

edited

Loading