Skip to content
0trmPublic

About

🏵️ Skills that make your coding agent think like a product ML engineer.

Resources

Stars

1 star

Watchers

0 watching

Forks

Latest commit

 

History

56 Commits

Folders and files

Repository files navigation

bluflowers by Ono Chikkyō: three stems on a blue ground, two in orange bloom and one in bud

bluML

Skills that make your coding agent think like a product ML engineer.

how it works · skills · evals · install

Why

bluML gives the agent the habits of an engineer who has shipped models into products: ask what the prediction will decide before training anything, beat the rule before beating a model, distrust a good number until the split and the features are checked, launch in steps that can be taken back, and find the cause before retraining.

With its sibling gallop installed, gallop keeps the questions around a model: forecasts made for a plan, uplift targeting, and reading whether a launch moved anything.

Five laws

  1. Ship a rule first. It is the baseline to beat and the pipeline that collects the labels.
  2. Write the metric ladder down before training anything. The number the model moves has to move the number the business cares about.
  3. Better data beats a cleverer model, and a leak makes any model look better than it is.
  4. A threshold is a product decision. Set it with the people who pay for each kind of error.
  5. Assume the live model is getting worse right now, and keep the instruments that would show it.

How it works

The bluML map, a flowchart in three phases. It starts at a request for a model. Design: the model is framed, then a decision: is a model worth it? If not, the rule ships. If so, design and data follow. Development: train and evaluate. Deployment: launch and monitor, then a decision: is the model still paying? If not, it is retired to the rule; if so, monitoring continues. Dashed arrows loop back: a rejected design and a lost online test return to framing, a model that loses to the rule returns to data, and a live model returns to training when the world moves and to data when an input breaks.

Every model gets one file, MODEL.md, next to its code. Each skill reads it first and writes its section last: the decision and what each error costs, the design review, the data, the runs, the verdict, the launch, the incidents. Each section ends at a gate, a short checklist that needs evidence: the agent marks the items that hold, only the model's owner ticks them, and a stage starts when the owner has ticked the gate before it and said go. The file is how a model's reasoning survives between sessions, between people and between versions. A filled example follows a delivery-time model from framing to its first incident: its MODEL.md and its design doc.

Solid arrows are the main path, and each step on it ends at a gate; dashed arrows are the ways work loops back. The grey bands group the steps into design, development and deployment. Between framing and the design doc, exploring-data puts the data in front of the model's owner in one JupyterLab notebook, chart by chart, with the code folded away and the owner's reading written before the agent's. The built table, training and evaluation get a notebook of the same shape (data.ipynb, train.ipynb, eval.ipynb), with a stop at every decision: the agent states the question, the options and the rule before running anything, the owner writes a reading and a pick before opening the agent's folded recommendation, then decides in their own words, and the next step starts only on the owner's go. A rule changed after its result was seen is logged as a deviation. Asked to work alone, the agent goes through the stops and leaves each decision open in MODEL.md for the owner. Before anything is built, a design doc goes through a rough pre-review and then a formal review, where design flaws are cheapest to fix. Work loops back: a rejected design returns to framing, errors point at the data, a live model goes back to training when the world moves and to the pipeline when an input breaks; a launch that loses its online test sends the objective back to framing, and a live model that stops paying for itself is retired to the rule. doubting-results runs across all of it, wherever a number is about to be reported.

Skills

Skill Reach for it when
framing-models someone asks for a model, a request names a model before the decision it serves, or someone asks whether a model is worth it
exploring-data a model is framed and its owner has not looked at the data yet, someone asks for EDA or to profile the data, or a source or the tracking changes
writing-design-docs a framed model is ambiguous, high impact or more than a few days of work, or someone asks for a design doc, spec or proposal
building-datasets assembling training data, choosing a label or a split, when labels are scarce or noisy, or when one column predicts suspiciously well
training-models fitting, comparing or tuning models, choosing features, or when a model will not beat a simple baseline
evaluating-models choosing a threshold, comparing with the rule, or deciding whether a model is ready
launching-models a model is about to replace a rule or an older model, or an offline gain is offered as the reason to ship
monitoring-models a live number moves, someone asks to retrain, a model has no alerts, or its costs change
doubting-results before any model number is reported, and whenever a result beats expectations

Four skills carry tested Python checks, run by the agent rather than rewritten each time: a leak screen, a cost and capacity threshold, a slice comparison with paired intervals and a drift report.

Does it help?

without with bluML
Sonnet 0.67 0.92
Opus 0.84 0.96
Opus 5.5, high effort 0.89 0.92

Sonnet and Opus: version 0.2.0 on seven cases, 21 October 2026. Opus 5.5 at high effort: version 0.5.0 on eight cases, 8 November 2026.

Per case, and what changed

Requests with traps planted in synthetic data, each run three times with bluML and three times without, and scored by an LLM judge against five or six criteria. A score is the weighted share of criteria met. The table is the round of 21 October. Method, criteria and caveats: evals/.

Case Sonnet Opus
withoutwith bluML withoutwith bluML
fraud-threshold0.521.000.630.96
design-doc0.591.000.931.00
ctr-drop0.580.960.880.96
rule-first0.250.580.500.83
ship-it0.881.000.961.00
churn-leak0.921.001.001.00
ranker-clicks0.960.921.001.00
Mean0.670.920.840.96

Without bluML, both models already catch the textbook traps: a backfilled column, position bias in clicks, prices sent in cents. What they skip is the product reasoning around the model. Neither compared the fraud model with the rule slice by slice, and neither noticed that it ranks new customers far worse (0 of 6 runs; 6 of 6 with bluML). Sonnet never scored the keyword rule on the same held-out tickets as its ticket-routing model (0 of 3; 3 of 3), Opus seldom weighed the model against that rule, which nearly matches it (1 of 3; 3 of 3), and none of Sonnet's design docs set out a product metric, guardrails and an online test (0 of 3; 3 of 3). One gap remains: Sonnet with bluML still seldom weighs the model's small gain over the rule against the cost of running it (1 of 3), and never prices a misrouted ticket.

On 8 November, version 0.5.0 ran on Opus 5.5 at high effort, with an eighth case, feature-choice: 0.92 with bluML and 0.89 without. On October's seven cases the arm with bluML holds at 0.94 (0.96 then), while the arm without it rose from 0.84 to 0.88. bluML still made the agent compare the fraud model with the rule by slice (2 of 3; 0 of 3), notice the new-customer gap (3 of 3; 1 of 3), and weigh the routing model against the keyword rule (3 of 3; 1 of 3). It lost on feature-choice, where 1 of 3 runs backed a column that matters only next to the class hour (3 of 3 without), and it still seldom says how a shipped rule is checked on live tickets (1 of 3; 3 of 3). Version 0.5.1 fixed the feature step: in five more runs of feature-choice, 4 backed that column with bluML and 2 without. Version 0.5.2 gave a shipped rule its live check: in five more runs of rule-first, every run in both arms said how readiness would be confirmed on live tickets, and the case scored 0.90 with bluML against 0.55 without. Version 0.5.3 priced the misrouting: 4 of 5 runs with bluML put a price on each kind of misrouted ticket into the case, against none without, and rule-first scored 0.95 against 0.60. Version 0.5.4 chains the stage skills: runs that start in framing-models go on to the next stage's skill (9 of 10), doubting-results loaded for the first time, in 6 of 10 runs, and rule-first scored 1.00 with bluML. Version 0.5.5 does the same from the other entry skills: runs that start in building-datasets, training-models or evaluating-models load the later stages' skills (7 of 8, against 0 of 4), and fraud-threshold scored 1.00. Version 0.5.6 puts each kept feature's number in the reply: feature-choice backed from_waitlist with evidence in 10 of 10 runs, against 16 of 25 before, and scored 0.99. Version 0.5.7 loads the later stages' skills from the top of framing-models: 11 of 11 runs that started there loaded them, against 13 of 15 before. Version 0.5.8 loads evaluating-models and doubting-results from the top of training-models: on a probe that starts there, 5 of 5 runs loaded them, against 3 of 5 on 0.5.7. The probe's score fell from 1.00 to 0.70, from a run that timed out and two replies that left out the duplicate columns. Version 0.5.9 names the dropped near-duplicates in the reply, each beside the column it repeats: feature-choice passed the duplicate-columns criterion in 10 of 10 runs and scored 1.00. Version 0.6.0 stops for the owner at every decision; working alone, as every eval prompt asks, all 14 runs that loaded a skill logged their decisions as open in MODEL.md and ticked no gate, rule-first scored 1.00 and feature-choice 0.95, where one run dropped from_waitlist after choosing a linear model.

Install

As a Claude Code plugin:

/plugin install bluml --marketplace 0trm/bluml

Or copy the skills into a project:

git clone https://github.com/0trm/bluml
mkdir -p .claude/skills
cp -r bluml/skills/* .claude/skills/

The scripts need Python 3.10 or later with numpy and pandas; uv run <script> installs them. Run their tests with uv run --with numpy,pandas,pytest pytest tests.

Scope

bluML covers supervised models that run inside a product: a classifier, a ranker or a forecast whose output the product acts on. LLM applications, research, ML platform work and unsupervised models are out of scope. What each skill is written from, and where its sources disagree: docs/sources.md.

License

MIT for text and code. Painting by Ono Chikkyō (小野竹喬, 1889–1979), For Haiku Poems in Bashō's Narrow Road to a Far Province.

About

🏵️ Skills that make your coding agent think like a product ML engineer.

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages