Skip to content

Add production tooling for MC-DATA embedding - #2550

Open
sawenzel wants to merge 5 commits into
AliceO2Group:masterfrom
sawenzel:swenzel/mc-data-embedding-production
Open

sawenzel wants to merge 5 commits into
AliceO2Group:masterfrom
sawenzel:swenzel/mc-data-embedding-production

Conversation

@sawenzel

Copy link
Copy Markdown
Contributor

This extends the MC-DATA embedding of b83568d to productions. A planner samples a fraction of a run's data AO2Ds and packs whole DF folders into jobs. Each job embeds exactly its DF folders and links every MC DF folder to its data parent via parentFiles. A DF folder is never split across jobs, so no index offsets are needed.

Submission goes through grid_submit.sh, with one split-on-collection masterjob per cycle, so that jobs run close to their data. The job script only reads JDL variables and one input file, the work table, which should also allow submitting the same jobs via LPM.

plan_embedding.py --run 544653 --data-path /alice/data/2023/LHC23zzm/544653/apass4 \
                  --fraction 0.01 --seed 1 --tfs-per-job 200 --out plan
submit_embedding.py --plan plan --anchor-env anchor.env --table-dir <alien dir> --jobname emb --submit
check_embedding_output.py check --plan plan --mc <MC AO2D> --parent <data AO2D>

A 1 % production of run 544653 (LHC23zzm apass4, 89 files, 7828 timeframes, pythia8pp signals) exposed problems that only appear at this scale: data timeframes starting after the recorded end of run, collisions not ordered in time within a timeframe, and the TPC looper generator aborting on contexts taken from data. These are fixed here, and failed tasks are now retried, which covers transient CCDB failures. All 89 outputs pass check_embedding_output.py: the DF folders, the parentFiles links and the BCs of the MC collisions match the data. With --data-anchoring, TPC loopers and QED are now off by default, which reduced the CPU time per timeframe from 85 to 31–67 minutes; --with-qed and GenTPCLoopers.loopersVeto=false bring them back. Each timeframe gets a seed hashed from ALIEN_PROC_ID, its DF folder and its timeframe ID; distinct timeframes share a seed only by chance.

Note that sampled anchored MC is not affected: the anchoring changes apply only to explicit timeframe IDs.

A job still copies the BC and collision tables of its whole data file, also when it treats only some of its DF folders; this is small (about 8 MB per file) and left for later.

Reading MC and data together in an analysis, with --aod-origin-level-mapping EMB:1 as in o2-testworkflows-analysis-emb, does not work yet on these files. The AOD reader opens the correct parent file and then fails; this is left for a separate investigation.

Assisted by Claude Code.

sawenzel and others added 5 commits November 16, 2026 18:32
This fixes the anchoring of data timeframes for MC-DATA embedding and adds unit tests.

- An explicit timeframe ID is counted from the run's first orbit, as in the data, not from the start moved past the ITS ramp-up.
- A data timeframe may start after the end of run recorded in CCDB; it is no longer refused.
- Its interaction rate is read at the last time the CTP counted collisions.
- Its conditions are read just before the end of run, where the run's objects are valid.
- Sampled anchored jobs are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This switches off the TPC looper generator and the QED background for --data-anchoring and stamps the production's interaction rate into imported contexts.

- The data already carries both backgrounds.
- The looper generator aborted on contexts taken from data, which carried no interaction rate.
- --with-qed and an explicit GenTPCLoopers.loopersVeto still win.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This extends the MC-DATA embedding utils for production use and adds unit tests.

- --dfs restricts the collision contexts to given DF folders, and --expect-tfs stops with exit code 3 if the timeframes differ from the plan.
- --list-units writes DF folder -> timeframe IDs as JSON.
- Collisions are ordered by BC time plus collision time; the AO2D order made the ITS/MFT digitizer fail with "deduced ROFrame precedes already processed one".
- DF folders without triggered collisions no longer crash, and files with more than 1000 DF folders are no longer truncated.
- --set-interaction-rate stores a rate in existing contexts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This provides new tools in MC/run/ANCHOR/dataembedding to run MC-DATA embedding as a production.

- plan_embedding.py samples a fraction of a run's data AO2Ds and packs whole DF folders into jobs (embedding_table.json); a DF folder is never split.
- submit_embedding.py submits the table via grid_submit.sh: one split-on-collection masterjob per cycle, so jobs run close to their data, or one production split.
- check_embedding_output.py merges per parent file and checks the DF folders, the parentFiles links and that MC collisions sit at the data collision BCs; excluded timeframes are taken into account.
- embedding_row.py finds a job's row from its input file and cycle, or from a row index.
- Seeds are a hash of the job id (ALIEN_PROC_ID), DF folder and timeframe; distinct timeframes share a seed only by chance.

    plan_embedding.py --run 544653 --data-path /alice/data/2023/LHC23zzm/544653/apass4 \
                      --fraction 0.01 --seed 1 --tfs-per-job 200 --out plan
    submit_embedding.py --plan plan --anchor-env anchor.env --table-dir <alien dir> --jobname emb --submit
    check_embedding_output.py check --plan plan --mc <MC AO2D> --parent <data AO2D>

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This lets anchorMC_DataEmbedding.sh treat one row of a work table and makes the job robust at scale.

- With ALIEN_JDL_MC_DATA_EMBEDDING_TABLE the job takes its data file and DF folders from the table.
- Each timeframe gets its own seed from the job id (ALIEN_PROC_ID), its DF folder and its timeframe ID.
- Failed tasks are retried twice, and workflow creation is tried twice.
- Each parallel workflow gets its share of the job's CPU and memory.
- The final merge orders inputs by DF folder and timeframe and cannot split a folder on size.
- Timeframes are reported done, failed or excluded in embedding_status.tsv; a failed one makes the job exit non-zero, excluded ones do not.
- The excluded-timestamp and workflow-creation-only paths no longer nest the workspaces.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sawenzel
sawenzel requested a review from jackal1-66 as a code owner November 16, 2026 17:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant