Skip to content

DM-56864: Stage chunks with BigQuery load jobs instead of Dataflow - #38

Open
JeremyMcCormick wants to merge 7 commits into
mainfrom
tickets/DM-56864
Open

DM-56864: Stage chunks with BigQuery load jobs instead of Dataflow#38
JeremyMcCormick wants to merge 7 commits into
mainfrom
tickets/DM-56864

Conversation

@JeremyMcCormick

@JeremyMcCormick JeremyMcCormick commented Nov 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Summary

The stage chunk function now stages replica chunks with dax_ppdb's chunk stager, which uses BigQuery load jobs, instead of launching a Dataflow job (lsst/dax_ppdb#69). The Dataflow staging job and the track chunk function are removed.

Changes

Stage chunk function

  • Stages each chunk with the dax_ppdb chunk stager instead of a Dataflow job
  • The stager, PPDB connection and BigQuery client are created once per instance and reused
  • Reads the chunk ID and folder from the message, and logs and drops a message that is missing either
  • Acknowledges bad messages, rejected chunks and permanent staging failures, and re-raises a retryable staging error so that Pub/Sub retries it; the stager records the chunk's status and logs the outcome
  • Adds the chunk ID to every log entry
  • Reads the BigQuery limits (BIGQUERY_CHECK_TIMEOUT_SECONDS, BIGQUERY_JOB_TIMEOUT_SECONDS, BIGQUERY_RETRY_TIMEOUT_SECONDS) from environment variables set by Terraform, and fails at startup if any is missing
  • Renamed from trigger_stage_chunk to stage_chunk, deployed as stage-chunk, with its GitHub variables renamed to STAGE_CHUNK_*

Removed

  • The Dataflow staging job, its container, its flex template and its build workflows
  • The track chunk function and its deploy workflows, since only the Dataflow job used it

Other changes

  • The deploy-all workflows no longer include the removed jobs
  • The README describes the function as it works now
  • dax_ppdb is pinned to the ticket branch for now, in a "DO NOT MERGE" commit

Deployment notes

  • Deploy this function before the uploader with the new dax_ppdb
  • Copy the TRIGGER_STAGE_CHUNK_* GitHub variables to STAGE_CHUNK_* in each environment, keeping the old ones until dev and int are migrated
  • Set STAGE_CHUNK_TIMEOUT to about 30 minutes, since each BigQuery request can now be retried for up to 5 minutes
  • Deploys are manual: run stage-chunk-deploy-<env> after merging

Related

🤖 Generated with Claude Code

https://claude.ai/code/session_01VJ51nAbGJ4BBgwpjau1VX7

@JeremyMcCormick JeremyMcCormick mentioned this pull request Nov 2, 2026
Open
@JeremyMcCormick JeremyMcCormick mentioned this pull request Nov 3, 2026
Merged
@JeremyMcCormick
JeremyMcCormick force-pushed the tickets/DM-56864 branch 2 times, most recently from 23ed333 to 22c85c8 Compare November 4, 2026 19:59
@JeremyMcCormick
JeremyMcCormick deployed to dev/load-sso-dataflow November 4, 2026 22:18 — with GitHub Actions Active
@JeremyMcCormick
JeremyMcCormick deployed to dev/stage-chunk November 4, 2026 22:26 — with GitHub Actions Active
JeremyMcCormick and others added 5 commits November 4, 2026 19:09
@JeremyMcCormick
DO NOT MERGE: Use dax_ppdb ticket branch
1a6237e
@JeremyMcCormick @claude
Rename trigger_stage_chunk to stage_chunk
e0e1327 
The function's entry point is stage_chunk and it deploys as the Cloud
Run function stage-chunk. Its GitHub variables are now STAGE_CHUNK_CPU,
STAGE_CHUNK_MEMORY and STAGE_CHUNK_TIMEOUT.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VJ51nAbGJ4BBgwpjau1VX7
@JeremyMcCormick @claude
Stage chunks with dax_ppdb's ChunkStager instead of Dataflow
50ee16e 
The function stages the chunk named in each message with ChunkStager,
sharing the PPDB, a BigQuery client and the stager across requests. It
reads the chunk ID and folder from the message, adds the chunk ID to
every log entry, and re-raises only retryable staging errors so that
Pub/Sub retries them. The BigQuery limits (BIGQUERY_CHECK_TIMEOUT_SECONDS,
BIGQUERY_JOB_TIMEOUT_SECONDS and BIGQUERY_RETRY_TIMEOUT_SECONDS) are read
from the environment, set by Terraform.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VJ51nAbGJ4BBgwpjau1VX7
@JeremyMcCormick @claude
Remove the track_chunk function
3347100 
Only the Dataflow staging job published to it. ChunkStager now records
chunk statuses in the database directly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VJ51nAbGJ4BBgwpjau1VX7
@JeremyMcCormick @claude
Remove the stage chunk Dataflow job
4d7bc2f 
ChunkStager stages chunks with BigQuery load jobs, so the Beam job, its
container, flex template and build workflows are no longer used.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VJ51nAbGJ4BBgwpjau1VX7
@JeremyMcCormick
JeremyMcCormick deployed to dev/promote-chunks November 5, 2026 01:41 — with GitHub Actions Active
@JeremyMcCormick
JeremyMcCormick deployed to dev/stage-chunk November 6, 2026 21:44 — with GitHub Actions Active
@JeremyMcCormick
JeremyMcCormick marked this pull request as ready for review November 6, 2026 23:48
@JeremyMcCormick
JeremyMcCormick deployed to int/load-sso-dataflow November 10, 2026 23:49 — with GitHub Actions Active
@JeremyMcCormick
JeremyMcCormick deployed to int/promote-chunks November 10, 2026 23:49 — with GitHub Actions Active
@JeremyMcCormick @claude
Install sdm_schemas from main in stage_chunk and promote_chunks
2be82fa 
The PyPI release of lsst-sdm-schemas has ppdb.yaml 10.0.0, and the
databases are created from main (11.0.0). Temporary, like the dax_ppdb
branch pin.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01VJ51nCVBE3sWQhjpm83YEC
@JeremyMcCormick
JeremyMcCormick deployed to int/stage-chunk November 11, 2026 00:09 — with GitHub Actions Active
@JeremyMcCormick
JeremyMcCormick deployed to int/promote-chunks November 11, 2026 00:09 — with GitHub Actions Active
@JeremyMcCormick
JeremyMcCormick deployed to int/load-sso-dataflow November 11, 2026 00:09 — with GitHub Actions Active
@JeremyMcCormick
JeremyMcCormick requested review from andy-slac and removed request for andy-slac November 11, 2026 01:32
fajpunk
fajpunk approved these changes Nov 11, 2026
View reviewed changes
Comment thread .github/workflows/deploy-all-dev.yaml
workflow_dispatch:

# Independent jobs deploy all six workloads in parallel.
# Independent jobs deploy all four workloads in parallel.

@fajpunk fajpunk Nov 11, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can just change all these to "all workloads in parallel." (no number)

This branch was successfully deployed

8 active (4 outdated) deployments
int/promote-chunks — 2be82fa6 Deployed Nov 11, 2026 by JeremyMcCormick via promote-chunks / deploy #2
int/stage-chunk — 2be82fa6 Deployed Nov 11, 2026 by JeremyMcCormick via stage-chunk / deploy #2
int/load-sso-dataflow — 2be82fa6 Deployed Nov 11, 2026 by JeremyMcCormick via load-sso-build-flex / build-and-push #2
int/load-sso — 2be82fa6 Deployed Nov 11, 2026 by JeremyMcCormick via load-sso / deploy #2
dev/stage-chunk — 4d7bc2fd Deployed Nov 6, 2026 by JeremyMcCormick via deploy #62
dev/promote-chunks — 4d7bc2fd Deployed Nov 5, 2026 by JeremyMcCormick via deploy #81
dev/load-sso-dataflow — 22c85c8f Deployed Nov 4, 2026 by JeremyMcCormick via load-sso-build-flex / build-and-push #4
dev/load-sso — 22c85c8f Deployed Nov 4, 2026 by JeremyMcCormick via load-sso / deploy #4
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@fajpunk fajpunk fajpunk approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@JeremyMcCormick @fajpunk