[JeremyMcCormick]

Summary

PPDB infrastructure for staging replica chunks with BigQuery load jobs instead of a Dataflow job. The stage chunk function now loads each chunk into temporary landing tables and copies them into the staging tables itself (lsst/dax_ppdb#69, lsst-dm/ppdb-cloud-functions#38). The Dataflow staging resources and the track chunk function are removed, and the function and the ingest bucket are renamed.

Changes

All changes are in environment/deployments/ppdb/.

• Landing dataset (services/bigquery.tf)
  • New ppdb_landing dataset for the per-chunk landing tables, in the same location (US) as staging
  • Its tables expire after bigquery_landing_table_expiration_days (default 1)
• Stage chunk function (services/stage-chunk.tf, renamed from trigger-stage-chunk.tf)
  • Renamed from trigger-stage-chunk to stage-chunk, along with its service account (cloudrun-stage-chunk), Cloud SQL user, push subscription and variables (trigger_stage_chunk_* → stage_chunk_*)
  • The service account gets dataEditor on the landing and staging datasets (staging was dataViewer)
  • Its storage access is limited to storage.objectViewer on the chunk ingest bucket (was project-wide storage.admin), and its unneeded topic subscriber role is removed
  • New environment variables BIGQUERY_CHECK_TIMEOUT_SECONDS, BIGQUERY_JOB_TIMEOUT_SECONDS and BIGQUERY_RETRY_TIMEOUT_SECONDS, from stage_chunk_bigquery_check_timeout_seconds (60), stage_chunk_bigquery_job_timeout_seconds (240) and stage_chunk_bigquery_retry_timeout_seconds (300)
  • Keeps one instance running (min_instance_count = 1) to avoid cold starts
  • Its push subscription allows 100 delivery attempts before dead-lettering (stage_chunk_pubsub_max_delivery_attempts), about 16 hours of retries, so a chunk can wait out a BigQuery quota
  • Removed its Dataflow permissions and Dataflow environment variables
• Dataflow staging removed (services/dataflow-stage-chunk.tf): the Dataflow service account and its IAM, and the template path and temp location variables. The load_sso Dataflow resources are unchanged.
• Pub/Sub delivery attempts per subscription: the shared pubsub_max_delivery_attempts is replaced by stage_chunk_pubsub_max_delivery_attempts (100) and load_sso_pubsub_max_delivery_attempts (15, unchanged)
• Track chunk removed (services/track-chunk.tf): the function, its service account and Cloud SQL user, the track-chunk-topic and track-chunk-dlt topics, its subscriptions and variables
• Chunk ingest bucket (services/gcs.tf): ppdb-<env>-ingest is replaced by ppdb-<env>-chunk-ingest, with its IAM bindings and variables moved over
• Other
  • The staging dataset description no longer mentions Dataflow
  • The Sentry DSN secret accessors follow the renamed and removed service accounts
  • SQL/iam-sa-permissions.sql updated for the renamed and removed service accounts

Before applying

• Pause chunk uploads: pause the USDF ppdb-replication uploads. The apply deploys placeholder code for stage-chunk, which drops messages until the real function is deployed from ppdb-cloud-functions.
• Empty the old bucket: empty ppdb-<env>-ingest first. GCS can't rename buckets, so the apply deletes it, and it must be empty.
• Revoke old Postgres grants: revoke the grants of the cloudrun-track-chunks and cloudrun-trigger-stage-chunk Postgres users, so their Cloud SQL users can be dropped.
• Drain the old subscription: let the trigger-stage-chunk subscription drain. It is replaced, and messages still in it are lost.

After applying

• Grant the new Postgres user: grant cloudrun-stage-chunk its chunk_tracking permissions (SQL/iam-sa-permissions.sql).
• Upload the configs: upload the PPDB configs with bucket_name: ppdb-<env>-chunk-ingest to gs://ppdb-<env>-config.
• Deploy the function: deploy stage-chunk from ppdb-cloud-functions, then resume the uploads.

Related

• Jira: https://rubinobs.atlassian.net/browse/DM-56864
• dax_ppdb: DM-56864: Stage chunks with BigQuery load jobs dax_ppdb#69
• ppdb-cloud-functions: DM-56864: Stage chunks with BigQuery load jobs instead of Dataflow lsst-dm/ppdb-cloud-functions#38

🤖 Generated with Claude Code

https://claude.ai/code/session_01VJ51nAbGJ4BBgwpjau1VX7

[JeremyMcCormick]

@fajpunk This is tested and working in dev now (staging and promotion).

Some resources like deprecated storage buckets may need to be cleaned-up/deleted manually.

[fajpunk]

I applied this manually. As you said, I had to destroy a vestigial bucket manually, and we had to do some manual stuff in the database to dissociate database objects from service accounts we are getting rid of.

[JeremyMcCormick]

@fajpunk I am going to merge this as I need to work on getting the changes deployed to int and prod. The other two PRs are not approved and merged yet but I'm anticipating that they will be shortly. I will deploy them from their branches to update int. (dev is already updated).