Repository navigation

DM-56864: Stage chunks with BigQuery load jobs - #69

Open

JeremyMcCormick wants to merge 30 commits into

mainfrom

tickets/DM-56864

Open

DM-56864: Stage chunks with BigQuery load jobs#69
JeremyMcCormick wants to merge 30 commits into
mainfrom
tickets/DM-56864

Commits on Nov 4, 2026
- Set geo_point when a chunk is staged and copy it on promotion
  
  JeremyMcCormick
  and
  claude
  committed
  22c142f
- Cluster the staging DIA tables by apdb_replica_chunk
  
  JeremyMcCormick
  and
  claude
  committed
  7d9db5d
- Remove object_prefix and upload chunks to the top of the bucket
  
  JeremyMcCormick
  and
  claude
  committed
  f7b909a
- Fix dataset cleanup in the dataset tests
  
  JeremyMcCormick
  and
  claude
  committed
  50df586
- Add aiohttp dependency
  
  JeremyMcCormick
  and
  claude
  committed
  0b823bc
Commits on Nov 6, 2026
- Add chunk staging with BigQuery load jobs
  
  JeremyMcCormick
  and
  claude
  committed
  3fa6f55
Commits on Nov 9, 2026
- fixup! Claim a chunk for staging with a condition update
  
  JeremyMcCormick
  and
  claude
  committed
  fa347ce
- fixup! Fail a chunk when submitting its staging transaction has an unknown outcome
  
  JeremyMcCormick
  and
  claude
  committed
  2453cb2
- fixup! Move the network error tuple to gcp.error_handling as NETWORK_ERRORS
  
  JeremyMcCormick
  and
  claude
  committed
  ef81c00
- fixup! Decide from the job's state whether a staging wait error has an unknown outcome
  
  JeremyMcCormick
  and
  claude
  committed
  0bcc650
- fixup! Retry database errors from google.auth that are marked retryable
  
  JeremyMcCormick
  and
  claude
  committed
  24c4032
- fixup! Treat a staging job that finished without an error as committed
  
  JeremyMcCormick
  and
  claude
  committed
  019e22a
- fixup! Clear the staging job ID of an earlier attempt when claiming a chunk
  
  JeremyMcCormick
  and
  claude
  committed
  256f98c
- fixup! Classify Cloud SQL Connector errors as database errors
  
  JeremyMcCormick
  and
  claude
  committed
  a7fb473
- fixup! Retry a chunk when submitting its staging transaction is rate limited
  
  JeremyMcCormick
  and
  claude
  committed
  6002e50
- fixup! Test a chunk's final status when staging fails
  
  JeremyMcCormick
  and
  claude
  committed
  f87831a
- fixup! Use a realistic error message in the staging exception pickle test
  
  JeremyMcCormick
  and
  claude
  committed
  0de993b
- fixup! Remove the staging exception pickle test
  
  JeremyMcCormick
  and
  claude
  committed
  3ad013f
- fixup! Name what each part of the stager error test setup creates
  
  JeremyMcCormick
  and
  claude
  committed
  60cb98a
- fixup! Skip the staging row check in the stager error tests instead of faking its query
  
  JeremyMcCormick
  and
  claude
  committed
  5423c98
- fixup! Name the chunk ID and folder once and drop the unused schema in the stager error tests
  
  JeremyMcCormick
  and
  claude
  committed
  9c9d825
- fixup! Replace the staging row check on the stager instance instead of patching the class
  
  JeremyMcCormick
  and
  claude
  committed
  b89d302
- fixup! Comment the assertion blocks in the stager error tests
  
  JeremyMcCormick
  and
  claude
  committed
  a275c1f
- fixup! Note why the stager integration checks are in one test
  
  JeremyMcCormick
  and
  claude
  committed
  5516a8b
Commits on Nov 11, 2026

Navigation Menu

Repository navigation

DM-56864: Stage chunks with BigQuery load jobs - #69

DM-56864: Stage chunks with BigQuery load jobs#69
JeremyMcCormick wants to merge 30 commits into
mainfrom
tickets/DM-56864

Commits on Nov 4, 2026

Set geo_point when a chunk is staged and copy it on promotion

Cluster the staging DIA tables by apdb_replica_chunk

Remove object_prefix and upload chunks to the top of the bucket

Fix dataset cleanup in the dataset tests

Add aiohttp dependency

Commits on Nov 6, 2026

Add chunk staging with BigQuery load jobs

Commits on Nov 9, 2026

fixup! Claim a chunk for staging with a condition update

fixup! Fail a chunk when submitting its staging transaction has an unknown outcome

fixup! Move the network error tuple to gcp.error_handling as NETWORK_ERRORS

fixup! Decide from the job's state whether a staging wait error has an unknown outcome

fixup! Retry database errors from google.auth that are marked retryable

fixup! Treat a staging job that finished without an error as committed

fixup! Clear the staging job ID of an earlier attempt when claiming a chunk

fixup! Classify Cloud SQL Connector errors as database errors

fixup! Retry a chunk when submitting its staging transaction is rate limited

fixup! Test a chunk's final status when staging fails

fixup! Use a realistic error message in the staging exception pickle test

fixup! Remove the staging exception pickle test

fixup! Name what each part of the stager error test setup creates

fixup! Skip the staging row check in the stager error tests instead of faking its query

fixup! Name the chunk ID and folder once and drop the unused schema in the stager error tests

fixup! Replace the staging row check on the stager instance instead of patching the class

fixup! Comment the assertion blocks in the stager error tests

fixup! Note why the stager integration checks are in one test

Commits on Nov 11, 2026

fixup! Put all before the imports in ppdb_bigquery_config and error_handling

fixup! Move the manifest's table mapping to Manifest.table_files, as one dict of file names and row counts

fixup! Make ChunkStagingRetryableError unrelated to ChunkStagingError

fixup! Drop the claim that the job was cancelled from the staging result error

fixup! Treat only 5xx Cloud SQL Admin API responses, not 600 and above, as retryable

fixup! Give the stager error tests' fake manifest table_files instead of files