Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
22c142f
Set geo_point when a chunk is staged and copy it on promotion
JeremyMcCormick Nov 4, 2026
7d9db5d
Cluster the staging DIA tables by apdb_replica_chunk
JeremyMcCormick Nov 4, 2026
f7b909a
Remove object_prefix and upload chunks to the top of the bucket
JeremyMcCormick Nov 4, 2026
50df586
Fix dataset cleanup in the dataset tests
JeremyMcCormick Nov 4, 2026
0b823bc
Add aiohttp dependency
JeremyMcCormick Nov 4, 2026
3fa6f55
Add chunk staging with BigQuery load jobs
JeremyMcCormick Nov 4, 2026
fa347ce
fixup! Claim a chunk for staging with a condition update
JeremyMcCormick Nov 9, 2026
2453cb2
fixup! Fail a chunk when submitting its staging transaction has an un…
JeremyMcCormick Nov 9, 2026
ef81c00
fixup! Move the network error tuple to gcp.error_handling as NETWORK_…
JeremyMcCormick Nov 9, 2026
0bcc650
fixup! Decide from the job's state whether a staging wait error has a…
JeremyMcCormick Nov 9, 2026
24c4032
fixup! Retry database errors from google.auth that are marked retryable
JeremyMcCormick Nov 9, 2026
019e22a
fixup! Treat a staging job that finished without an error as committed
JeremyMcCormick Nov 9, 2026
256f98c
fixup! Clear the staging job ID of an earlier attempt when claiming a…
JeremyMcCormick Nov 9, 2026
a7fb473
fixup! Classify Cloud SQL Connector errors as database errors
JeremyMcCormick Nov 9, 2026
6002e50
fixup! Retry a chunk when submitting its staging transaction is rate …
JeremyMcCormick Nov 9, 2026
f87831a
fixup! Test a chunk's final status when staging fails
JeremyMcCormick Nov 9, 2026
0de993b
fixup! Use a realistic error message in the staging exception pickle …
JeremyMcCormick Nov 9, 2026
3ad013f
fixup! Remove the staging exception pickle test
JeremyMcCormick Nov 9, 2026
60cb98a
fixup! Name what each part of the stager error test setup creates
JeremyMcCormick Nov 9, 2026
5423c98
fixup! Skip the staging row check in the stager error tests instead o…
JeremyMcCormick Nov 9, 2026
9c9d825
fixup! Name the chunk ID and folder once and drop the unused schema i…
JeremyMcCormick Nov 9, 2026
b89d302
fixup! Replace the staging row check on the stager instance instead o…
JeremyMcCormick Nov 9, 2026
a275c1f
fixup! Comment the assertion blocks in the stager error tests
JeremyMcCormick Nov 9, 2026
5516a8b
fixup! Note why the stager integration checks are in one test
JeremyMcCormick Nov 9, 2026
67b454c
fixup! Put __all__ before the imports in ppdb_bigquery_config and err…
JeremyMcCormick Nov 11, 2026
dd377ea
fixup! Move the manifest's table mapping to Manifest.table_files, as …
JeremyMcCormick Nov 11, 2026
564455a
fixup! Make ChunkStagingRetryableError unrelated to ChunkStagingError
JeremyMcCormick Nov 11, 2026
1f68223
fixup! Drop the claim that the job was cancelled from the staging res…
JeremyMcCormick Nov 11, 2026
e6c3d83
fixup! Treat only 5xx Cloud SQL Admin API responses, not 600 and abov…
JeremyMcCormick Nov 11, 2026
131243d
fixup! Give the stager error tests' fake manifest table_files instead…
JeremyMcCormick Nov 11, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,7 @@ classifiers = [
]
keywords = ["lsst"]
dependencies = [
"aiohttp",
"astropy",
"cloud-sql-python-connector[pg8000]",
"cloudevents <2",
Expand Down
1 change: 1 addition & 0 deletions python/lsst/dax/ppdb/bigquery/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
# along with this program. If not, see <https://www.gnu.org/licenses/>.

from .chunk_promoter import *
from .chunk_stager import *
from .chunk_uploader import *
from .manifest import *
from .ppdb_bigquery import *
Expand Down
10 changes: 2 additions & 8 deletions python/lsst/dax/ppdb/bigquery/chunk_promoter.py
Original file line number Diff line number Diff line change
Expand Up @@ -288,17 +288,11 @@ def _copy_staging_to_promotion(self) -> None:
)

# Build target column list for SQL statement from the internal
# table's schema. This will include the geo_point column.
# table's schema. geo_point is copied from staging.
target_schema = self._bq_client.get_table(internal_table_fqn).schema
target_names = [target_column.name for target_column in target_schema]
target_list_sql = ", ".join(f"`{column_name}`" for column_name in target_names)

# Build source column list for SQL statement from the target list,
# converting ra/dec to ST_GEOGPOINT for geo_point column.
source_list_sql = ", ".join(
"ST_GEOGPOINT(s.`ra`, s.`dec`)" if column_name == "geo_point" else f"s.`{column_name}`"
for column_name in target_names
)
source_list_sql = ", ".join(f"s.`{column_name}`" for column_name in target_names)

# Insert staged rows into the promotion table. If the staging
# and internal schemas do not match, this will fail.
Expand Down
Loading
Loading