Skip to content

README.md

Latest commit

 

History

History
280 lines (197 loc) · 8.42 KB

README.md

File metadata and controls

280 lines (197 loc) · 8.42 KB
Raw

DataHub Smoke Tests

This directory contains end-to-end smoke tests for DataHub functionality. These tests can be run locally for faster development and debugging compared to the full CI pipeline.

Writing tests: follow AGENTS.md (isolation, fixtures, markers, cleanup). Running tests: this README, or from the repo root scripts/dev/datahub-dev.sh test <test-path> (preferred over ./gradlew quickstartDebug).

Quick Start

Prerequisites

  1. DataHub must be running locally

    # From project root
./gradlew quickstartDebug

  2. Set up Python environment (one-time setup)

    # From project root - sets up metadata-ingestion venv
./gradlew :metadata-ingestion:installDev

# Set up smoke-test specific environment
cd smoke-test
python3 -m venv venv
source venv/bin/activate
pip install --upgrade pip wheel setuptools
pip install -r requirements.txt

Environment Variables

export DATAHUB_VERSION=v1.0.0rc3-SNAPSHOT  # or current version
export TEST_STRATEGY=pytests

Running Tests

cd smoke-test
source venv/bin/activate

# Set environment variables
export DATAHUB_VERSION=v1.0.0rc3-SNAPSHOT
export TEST_STRATEGY=pytests

# Run all live-stack tests (WARNING: Takes a long time, requires full setup)
pytest -vv

# Run specific test file (RECOMMENDED for development)
pytest tests/e2e/test_system_info.py -vv

# Run specific test method
pytest tests/e2e/test_system_info.py::test_system_info_main_endpoint -vv

# Run multiple specific tests
pytest tests/e2e/test_e2e.py::test_healthchecks tests/e2e/test_e2e.py::test_gms_usage_fetch -v

# CPU tests (no DataHub stack)
./gradlew :smoke-test:stackFreePytest
# or: pytest tests/unit -vv

Selecting tests by domain

Tests can declare the product domain that owns them with @pytest.mark.domain(...), using the Domain enum in utilities/domains.py (platform, observe, ingestion, ai, catalog). The --domain option then runs only the tests those domains own:

# One domain
pytest --domain catalog -vv

# Several — a test owned by any of them runs
pytest --domain catalog --domain ingestion -vv

A test that spans domains declares all of them, e.g. @pytest.mark.domain(Domain.CATALOG, Domain.INGESTION), and is selected by either.

Selecting tests by criticality tier

Tests critical enough to gate every pull request carry @pytest.mark.p0. Select them with an ordinary marker expression:

# Only the p0 tier
pytest -m p0 -vv

# The p0 tier within one domain
pytest -m p0 --domain catalog -vv

CI drives the choice through the SMOKE_TIER env var that smoke.sh reads: docker-unified.yml sets it to p0 for pull requests while the PYTEST_P0_SMOKE repository variable is true, and leaves post-merge runs on the full suite.

On a pull request CI additionally runs any test module the PR itself touches, even when it carries no p0 marker — conftest marks those modules p0 during collection so -m p0 keeps them. That is per-module: a change to a shared fixture or to conftest.py pulls in no test module of its own, so a PR needing broader coverage asks for the whole suite with the run-all-tests PR label.

Add run-all-tests to a pull request to run every test on it instead of the p0 tier — for changes whose blast radius is wider than their own touched modules, such as a shared fixture, conftest.py, or a broad refactor. The label is read from the event payload, as every other label this CI honours is, so it applies on the PR's next push; re-running an existing workflow replays the original payload and will not see it.

Test Categories

System Info Tests (test_system_info.py)

✅ FAST - Can run independently

Tests the system info API endpoints:

  • /openapi/v1/system-info - Spring components only
  • /openapi/v1/system-info/properties - Detailed properties
  • /openapi/v1/system-info/spring-components - Component status
# Run all system info tests (takes ~30 seconds)
pytest test_system_info.py -vv

Core E2E Tests (test_e2e.py)

⚠️ SLOW - Requires full ingestion pipeline

Tests that require data ingestion and full DataHub functionality. Many tests depend on the initial ingestion fixture which can fail if Kafka/Schema Registry aren't properly configured.

# Run health checks only (fast)
pytest test_e2e.py::test_healthchecks -v

# Run authentication tests (fast)
pytest test_e2e.py::test_frontend_auth -v

# Run full e2e tests (slow, requires full setup)
pytest test_e2e.py -vv

Development Workflow

Testing System Info Changes

After making changes to system info APIs:

  1. Restart DataHub

    # Kill existing processes
./gradlew :datahub-frontend:stop :datahub-gms:stop

# Restart
./gradlew quickstartDebug

  2. Run System Info Tests

    cd smoke-test
source venv/bin/activate
export DATAHUB_VERSION=v1.0.0rc3-SNAPSHOT
export TEST_STRATEGY=pytests

pytest test_system_info.py -vv

Quick API Verification

# Check if DataHub is running
curl -s http://localhost:8080/health | head -5

# Test system info endpoint directly
curl -s http://localhost:8080/openapi/v1/system-info | jq . | head -20

Troubleshooting

Common Issues

❌ "Connection refused" errors

  • DataHub is not running
  • Wrong port (should be 8080 for GMS)
  • Services still starting up (wait a few minutes)

❌ "401 Unauthorized" for direct curl

  • Expected behavior - tests handle authentication
  • Use the test suite instead of direct curl for authenticated endpoints

❌ Kafka/Schema Registry connection errors

  • Only affects full e2e tests with ingestion
  • System info tests should still work
  • Try running individual test methods instead of full suite

❌ Python environment issues

# Recreate virtual environment
rm -rf venv
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

Environment Debug

# Check if services are running
curl -s http://localhost:8080/health
curl -s http://localhost:9092  # Kafka (will show connection refused if not running)

# Verify Python environment
source venv/bin/activate
which python
python --version
pip list | grep datahub

# Check environment variables
echo "DATAHUB_VERSION: $DATAHUB_VERSION"
echo "TEST_STRATEGY: $TEST_STRATEGY"

CI vs Local Testing

  • CI: Uses ./gradlew :smoke-test:pytest - full pipeline with Docker containers
  • Local: Uses direct pytest - faster, uses locally running DataHub instance
  • Recommendation: Use local for development, CI for final validation

Simulate batch packing without running tests

CI splits pytests with BATCH_COUNT (3 on forks, 4 for the p0 tier, 7 for the full suite) and PYTEST_XDIST_WORKERS=3. To dump the predicted pack plan for one of those counts without executing tests, load the opt-in plugin with -p (it is not registered in conftest.py):

cd smoke-test
source venv/bin/activate
export PYTHONPATH=.
PLUGIN="-p batch_plan_dump"

# Full suite, 7 batches
BATCH_COUNT=7 PYTEST_XDIST_WORKERS=3 \
  pytest $PLUGIN --collect-only --dump-batch-plan /tmp/plan-7.json

# p0 gate, 4 batches
BATCH_COUNT=4 PYTEST_XDIST_WORKERS=3 \
  pytest $PLUGIN --collect-only -m p0 --dump-batch-plan /tmp/plan-p0-4.json

# CI: artifact path from env (overrides --dump-batch-plan)
SMOKE_DUMP_BATCH_PLAN=/tmp/plan-catalog-3.json BATCH_COUNT=3 PYTEST_XDIST_WORKERS=3 \
  pytest $PLUGIN --collect-only --domain catalog

PYTHONPATH=. is required: pytest loads -p plugins before the smoke-test directory is on sys.path. Use -p batch_plan_dump from smoke-test/ (loader next to shard_pack.py). The dump logic lives in utilities/batch_plan_dump.py.

--collect-only is required if you do not want tests to run; collection still imports the suite and needs this venv, but it does not start DataHub. The plugin also clears collected items as a backstop. Compare shard counts by invoking the same command again with a different BATCH_COUNT. If the plugin is loaded and neither SMOKE_DUMP_BATCH_PLAN nor --dump-batch-plan is set, JSON is written to stdout.

Test Organization

  • test_e2e.py - Main test suite (1387 lines)
  • test_system_info.py - System info API tests (169 lines)
  • conftest.py - Test configuration and fixtures
  • tests/utils.py - Test utilities and helpers

💡 Pro Tip: For rapid development, use pytest test_system_info.py -vv which runs in ~30 seconds vs the full test suite which can take 30+ minutes.