Skip to content

feat(flat-files): add scheduled batch and S3 test jobs - #68

Open
3rdmike wants to merge 6 commits into
mainfrom
LOC-7473-create-cronjobs-for-batch-files
Open

3rdmike wants to merge 6 commits into
mainfrom
LOC-7473-create-cronjobs-for-batch-files

Conversation

@3rdmike

@3rdmike 3rdmike commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

Add a single combined runner that generates the daily partner files and retries pending delivery/archive steps. Define a suspended-by-default Vancouver CronJob at 19:00 through 23:00 and allow environments to activate it with FLAT_FILE_JOB_SUSPEND=false.

Add an S3 connectivity test command and unit coverage for runner sequencing and error handling.

Description

Please provide a summary of the change and the issue fixed. Please include relevant context. List dependency changes.

Fixes # (issue)

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update
  • Documentation update

How Has This Been Tested?

  • New unit tests
  • New integrated tests
  • New component tests
  • New end-to-end tests
  • New user flow tests
  • No new tests are required
  • Manual tests (description below)
  • Updated existing tests

Checklist

  • I have read the CONTRIBUTING doc
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • Any dependent changes have already been accepted and merged

Further comments

Add a single combined runner that generates the daily partner files and retries pending delivery/archive steps. Define a suspended-by-default Vancouver CronJob at 19:00 through 23:00 and allow environments to activate it with FLAT_FILE_JOB_SUSPEND=false.

Add an S3 connectivity test command and unit coverage for runner sequencing and error handling.
@github-actions

Copy link
Copy Markdown

🚀 Deployments Available

Prefix flat-file archives and the S3 connectivity-test object with address_change/ so both are grouped under the requested bucket folder. Update tests and docs to match.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The job pods lack database ingress access when run in disposable or PR environments; they need a distinct worker label allowed by the database policy to avoid routing backend HTTP traffic to them.

1 open issue
1 file not reviewed
  • backend/package-lock.json: Generated file
What changed in this PR

Adds scheduled partner flat-file processing and an S3 connectivity check, using the backend’s existing generation, delivery, and archival services.

Changes:

  • Adds a combined generation-and-retry runner with unit coverage.
  • Defines a suspended-by-default CronJob running hourly from 19:00 through 23:00 Vancouver time, controlled through deployment configuration.
  • Adds an S3 upload/download/cleanup check, documents its use, and updates the archive prefix.
File Description
docs/​src/​content/​docs/​flat-file-s3-test.mdx Documents S3 verification and required configuration.
docs/​astro.config.mjs Adds the S3 guide to navigation.
backend/​src/​flat-file/​test-s3-upload.ts Implements S3 round-trip verification and cleanup.
backend/​src/​flat-file/​test-s3-upload.spec.ts Tests verification and failure handling.
backend/​src/​flat-file/​s3.transport.ts Adds download/delete operations and updates the object prefix.
backend/​src/​flat-file/​run-flat-files.ts Adds the combined runner’s application entry point.
backend/​src/​flat-file/​flat-file.service.spec.ts Updates expected S3 object paths.
backend/​src/​flat-file/​flat-file.job.ts Sequences generation and pending transfers.
backend/​src/​flat-file/​flat-file.job.spec.ts Tests runner sequencing and error handling.
backend/​package.json Adds runner and S3 verification commands.
backend/​package-lock.json Refreshes dependency classification metadata.
backend/​openshift.deploy.yml Adds the CronJob and shared, Secret-backed environment configuration.
.github/​workflows/​reusable-deploy.yml Passes the environment’s job suspension setting.

Review effort: Balanced

Comment thread backend/openshift.deploy.yml Resolved
Label scheduled batch pods separately from HTTP backends and allow same-preview workers through the disposable database ingress policy. Add offline selector tests for worker access, cross-preview isolation, and exclusion from backend Service routing.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants