Repository navigation
fix(flowfiles): bound all entries in pulled tar archives - #443
Open
yotatameguchi-boop wants to merge 1 commit into
Open
yotatameguchi-boop wants to merge 1 commit into
yotatameguchi-boop wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description of the Change
Problem
ExtractTarlimits regular files to 1,000 and their combined size to 2 GiB, but directory entries and skipped entries do not count toward either limit. A container directory with many empty subdirectories can therefore create an unbounded number of host-side staging directories during a pull. Skipped links and rejected paths also leave the number of iterations unbounded.Solution
Add
MaxPullEntries = 2 * MaxPullFilesand check it immediately after each successfultar.Reader.Next, before skipping or extracting the entry. Keep the existing regular-file and size limits. The 2,001st returned entry is rejected before it is processed.No linked issue.
Type of Change
Areas Affected
pkg/flowfiles)Testing and Verification
Test Configuration
843dfeb4bfc809c09d1bc5022e567c602d8fc9ddTest Steps
cd backend && go test ./pkg/flowfiles -count=1go vet ./pkg/flowfilesgo test ./pkg/flowfiles -run TestFiles_ExtractTar_BoundsAllEntries -count=1.Test Results
Package tests and vet pass. The regression check fails in all three new subtests (directories, symlinks, rejected paths) because the previous behavior accepts an archive over the entry limit. Restoring the guard makes the package pass again.
The new tests accept exactly 2,000 entries, reject 2,001, and verify that the excess entry is not created. Existing traversal, symlink, regular-file-count and content tests continue to pass.
Security Considerations
This bounds a previously unbounded dimension of container TAR processing. Pulling files requires authentication, access to the flow, and
containers.vieworcontainers.admin; this is not demonstrated as an unauthenticated attack. No new dependencies or permission changes. The existing service path closes the TAR stream and removes the staging directory on extraction errors.This is a returned-entry limit, not a general byte/time limit for TAR parsing or concurrent requests. End-to-end service behavior and host exhaustion were not tested.
Performance Impact
One counter increment and comparison per returned entry; stops processing once the limit is exceeded. No benchmark performed.
Documentation Updates
The limit is documented beside the existing pull limits and at the check. No API schema or environment variable changes.
Deployment Notes
No migrations. Archives with more than 2,000 returned entries now fail, even if they contain fewer than 1,000 regular files. The 1,000-file and 2 GiB limits still apply independently. The new limit intentionally also counts skipped links and rejected paths.
Checklist
gofmtgo vetpassAdditional Notes
Prepared with AI assistance; validation was run locally against the affected package.