Skip to content

Add basic auditor / pipeline support for static libraries - #96

Open
topolarity wants to merge 4 commits into
mainfrom
ct/static-library-auditing
Open

topolarity wants to merge 4 commits into
mainfrom
ct/static-library-auditing

Conversation

@topolarity

@topolarity topolarity commented Sep 30, 2026 •

Copy link
Copy Markdown
Member

Add basic audit support for StaticLibraryProduct, as introduced in #93.

The auditor doesn't have much to do here, since static libraries do not track dependency metadata or SONAME, etc. However we do need to possibly copy dependency information from the dynamic variant of this library and/or verify that the user's explicit-provided deps / static_deps information is reasonable.

This is only the preliminary support for now - just enough to successfully create static library products in a JLL.

Another follow-up will add some "proper" audit support to verify that linking with the provided deps / static_deps is possible / successful.


Stack created with GitHub Stacks CLI • Give Feedback 💬

@topolarity
topolarity added this pull request to stack #95 September 30, 2026 21:28
@topolarity
topolarity requested a review from staticfloat October 1, 2026 14:57
@topolarity
topolarity force-pushed the ct/static-library-auditing branch from 9aedcbc to d88da44 Compare October 2, 2026 02:38
@topolarity
topolarity force-pushed the ct/static-library-auditing branch from d88da44 to 7ac73e7 Compare October 6, 2026 21:45
@staticfloat
staticfloat force-pushed the ct/static-library-auditing branch from 7ac73e7 to dee7bb7 Compare October 7, 2026 03:12
Base automatically changed from ct/static-library-types to main October 7, 2026 04:32
@staticfloat
staticfloat force-pushed the ct/static-library-auditing branch from dee7bb7 to 145b121 Compare October 7, 2026 04:32

@staticfloat staticfloat left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Honestly way less accounting than I was expecting, especially the parts where we unwrap the dynamic libraries to get the static ones. Nice work, it feels quite natural.

Comment thread BinaryBuilderAuditor.jl/src/AuditInfo.jl Outdated
end

# Locate all static libraries
static_product_map = Dict{String,StaticLibraryProduct}()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The intended flow in scan_files() is to actually determine from file contents the types of files, so I would populate a list of static_libraries by looking at files with something like this:

function is_static_archive(path)
    open(path) do io
        magic = read(io, 8)
        magic == b"!<arch>\n" && return true
        # macOS universal archive: fat header, then an ar archive at the first slice's offset
        if length(magic) == 8 && ntoh(reinterpret(UInt32, magic[1:4])[1]) == 0xcafebabe
            skip(io, 8)
            seek(io, ntoh(read(io, UInt32)))
            return read(io, 8) == b"!<arch>\n"
        end
        return false
    end
end

in order to build up an exhaustive list of static libraries, and then from that construct static_product_map. This better matches the behavior of the other lists of files/products.

@topolarity topolarity Oct 8, 2026 •

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. I've pushed a rework, dependent on JuliaIO/ObjectFile.jl#63 and JuliaIO/ObjectFile.jl#59

# user provided explicitly, or (2) inherit the info from the dynamic audit.
#
# TODO: A proper audit would also verify that undefined symbols can be resolved as expected
# by these dependencies.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is gonna be pretty messy, because unlike the dynamic counterpart, we have no way to only check the first-order dependencies (e.g. are all DT_NEEDED's satisfied) with a static archive we'd have to actually have all recursive dependencies and check that all symbols are resolvable.

Tell me you want to write a linker without telling me you want to write a linker.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah. I suspect we'll probably use the actual linker to do this test link / audit, since that's the intended use case anyway. Luckily since we are linking statically we can do that on all targets, not just the host platform.

topolarity and others added 4 commits October 8, 2026 12:45
The auditor now locates every declared static archive in the scan prefix and
describes it as its own library entry.  Archives are not dynamic objects, so
they never enter `binary_objects` and remain exempt from the SONAME, DT_NEEDED
and RPATH passes.

An archive that shares its name with a library product is that library's static
form, and its dependency edges and system dependencies default to those of its
dynamic variant, which `resolve_dynamic_links!` has just derived from the shared
library's own linkage: the two are built from the same objects.  An explicit
list on a declared product replaces that set; a replacement that drops entries
the dynamic variant has is pointed out in the log, as a message rather than an audit
result, since the recipe author may well mean it.  An archive with no dynamic
variant is standalone, declares everything itself, and fails the audit if it
asks to inherit.  A declared edge is a recipe-side `LibraryDependency`, naming
the JLL its library belongs to or none for a library of this JLL, which is
exactly what the record's `JLLLibraryDep` says, so the pass records it as
declared.  Such an edge must name a library that exists, though: one of this
build's own, or one that a dependency of this build provides, which the pass
looks up in the `AuditInfo` it is now given.  Anything else fails the audit,
since nothing downstream could ever satisfy it.  Nothing is read from the
archives: the record states what the recipe declared and what the dynamic
variant was observed to link against.

A build's products are now a mix of kinds, so the vector carrying them widens
from library products to products in general, and the passes that only concern
loadable libraries pass over the archives of a dependency.

This commit was written with the assistance of generative AI (Claude).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Archives declared on a build are handed to the auditor, checked along with every
other product for being locatable, and folded into the extraction spec hash, so
that changing an archive's path or its declared dependencies invalidates the
build cache.  A subordinate archive would otherwise be invisible to all three,
since it never appears in the product list in its own right.  Each product
enters that hash as the recipe declares it, without any of the resolution the
auditor later does: an archive, subordinate or standalone, contributes its
fields as given, including its declared dependencies and system dependencies,
while a product without one hashes exactly as before, so existing cache entries
remain valid.

The vectors and caches that carry a build's JLL products widen from library
products to products in general, since an extraction's products are now a mix
of kinds, and a dependency's archives are handed to the auditor alongside its
shared libraries.

This commit was written with the assistance of generative AI (Claude).

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
As `scan_files()` does for binary objects, determine which files are
static archives from their contents (with ObjectFile's `ArchiveHandle`),
rather than by their names, into `ScanResult.static_libraries`.  Each
declared static library must then be one of these archives.

GNU thin archives are not counted, as they only refer to their members
by path, and so cannot be shipped.

This needs a release of ObjectFile.jl with `ArchiveHandle`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Point ObjectFile at a branch with `ArchiveHandle` (and our fixes for fat
Mach-O files) until it is released.  Drop this commit once it is, and
bump the ObjectFile compat instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@topolarity
topolarity force-pushed the ct/static-library-auditing branch from 145b121 to 09255a1 Compare October 8, 2026 16:47
@topolarity
topolarity requested a review from staticfloat October 8, 2026 16:48

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants