The requirements audit and threat modeler read a YAML requirements catalog. scripts/requirements/harvest_requirements.py can build that catalog from Confluence, Antora, or other HTML pages. It can also render selected functional sources as one OpenSpec Markdown file, one SpecDD .sdd file, or both.
flowchart LR
A["Your requirements pages<br/>(Confluence, Antora, wiki…)"] --> B["requirements/harvest_requirements.py<br/><i>crawls + parses</i>"]
B --> C[("appsec-requirements.yaml")]
B --> I[("application-requirements.openspec.md")]
B --> J[("application-requirements.sdd")]
C --> D{"Where does the plugin<br/>read it from?"}
D -->|"Committed in the repo"| E["raw.githubusercontent.com/…"]
D -->|"Hosted separately"| F["S3 / GitLab raw / CDN"]
D -->|"Local test loop"| G["mock-server.py on<br/>127.0.0.1:4444"]
E --> H["audit-security-requirements<br/>appsec-analyzer --requirements"]
F --> H
G --> H
The script crawls configured pages and writes the catalog. Publish that file at an HTTP URL, commit it as docs/security/requirements.yaml, or include it in an org profile.
The bundled mock server lets you test the audit before connecting an internal catalog. Its requirement text is a curated baseline informed by OWASP ASVS 5.0.0, the OWASP Top 10:2025, and the OWASP Cheat Sheets, rather than an official OWASP standard or a claim of complete coverage.
Its links model an organization's own portal at appsec.int.example.com, with a requirement page per category, each requirement anchored on that page, and a page per blueprint. That is the shape requirements/harvest_requirements.py writes from your own sources. The links are placeholders and do not resolve.
# Serve the bundled example requirements YAML on 127.0.0.1:4444
python3 scripts/mock-server.py
# In a second shell: point the plugin at the mock and run the auditor
/appsec-advisor:audit-security-requirements --requirements http://127.0.0.1:4444/requirements.yamlThe command prints FAIL and PARTIAL results with file and line evidence. Other statuses appear in the summary count.
If you do not have pages to crawl, start from data/appsec-requirements-fallback.yaml. Edit the IDs and text for your organization, then publish the file at a raw HTTP URL:
// skills/audit-security-requirements/config.json
{
"requirements_source": {
"enabled": true,
"requirements_yaml_url": "https://raw.githubusercontent.com/your-org/appsec-advisor/main/data/appsec-requirements-fallback.yaml"
}
}This is sufficient when the catalog changes infrequently.
Configure the harvester for the source pages and run it manually or on a CI schedule:
# Copy the template config
cp scripts/harvest-config.example.json scripts/harvest-config.json
# Set the requirements and blueprint page URLs
$EDITOR scripts/harvest-config.json
# Install deps once
pip install -r scripts/requirements.txt
# Dry-run first to verify reachability and parsing
python3 scripts/requirements/harvest_requirements.py --dry-run --verbose
# Write every output any source's `outputs` field names
HARVEST_AUTH_TOKEN=<token> python3 scripts/requirements/harvest_requirements.py
# Or restrict a run to just the catalog, e.g. for a CI job that only publishes it
HARVEST_AUTH_TOKEN=<token> python3 scripts/requirements/harvest_requirements.py --format yamlThe output setting controls the catalog destination and defaults to data/appsec-requirements-fallback.yaml. The openspec.output and specdd.output settings control the optional functional-spec files. The sources_meta block records the source page for each catalog section.
The output follows schemas/requirements-catalog.schema.yaml and is validated before the command exits. Invalid structure returns exit code 2; incomplete fields and duplicate IDs produce warnings. Validate a catalog manually with:
python3 scripts/requirements/requirements_state.py --validate data/appsec-requirements-fallback.yaml [--strict]Every written file opens with two comment lines naming the document and the page it was harvested from. A blueprint section keeps one line per paragraph, list item, and table row of the source page, so the section reads like the page it came from; a table row keeps its cells apart with an em dash, and a definition stays on the line of its term. Short lists such as topics and references stay on one line.
OpenSpec and SpecDD exports are single files. The renderer checks their required section and scenario structure without adding either project's CLI as a runtime dependency. The repository carries matching examples at examples/appsec-requirements-example.openspec.md and examples/appsec-requirements-example.sdd; their requirement statements are tested against the bundled YAML, while their concrete scenarios remain illustrative.
Existing sources default to outputs: ["catalog"]. Add openspec or specdd only when a source describes observable application behavior. Authentication, authorization, request results, error responses, and other behavior a test can observe belong in a functional spec. Implementation prescriptions such as a hashing algorithm, framework choice, parameterized-query API, or SAST setup remain catalog-only.
Selection is explicit rather than based on requirement wording. This keeps a secure-coding rule that happens to mention authentication from becoming product behavior. Only mandatory MUST requirements are emitted because both formats use their requirement sections as hard contracts; skipped SHOULD and MAY IDs are reported on stderr.
A plain run writes the catalog and both functional-spec files, since the config above declares sources for all three:
python3 scripts/requirements/harvest_requirements.pyPass --format to narrow a run to specific outputs, for example to test one format on its own:
python3 scripts/requirements/harvest_requirements.py --format openspecThe harvester never derives output paths, SpecDD ownership, or SpecDD modification permissions from crawled text. A generated .sdd file contains behavior and scenarios only; an operator decides where it belongs and what code it governs.
A run writes one catalog holding the requirements and every blueprint. A blueprint source can additionally write its own blueprints into a file of its own, which keeps a blueprint reviewable and diffable by itself:
{
"output": "../data/appsec-requirements.yaml",
"sources": [
{
"id": "spa-blueprint",
"type": "blueprint",
"crawl_url": "https://appsec.int.example.com/blueprints/spa",
"catalog_file": "../data/blueprints/spa.yaml"
},
{
"id": "api-blueprint",
"type": "blueprint",
"crawl_url": "https://appsec.int.example.com/blueprints/api",
"catalog_file": "../data/blueprints/api.yaml"
}
]
}catalog_file is resolved next to the config file and belongs to blueprint sources only. Requirements have output and the functional-spec exports.
Each file is a catalog in its own right: an empty categories list and the blueprints of that one source. It validates against the same schema and can be passed to --requirements alone. The main catalog keeps all blueprints, so nothing an existing consumer reads changes.
A source indexes its crawl_url and the child pages below it, so a source pointed at an index page writes every blueprint under it into one file. Point a source at a single blueprint page to get one file per blueprint.
The run fails before crawling when two sources claim the same file, or when a file would overwrite the catalog or a functional-spec export. A source that harvests nothing leaves its file untouched, so a failed crawl does not replace the last good one with an empty file.
examples/blueprints/api-hardening.yaml and examples/blueprints/llm-security.yaml show that shape, and examples/appsec-requirements-example.requirements.yaml shows a catalog that carries only requirements. All three are split from the combined examples/appsec-requirements-example.yaml and each is accepted by --requirements on its own.
The crawler reads scripts/harvest-config.json. This is the minimum useful configuration:
{
"description": "ACME Corp AppSec requirements",
"url": "https://appsec.int.example.com",
"output": "../data/appsec-requirements-fallback.yaml",
"request": {
"timeout_seconds": 15,
"auth_header_env": "HARVEST_AUTH_TOKEN",
"verify_ssl": true
},
"sources": [
{
"id": "internal-requirements",
"type": "requirement",
"mode": "structured",
"title": "Internal Security Requirements",
"crawl_url": "https://appsec.int.example.com/requirements"
},
{
"id": "api-blueprints",
"type": "blueprint",
"mode": "full",
"title": "API Security Blueprints",
"crawl_url": "https://appsec.int.example.com/blueprints/api"
}
]
}The harvester recognizes IDs such as SEC-AUTH-01, SCG-HARDENXML, OWASP-A01, and ISO27K-A12. It does not require a specific prefix.
For each source, the harvester reads crawl_url and direct same-origin child links below that path, up to max_pages. Add another sources[] entry for pages outside that path or deeper in the navigation tree.
When a blueprint mentions a harvested requirement ID, the output links the blueprint to that requirement.
| Flag | When you'd use it |
|---|---|
--dry-run --verbose |
Inspect parsing without writing a catalog |
--req-only / --blueprint-only |
Debug one source type at a time |
--config PATH |
Multiple environments (e.g. staging vs. prod requirements) |
--output PATH |
Override the config's output; useful in CI |
--format yaml|openspec|specdd|all |
Narrow a run to specific outputs; without it, every format any source's outputs names is written |
--openspec-output PATH |
Override openspec.output for one run |
--specdd-output PATH |
Override specdd.output for one run |
--token TOKEN |
Pass a bearer token directly; prefer auth_header_env in CI |
See scripts/harvest-config.example.json for all fields.
The harvester runs once and exits. Schedule it with cron or CI if the catalog should update automatically. This GitHub Actions example writes and commits changed output:
# .github/workflows/harvest-requirements.yml
name: Harvest Security Requirements
on:
schedule: [{ cron: '0 2 * * *' }] # nightly at 02:00 UTC
workflow_dispatch: # manual trigger too
permissions:
contents: write
jobs:
harvest:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with: { python-version: '3.11' }
- run: pip install -r scripts/requirements.txt
- env: { HARVEST_AUTH_TOKEN: "${{ secrets.HARVEST_AUTH_TOKEN }}" }
run: python3 scripts/requirements/harvest_requirements.py
- name: Commit if changed
run: |
if ! git diff --quiet data/appsec-requirements-fallback.yaml; then
git config user.email ci@github.com
git config user.name "GitHub Actions"
git commit -am "chore: refresh appsec requirements [harvester]"
git push
fiInstead of committing the file, the job can publish it to S3, a GitLab raw URL, or an internal CDN. Point requirements_yaml_url to that location.
Set requirements_yaml_url to use the catalog for both the threat modeler and requirements audit:
// skills/audit-security-requirements/config.json
{
"requirements_source": {
"enabled": true,
"requirements_yaml_url": "https://raw.githubusercontent.com/your-org/appsec-advisor/main/data/appsec-requirements-fallback.yaml"
}
}The plugin caches the fetched catalog. An explicit --requirements <url> overrides the config; --no-requirements disables it for one run.
Parser returns zero requirements. Run with --verbose to see every parser attempt per page. If all five strategies miss, either the ID shape doesn't match PREFIX-PART[-PART…] (e.g. pure numeric IDs like REQ_001) or the HTML is an SPA that needs JavaScript to render content (the harvester fetches static HTML only).
OpenSpec or SpecDD says no source targets the format. Add that format to the outputs array of the functional requirement source. Catalog-only remains the default so secure-coding guidance is never exported as application behavior implicitly. If you passed --format, check that a source declares that format in outputs. The flag only narrows the selected formats.
A configured blueprint page is missing from the YAML. The current harvester indexes the configured crawl_url itself and direct same-origin child links below that path. If a blueprint still does not appear, check the dry-run output for Found N sub-page link(s) and the blueprint count. Common causes are JavaScript-rendered content, links outside the configured base path, deeper nested pages that are not linked directly from crawl_url, or max_pages capping the discovered links before the page is reached. Fix by adding explicit sources[] entries for those pages or raising max_pages.
Auth token works interactively but fails in CI. Set HARVEST_AUTH_TOKEN as a CI secret and pass it through the job's env: block.
Mock server returns the bundled YAML after I ran the harvester. The mock intentionally hardcodes examples/appsec-requirements-example.yaml as its demo payload and does not serve a harvested production catalog. To test a generated file without overwriting the bundled example, serve its containing directory on loopback with python3 -m http.server 4445 --bind 127.0.0.1 --directory data and pass the resulting URL explicitly with --requirements.
--requirements on the CLI is ignored. The resolution order is: explicit --requirements <url> > config requirements_yaml_url (when enabled: true) > cache. If you passed --no-requirements earlier, it wins regardless.
ModuleNotFoundError: requirements_state outside this repository. The harvester is not a single file. Running it from another repository or a CI job needs scripts/requirements/requirements_state.py in the same directory as requirements/harvest_requirements.py; it holds the catalog schema validation that runs over the written output. The module is standard library only, so no extra install is needed for it. The schema itself is read from <script directory>/../schemas/requirements-catalog.schema.yaml; copy that file along to keep full schema validation, otherwise a reduced structural check runs instead. Full validation also needs jsonschema on top of scripts/requirements.txt.
{ "openspec": { "output": "../data/application-requirements.openspec.md", "title": "ACME Application Specification" }, "specdd": { "output": "../data/application-requirements.sdd", "name": "ACME Application Behavior" }, "sources": [ { "id": "application-behavior", "type": "requirement", "crawl_url": "https://product.int.example.com/requirements", "outputs": ["openspec", "specdd"] }, { "id": "secure-coding-guidelines", "type": "requirement", "crawl_url": "https://appsec.int.example.com/secure-coding", "outputs": ["catalog"] } ] }