| title | IDP CLI - Command Line Interface for Batch Document Processing |
|---|
Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. SPDX-License-Identifier: MIT-0
A command-line tool for batch document processing with the GenAI IDP Accelerator.
✨ Batch Processing - Process multiple documents from CSV/JSON manifests
📊 Live Progress Monitoring - Real-time updates with rich terminal UI
🔄 Resume Monitoring - Stop and resume monitoring without affecting processing
📁 Flexible Input - Support for local files and S3 references
🔍 Comprehensive Status - Track queued, running, completed, and failed documents
📈 Batch Analytics - Success rates, durations, and detailed error reporting
🎯 Evaluation Framework - Validate accuracy against baselines with detailed metrics
💬 Agent Chat - Interactive Agent Companion Chat from the terminal with Analytics, Error Analyzer, and more
Demo:
idp-cli.mp4
- Installation
- Quick Start
- Commands Reference
- deploy
- publish
- delete
- bootstrap
- process
- reprocess
- status
- download-results
- use-as-baseline
- delete-documents
- generate-manifest
- validate-manifest
- list-batches
- stop-workflows
- load-test
- tenancy
- terraform migrate-from-genai-idp-terraform
- discover
- discover-multidoc
- remove-deleted-stack-resources
- config-create
- config-validate
- config-download
- config-upload
- config-list
- config-revisions
- config-activate
- config-delete
- test-result
- abort-test-run
- test-compare
- chat
- Complete Evaluation Workflow
- Evaluation Analytics
- Manifest Format Reference
- Advanced Usage
- Troubleshooting
- Python 3.12 or higher
uvonPATH, whichmake setup-venvuses to read the dependency lock- AWS credentials configured (via AWS CLI or environment variables)
- An active IDP Accelerator CloudFormation stack
make setup-venv
source .venv/bin/activateRun this from the repository root. The CLI requires idp-sdk, which requires
idp_common; both names on public PyPI belong to unrelated parties, so all three
packages go in a single command and all three come from a path. See
Installing First-Party Packages Safely.
pip install -e lib/idp_common_pkg -e lib/idp_sdk -e "lib/idp_cli_pkg[test]"The root Makefile provides convenience wrappers for the most common idp-cli commands. These use the project's .venv Python automatically — no need to source .venv/bin/activate first.
# Publish artifacts to S3
make publish REGION=us-east-1
make publish REGION=us-east-1 BUCKET_BASENAME=my-artifacts PREFIX=v1
make publish REGION=us-gov-west-1 HEADLESS=1
# Deploy / update a stack (--wait is the default)
make deploy STACK_NAME=my-idp ADMIN_EMAIL=me@example.com
make deploy STACK_NAME=my-idp-dev ADMIN_EMAIL=me@example.com FROM_CODE=1
make deploy STACK_NAME=my-idp CUSTOM_CONFIG=./my-config.yaml
make deploy STACK_NAME=my-idp TAGS="Owner=docs-team,Environment=prod"
# Delete a stack
make delete-stack STACK_NAME=test-stack FORCE=1 FORCE_DELETE_ALL=1First-class Make variables (common flags):
| Variable | Target(s) | Maps to CLI flag |
|---|---|---|
REGION |
publish, deploy, delete-stack | --region |
STACK_NAME |
deploy, delete-stack | --stack-name |
ADMIN_EMAIL |
deploy | --admin-email |
FROM_CODE=1 |
deploy | --from-code . |
HEADLESS=1 |
publish, deploy | --headless |
PUBLIC=1 |
publish | --public |
BUCKET_BASENAME |
publish | --bucket-basename |
PREFIX |
publish | --prefix |
CUSTOM_CONFIG |
deploy | --custom-config |
TAGS |
deploy | --tags |
TEMPLATE_URL |
deploy | --template-url |
TEMPLATE_FILE |
deploy | --template-file |
NO_WAIT=1 |
deploy, delete-stack | omits --wait |
FORCE=1 |
delete-stack | --force |
EMPTY_BUCKETS=1 |
delete-stack | --empty-buckets |
FORCE_DELETE_ALL=1 |
delete-stack | --force-delete-all |
Uncommon flags — pass anything else via EXTRA_ARGS:
make deploy STACK_NAME=my-idp EXTRA_ARGS="--role-arn arn:aws:iam::123:role/Foo --no-rollback"
make publish REGION=us-east-1 EXTRA_ARGS="--clean-build --verbose"Run make help to see all available targets, or idp-cli <command> --help for the full option reference.
The CLI supports an optional --profile parameter to specify which AWS credentials profile to use:
idp-cli --profile my-profile <command> [options]- Can be placed anywhere in the command
- Only affects that specific command execution
- Automatically applies to all AWS SDK calls
- If not specified, uses default AWS credentials
Examples:
# Profile before command
idp-cli --profile production deploy --stack-name my-stack ...
# Profile after command
idp-cli deploy --profile production --stack-name my-stack ...
# Profile at the end
idp-cli deploy --stack-name my-stack --profile production ...--region is a per-command option, not a global one, so it goes after the
subcommand:
idp-cli config-upload --stack-name my-stack --config-file ./config.yaml \
--config-profile v2 --region eu-west-1The resolution order is:
--regionon the subcommand, if given.- Otherwise boto3's own chain:
AWS_REGION, thenAWS_DEFAULT_REGION, then theregionconfigured for the selected--profile(orAWS_PROFILE), then EC2 instance metadata.
Nothing substitutes a hardcoded region for the configuration commands, so a
command run with no --region and no region resolvable from the environment fails
with boto3's NoRegionError rather than guessing.
--region applies to every AWS call a command makes, not only to the
CloudFormation lookup that resolves a resource's name. That distinction is the
whole point: a stack's ConfigurationTable physical id is not region-qualified, so
a command that looked the name up in one region and then read or wrote it in
another would hit a different stack's table on a multi-region account — and
report success. It therefore covers
- the DynamoDB read and write of the Configuration Table,
- the S3 write of configuration revision history,
- the document classes
config-sync-bdaderives from a BDA project, and the BDA project calls themselves, - the schema and rules that
discoveranddiscover-multidocwrite back, - the model-limits read on
config-upload's validation path, which would otherwise fall back silently to the on-disk defaults and could reject a configuration that is legitimately above a default cap.
A whole-tree check (scripts/tests/test_config_region_threading.py) asserts that
no code outside a Lambda builds a configuration client without a region, so a new
command cannot reintroduce the gap.
Three commands take no --region because they make no AWS calls at all:
config-create, config-validate and validate-manifest.
Every payload the CLI writes to stdout for a program to read is written verbatim: no colour, no syntax highlighting, and no wrapping to the terminal width. On the commands below, stdout carries only that payload — progress and status lines go to stderr — so piping and redirecting are safe:
| Command | Payload on stdout |
|---|---|
config-revisions --json |
JSON revision history |
status --format json |
JSON status document |
config-download without --output |
configuration YAML |
config-create without --output |
configuration-template YAML |
bootstrap without --stack-name |
the authored JSON schema |
# Parse JSON directly
idp-cli config-revisions --stack-name my-stack --config-profile lending --json \
| jq -r '.revisions[] | select(.published) | .revision'
# Redirect YAML straight to a file
idp-cli config-download --stack-name my-stack > config.yaml
# Progress is on stderr, so discard it without touching the payload
idp-cli status --stack-name my-stack --batch-id batch-123 --format json 2>/dev/null \
| jq '.exit_code'discover and discover-multidoc are the exception. Their schemas are written
unrendered too, but they print a Discovered schemas: heading and per-document
progress to stdout alongside them, so use -o / --output to capture a
schema from those two rather than redirecting stdout:
idp-cli discover-multidoc --dir ./samples/ -o ./schemas/Human-facing output — tables, progress, status lines — is still styled when
stdout is a terminal, and Rich disables the styling itself when it is not. Before
v0.6.9 these payloads were rendered the same way as that human output, so
--json carried ANSI escape codes and a long line of downloaded YAML was folded
to the console width (#905).
# 1. Deploy stack (10-15 minutes)
idp-cli deploy \
--stack-name my-idp-stack \
--admin-email your.email@example.com \
--wait
# 2. Process documents from a local directory
idp-cli process \
--stack-name my-idp-stack \
--dir ./my-documents/ \
--monitor
# 3. Download results
idp-cli download-results \
--stack-name my-idp-stack \
--batch-id <batch-id-from-step-2> \
--output-dir ./results/That's it! Your documents are processed with OCR, classification, extraction, assessment, and summarization.
For evaluation workflows with accuracy metrics, see the Complete Evaluation Workflow section.
Flag naming: the named configuration entity is a Configuration Profile, selected with
--config-profile. The former name,--config-version, is kept as an accepted alias so existing scripts keep working — both spellings set the same value, and there is no plan to remove the old one. New scripts should use--config-profile.--config-revisionis unrelated: it selects a revision within a profile. See configuration-profiles.md.
Comma-separated options: every option documented below as comma-separated (
--document-ids,--test-run-ids,--file-types,--check-stack-regions,--features,--tags) drops blank segments, so a trailing or doubled comma is harmless —--document-ids "a,b,"names two documents, not three. All but--featuresparse through one shared helper;--featuresis parsed by the SDK'sconfig.create, by the same rule.For all of them except
--tags, a value containing no non-blank segment is refused with exit 1 rather than treated as one value that is the empty string. That matters most for a list a script built from a variable that turned out to be empty:--document-ids ""used to ask about a document whose S3 object key was"", and--document-ids ","used to announce "Selected 2 document(s) for deletion".--tags ","is not refused — it sets no tags and proceeds, which is exactly what omitting--tagsdoes, so there is no wrong action for a refusal to prevent.
Deploy or update an IDP CloudFormation stack.
Usage:
idp-cli deploy [OPTIONS]Required for New Stacks:
--stack-name: CloudFormation stack name--admin-email: Admin user email
Optional Parameters:
--from-code: Deploy from local code by building and publishing artifacts (path to project root). With--terraform, it also publishes the Terraform module from that checkout and deploys it--template-url: URL to CloudFormation template in S3 (optional, auto-selected based on region)--template-file: Local path to a pre-built CloudFormation template (from a previouspublish)--custom-config: Path to local config file or S3 URI--max-concurrent: Maximum concurrent workflows (default: 100)--log-level: Logging level (DEBUG,INFO,WARN,ERROR). No CLI default: omit it to take the template default (WARN) on a new stack, or to keep an existing stack's current value on an update.INFOandDEBUGcan write presigned URLs, document contents and PII to CloudWatch — see Monitoring--enable-hitl: Deprecated and refused iftrue. HITL is a configuration setting rather than a stack parameter (theEnableHITLparameter was removed in v0.4.11) — enable it in the Web UI under Configuration → Human-in-the-Loop (HITL) Review (Optional), or sethitl.enabledin the config YAML passed to--custom-config. The flag is still accepted asfalseso existing scripts keep working.--parameters: Additional CloudFormation parameters askey=value,key2=value2. A new pair starts at a comma or whitespace followed bykey=(so a space-separated list, asaws cloudformation deploy --parameter-overridestakes, also works), and everything up to the next pair is one value — so a value may itself contain commas (SubnetIds=subnet-a,subnet-b) and=signs (a metadata URL with a query string, a base64 value). Whitespace around the=is ignored. Two things it cannot read as a pair are printed back to you rather than passing unremarked: text before the first pair, which is named and not submitted, and a value that looks like it swallowed a pair — a key holding a character CloudFormation does not allow, or pairs separated with;,|or a stray backslash — which is named together with the parameter it landed in, since a value may contain commas and so cannot be split back apart. The reason for the noise is that a parameter which never reached CloudFormation is indistinguishable afterwards from one deliberately left at its default. Pairs separated with&or?are the one case read silently as a value, because that is exactly what a query string looks like.--tags: Stack tags askey=value,key2=value2. CloudFormation applies these to the stack and propagates them to all taggable resources and nested stacks — useful for governance/ownership (e.g.Owner,Team,Environment). See Resource tagging below.--wait: Wait for stack operation to complete--no-rollback: Disable rollback on stack creation failure--region: AWS region (optional, auto-detected)--role-arn: CloudFormation service role ARN (optional)--headless: Deploy a headless (no-UI) stack — removes CloudFront, the UI REST API (theAPIRESOLVERSTACKnested stack holding the API Gateway REST API, its dispatcher, and the UI-only resolver Lambdas), Cognito, WAF, agents, HITL, and Test Studio. Required for GovCloud; also valid in Commercial regions for API-only / pipeline integrations. See Headless Deployment.--govcloud: Deploy the GovCloud template variant — keeps the full Web UI but removes everyAWS::CloudFront::*resource (CloudFront does not exist in GovCloud) and forces API Gateway UI hosting. Mutually exclusive with--headless. If the GovCloud template cannot be produced, the deploy is refused rather than falling back to the commercial template: that template's CloudFront resources cannot exist in a GovCloud partition, so deploying it fails part-way through CREATE on a resource that looks unrelated to the flag. The error names the template that is missing (.aws-sam/idp-govcloud.yaml) and theidp-cli publish --govcloudcommand that produces it. See GovCloud Deployment.--bucket-basename: S3 bucket basename for build artifacts (used with--from-code; region is appended automatically)--prefix: S3 key prefix for build artifacts (default:idp-cli, used with--from-code)--public: Make published S3 artifacts publicly readable (used with--from-code)--build-max-workers: Maximum concurrent build workers (used with--from-code)--clean-build: Force full rebuild by deleting checksum files (used with--from-code)--no-validate-template: Skip CloudFormation template validation--terraform(experimental): Deploy with Terraform instead of CloudFormation. See Terraform deployments below and the Terraform Deployment guide.--terraform-module-url: Terraform module source URL (used with--terraform; default: the latest published module for the region; not with--from-code, which publishes its own)--terraform-dir: Directory for the generated root configuration and local state (used with--terraform; default:./idp-terraform/<stack-name>)--terraform-bin:terraformortofu(used with--terraform)--terraform-assume-role-arn: IAM role both Terraform providers assume (used with--terraform; refused without it)
Note: --from-code, --template-url, and --template-file are mutually exclusive. Use --from-code for development/testing from local source, --template-url for production deployments from a pre-published template, and --template-file to deploy a locally-built template.
Headless mode:
--headlesscan be combined with--from-code .(builds both variants and deploys the headless one) or used standalone against a pre-published template (the CLI downloads, transforms to headless, uploads to a temporary S3 location in your account, and deploys).- When
--headlessis used without--from-code, the region must have a pre-publishedidp-main.yaml. For GovCloud (us-gov-*) or any unsupported region, add--from-code .. - GovCloud regions are auto-detected and receive GovCloud-appropriate configuration defaults (ARN partition fixes, GovCloud Bedrock models,
lending-package-sample-govcloudconfiguration preset). - See the Headless Deployment Guide and GovCloud Deployment Guide for details.
Auto-Monitoring for In-Progress Operations:
If you run deploy on a stack that already has an operation in progress (CREATE, UPDATE, ROLLBACK), the command automatically switches to monitoring mode instead of failing. This is useful if you forgot to use --wait on the initial deploy - simply run the same command again to monitor progress:
# First run without --wait starts the deployment
$ idp-cli deploy --stack-name my-stack --admin-email user@example.com
✓ Stack CREATE initiated successfully!
# Second run - automatically monitors the in-progress operation
$ idp-cli deploy --stack-name my-stack
Stack 'my-stack' has an operation in progress
Current status: CREATE_IN_PROGRESS
Switching to monitoring mode...
[Live progress display...]
✓ Stack CREATE completed successfully!Supported in-progress states: CREATE_IN_PROGRESS, UPDATE_IN_PROGRESS, DELETE_IN_PROGRESS, ROLLBACK_IN_PROGRESS, UPDATE_ROLLBACK_IN_PROGRESS, and cleanup states.
Examples:
# Create new stack
idp-cli deploy \
--stack-name my-idp \
--admin-email user@example.com \
--wait
# Update with custom config
idp-cli deploy \
--stack-name my-idp \
--custom-config ./updated-config.yaml \
--wait
# Update parameters
idp-cli deploy \
--stack-name my-idp \
--max-concurrent 200 \
--log-level DEBUG \
--wait
# Deploy with governance/ownership tags (propagated to all resources)
idp-cli deploy \
--stack-name my-idp \
--admin-email user@example.com \
--tags "Owner=docs-team,Team=idp,Environment=prod" \
--wait
# Deploy with custom template URL (for regions not auto-supported)
idp-cli deploy \
--stack-name my-idp \
--admin-email user@example.com \
--template-url https://s3.eu-west-1.amazonaws.com/my-bucket/idp-main.yaml \
--region eu-west-1 \
--wait
# Deploy with CloudFormation service role and permissions boundary.
# If you use the example service role from iam-roles/cloudformation-management/,
# the stack name must start with its ManagedStackNamePrefix (default `idp`) and
# PermissionsBoundaryArn must match the boundary that role requires — role
# creation is denied otherwise. Both values are stack outputs of that role.
idp-cli deploy \
--stack-name idp-demo \
--admin-email user@example.com \
--role-arn arn:aws:iam::123456789012:role/idp-service-role-CFServiceRole \
--parameters "PermissionsBoundaryArn=arn:aws:iam::123456789012:policy/MyPermissionsBoundary" \
--wait
# Deploy from local source code (for development/testing)
idp-cli deploy \
--stack-name my-idp-dev \
--from-code . \
--admin-email user@example.com \
--wait
# Update existing stack from local code changes
idp-cli deploy \
--stack-name my-idp-dev \
--from-code . \
--wait
# Deploy with rollback disabled (useful for debugging failed deployments)
idp-cli deploy \
--stack-name my-idp \
--admin-email user@example.com \
--no-rollback \
--wait
# Deploy a HEADLESS stack in a commercial region (no UI — API-only)
# The CLI downloads the published template, transforms it to headless,
# uploads to a temporary S3 location, and deploys.
idp-cli deploy \
--stack-name my-idp-headless \
--region us-east-1 \
--headless \
--wait
# Deploy a HEADLESS stack from local source (development iteration)
idp-cli deploy \
--stack-name my-idp-headless-dev \
--region us-east-1 \
--from-code . \
--headless \
--wait
# Deploy to GovCloud WITH the full Web UI (--govcloud; --from-code is required)
idp-cli deploy \
--stack-name my-idp-govcloud \
--region us-gov-west-1 \
--from-code . \
--govcloud \
--admin-email user@example.com \
--wait
# Deploy to GovCloud headless (no UI; --from-code is required)
idp-cli deploy \
--stack-name my-idp-govcloud \
--region us-gov-west-1 \
--from-code . \
--headless \
--waitHeadless? See the Headless Deployment Guide for when to use it (not just GovCloud — also API-only / pipeline integrations in Commercial regions) and the GovCloud Deployment Guide for GovCloud-specific considerations.
--tags "key=value,key2=value2" applies CloudFormation stack-level tags. CloudFormation adds them to the stack and automatically propagates them to all taggable resources it creates — including the nested stacks (pattern, API resolvers, KB, discovery, feature platform) and their resources — so you tag the whole deployment in one place.
Notes and caveats:
- Format: comma-separated
key=valuepairs. Tag keys may contain spaces and the characters. : / + - _(a value may itself contain=; only the first=splits key from value). Commas inside a tag value are not supported. - Update behavior: re-running
deploywith--tagsreplaces the stack's entire tag set with what you pass. Omitting--tagson an update preserves the existing tags (unlike a bare AWS API call, which would clear them). - Not every resource type accepts propagated tags. CloudFormation propagation is best-effort — a small number of resource types (e.g. some Cognito, CloudFront, and custom resources) do not receive stack tags. This is an AWS platform limitation, not a configuration option.
- Cost allocation: to use these tags in AWS Cost Explorer / cost allocation reports you must additionally activate them as cost allocation tags in the Billing console (a one-time, account-level step); this CLI option does not do that for you.
idp-cli deploy \
--stack-name my-idp \
--admin-email user@example.com \
--tags "Owner=docs-team,Team=idp,Environment=prod" \
--waitdeploy --terraform deploys the experimental Terraform module instead of a CloudFormation stack. Its options, and how it writes and reuses the Terraform directory, are under Terraform deployments below; parameters, the label stack and upgrading are in the Terraform Deployment guide.
Build IDP CloudFormation artifacts locally and publish them to S3. Produces a template_url and a 1-click CloudFormation launch URL. Optionally also produces a headless (no-UI) template variant.
Use publish when you want to build and stage artifacts without deploying immediately — for example, to share the template with other accounts, keep a known-good build, or run a separate deploy step later.
Usage:
idp-cli publish [OPTIONS]Options:
--source-dir(default:.): Path to the IDP project root directory--region(required): AWS region where artifacts will be uploaded and deployed--bucket-basename: S3 bucket basename for artifacts (region is appended automatically; auto-generated if not provided)--prefix: S3 key prefix for artifacts (default:idp-cli)--headless: Also generate a headless (no-UI) template variant. For commercial regions this producesidp-main.yamlandidp-headless.yaml; for GovCloud (us-gov-*) the headless template is additionally updated with GovCloud configuration defaults (ARN partition, GovCloud Bedrock models,lending-package-sample-govcloudpreset).--govcloud: Also generate the GovCloud template variant — the full Web UI with everyAWS::CloudFront::*resource removed and API Gateway UI hosting forced. Writes.aws-sam/idp-govcloud.yamlbesideidp-main.yamland uploads it asidp-govcloud.yaml. The transform is linted against a GovCloud region, so an unsupported resource type that survived it fails the publish with thecfn-lintfinding rather than at deploy time. Deploy the result withidp-cli deploy --template-file .aws-sam/idp-govcloud.yaml, or build and deploy in one step withidp-cli deploy --from-code . --govcloud.--public: Make S3 artifacts publicly readable (for shared deployments)--terraform(experimental): After the CloudFormation publish, also generate the Terraform module from the published main template and upload it beside the templates:<prefix>/<version>/terraform/idp-terraform-<version>.zip, its.zip.sha256checksum, and<prefix>/terraform/idp-terraform-latest.json({"version", "moduleUrl", "sha256"}). The command prints the module source URL, an HTTPS S3 URL ending.zipthat Terraform accepts as a modulesource. See Terraform Deployment. It runs the translator interraform/translatorof the source checkout, which needs the SDK'sterraformextra (pip install -e "lib/idp_sdk[terraform]";make setupinstalls it).--max-workers: Maximum concurrent build workers (default: auto-detect)--clean-build: Force full rebuild by deleting all checksum files--no-validate: Skip CloudFormation template validation--lint / --no-lint: Enable/disable ruff linting and cfn-lint (default: enabled)--verbose,-v: Enable verbose build output--terraform: After the CloudFormation publish, translate the published main template into a Terraform module (nested templates are fetched by theirTemplateURL) and upload it to the same bucket and prefix:<prefix>/<version>/terraform/idp-terraform-<version>.zip, its.zip.sha256, and the pointer<prefix>/terraform/idp-terraform-latest.json({"version", "moduleUrl", "sha256"}). The zip is deterministic, so republishing unchanged files gives the same checksum. Prints the module's HTTPS URL. With--headlessand/or--govcloudas well, each variant template is also translated and published as its own module, namedidp-terraform-<variant>-<version>.zip(plus.sha256) with the pointer<prefix>/terraform/idp-terraform-<variant>-latest.json; if a variant's translation fails, the publish fails and names the variant. Needs the source checkout'sterraform/translator. See Terraform deployments.
Prerequisites (same as --from-code deployments):
- AWS SAM CLI, Docker (for container-image Lambdas), Node.js ≥ 22.12, npm ≥ 10, Python 3.12.
Examples:
# Standard build and publish (UI + backend)
idp-cli publish --source-dir . --region us-east-1
# With custom bucket and prefix
idp-cli publish \
--source-dir . \
--region us-east-1 \
--bucket-basename my-idp-artifacts \
--prefix v1
# Build BOTH standard and headless templates
idp-cli publish --source-dir . --region us-east-1 --headless
# Build a headless template for GovCloud (GovCloud-specific config applied automatically)
idp-cli publish --source-dir . --region us-gov-west-1 --headless
# Full rebuild with verbose output
idp-cli publish --source-dir . --region us-east-1 --clean-build --verbose
# Make artifacts publicly readable (shared template)
idp-cli publish --source-dir . --region us-east-1 --public
# Also generate and upload the Terraform module (experimental)
idp-cli publish --source-dir . --region us-east-1 --terraformOutput:
On success, publish prints:
📦 Template URL (for updating existing stack):
https://s3.us-east-1.amazonaws.com/<bucket>/<prefix>/idp-main.yaml
🚀 1-Click Launch (creates new stack):
https://us-east-1.console.aws.amazon.com/cloudformation/home?...
🔧 Headless Template URL: # only with --headless
https://s3.us-east-1.amazonaws.com/<bucket>/<prefix>/idp-headless.yaml
🚀 Headless 1-Click Launch: # only with --headless
https://us-east-1.console.aws.amazon.com/cloudformation/home?...
Relationship to deploy --from-code:
deploy --from-code . internally runs the same build + publish pipeline and then creates/updates the CloudFormation stack. Use publish when you want to decouple the build from the deployment step or share the template with other accounts/regions.
Legacy: The standalone
publish.pyscript andscripts/generate_govcloud_template.pyare deprecated. Useidp-cli publish(with or without--headless) instead.
Delete an IDP CloudFormation stack.
Usage:
idp-cli delete [OPTIONS]Options:
--stack-name(required): CloudFormation stack name--force: Skip confirmation prompt--empty-buckets: Empty S3 buckets before deletion (required if buckets contain data)--force-delete-all: Force delete ALL remaining resources after CloudFormation deletion (S3 buckets, CloudWatch logs, DynamoDB tables)--wait: Wait for deletion to complete (default: no-wait)--region: AWS region (optional)--terraform(experimental): Delete a Terraform deployment made withdeploy --terraform, from its--terraform-dir. See Deleting a Terraform deployment below.--terraform-dir: The directorydeploy --terraformwrote (used with--terraform)--terraform-assume-role-arn: IAM role both Terraform providers assume during the delete (used with--terraform)
S3 Bucket Behavior:
- LoggingBucket:
DeletionPolicy: Retain- Always kept (unless using--force-delete-all) - All other buckets:
DeletionPolicy: RetainExceptOnCreate- Deleted if empty - CloudFormation can ONLY delete S3 buckets if they're empty
- Use
--empty-bucketsto automatically empty buckets before deletion - Use
--force-delete-allto delete ALL remaining resources after CloudFormation completes
Force Delete All Behavior:
The --force-delete-all flag performs a comprehensive cleanup AFTER CloudFormation deletion completes:
- CloudFormation Deletion Phase: Standard stack deletion
- Additional Resource Cleanup Phase (happens with
--waiton all deletions and always with--force-delete-all): Removes stack-specific resources not tracked by CloudFormation:- CloudWatch Log Groups (Lambda functions, Glue crawlers)
- AppSync APIs and their log groups (only ever present in stacks created before AppSync was removed; current stacks create none)
- This stack's own CloudFront distributions, identified from the stack's resource list rather than by their comment, which carries only the stack name and so is shared by a same-named stack in another Region. A distribution has to be disabled, and finish deploying (15-20 minutes), before it can be deleted, so each pass moves it one step: an enabled one is disabled, a disabled and deployed one is deleted, and one still deploying is left with a message to re-run. Other stacks' distributions, including disabled orphans, are
remove-deleted-stack-resources's job - This stack's own CloudFront Response Headers Policy, identified from the stack's resource list rather than by name. No other stack's policy is touched: CloudFront is global, so the account's policy list includes those of IDP stacks in every Region, and sweeping orphans out of it is
remove-deleted-stack-resources's job. A policy is usually already gone, because CloudFormation deletes it with the stack; one left by a failed stack deletion cannot be deleted until its distribution has finished deleting (15-20 minutes after it is disabled), and the CLI logs that and says to re-run rather than reporting a failure - IAM custom policies and permissions boundaries
- CloudWatch Logs resource policies
- Retained Resource Cleanup Phase (only with
--force-delete-all): Deletes remaining resources in order:- DynamoDB tables (disables PITR, then deletes)
- CloudWatch Log Groups (matching stack name pattern)
- S3 buckets (regular buckets first, LoggingBucket last)
--force-delete-all. The
cleanup phase still runs — that is what the flag is for — and the non-zero exit comes
after it, so you get both. Before this, --force-delete-all printed "Stack deletion
failed!" and exited 0, so a CI teardown job could not tell a stack that failed to delete
from one that deleted cleanly.
Resources Always Cleaned Up (with --wait or --force-delete-all):
- IAM custom policies (containing stack name)
- IAM permissions boundary policies
- This stack's own CloudFront distributions, one step per pass: disabled if enabled, deleted once disabled and deployed
- This stack's own CloudFront response headers policy, once its distribution is gone
- CloudWatch Logs resource policies (stack-specific)
- AppSync log groups (pre-migration stacks only)
- Additional log groups containing stack name
- Gracefully handles missing/already-deleted resources
Resources Deleted Only by --force-delete-all:
- All DynamoDB tables from stack
- All CloudWatch Log Groups (retained by CloudFormation)
- All S3 buckets including LoggingBucket
- Handles nested stack resources automatically
Examples:
# Interactive deletion with confirmation
idp-cli delete --stack-name test-stack
# Automated deletion (CI/CD)
idp-cli delete --stack-name test-stack --force
# Delete with automatic bucket emptying
idp-cli delete --stack-name test-stack --empty-buckets --force
# Force delete ALL remaining resources (comprehensive cleanup)
idp-cli delete --stack-name test-stack --force-delete-all --force
# Wait for the deletion to reach DELETE_COMPLETE before returning
idp-cli delete --stack-name test-stack --force --wait
# Delete a Terraform deployment (experimental)
idp-cli delete --terraform --stack-name idp-tf --forceDeletion does not wait by default: without --wait the command initiates the
delete and returns, printing the console link and the --wait form to follow it with.
There is no --no-wait flag — omitting --wait is already the no-wait path.
What you'll see (standard deletion):
⚠️ WARNING: Stack Deletion
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Stack: test-stack
Region: us-east-1
S3 Buckets:
• InputBucket: 20 objects (45.3 MB)
• OutputBucket: 20 objects (123.7 MB)
• WorkingBucket: empty
⚠️ Buckets contain data!
This action cannot be undone.
Are you sure you want to delete this stack? [y/N]: _
What you'll see (force-delete-all):
⚠️ WARNING: FORCE DELETE ALL RESOURCES
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Stack: test-stack
Region: us-east-1
S3 Buckets:
• InputBucket: 20 objects (45.3 MB)
• OutputBucket: 20 objects (123.7 MB)
• LoggingBucket: 5000 objects (2.3 GB)
⚠️ FORCE DELETE ALL will remove:
• All S3 buckets (including LoggingBucket)
• All CloudWatch Log Groups
• All DynamoDB Tables
• Any other retained resources
This happens AFTER CloudFormation deletion completes
This action cannot be undone.
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Are you ABSOLUTELY sure you want to force delete ALL resources? [y/N]: y
Deleting CloudFormation stack...
✓ Stack deleted successfully!
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Starting force cleanup of retained resources...
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Analyzing retained resources...
Found 4 retained resources:
• DynamoDB Tables: 0
• CloudWatch Logs: 0
• S3 Buckets: 3
⠋ Deleting S3 buckets... 3/3
✓ Cleanup phase complete!
Resources deleted:
• S3 Buckets: 3
- test-stack-inputbucket-abc123
- test-stack-outputbucket-def456
- test-stack-loggingbucket-ghi789
Stack 'test-stack' and all resources completely removed.
Use Cases:
- Cleanup test/development environments to avoid charges
- CI/CD pipelines that provision and teardown stacks
- Automated testing with temporary stack creation
- Complete removal of failed stacks with retained resources
- Cleanup of stacks with LoggingBucket and CloudWatch logs
Important Notes:
--force-delete-allautomatically includes--empty-bucketsbehavior- Cleanup phase runs even if CloudFormation deletion fails
- Includes resources from nested stacks automatically
- Safe to run - only deletes resources that weren't deleted by CloudFormation
- Progress bars show real-time deletion status
Auto-Monitoring for In-Progress Deletions:
If you run delete on a stack that already has a DELETE operation in progress, the command automatically switches to monitoring mode instead of failing. This is useful if you started a deletion without --wait - simply run the command again to monitor:
# First run: no --wait, so the deletion is initiated and the command returns
$ idp-cli delete --stack-name test-stack --force
✓ Stack deletion initiated!
Stack: test-stack
Region: us-east-1
Monitor progress in the AWS Console:
CloudFormation → Stacks → test-stack
Or wait for it with:
idp-cli delete --stack-name test-stack --force --wait
# Second run - automatically monitors the in-progress deletion
$ idp-cli delete --stack-name test-stack
Stack 'test-stack' is already being deleted
Current status: DELETE_IN_PROGRESS
Switching to monitoring mode...
[Live progress display...]
✓ Stack deleted successfully!Canceling In-Progress Operations:
If a non-delete operation is in progress (CREATE, UPDATE), the delete command offers options to handle it:
$ idp-cli delete --stack-name test-stack
Stack 'test-stack' has an operation in progress: CREATE_IN_PROGRESS
Options:
1. Wait for CREATE to complete first
2. Cancel the CREATE and proceed with deletion
Do you want to cancel the CREATE and delete the stack? [yes/no/wait]: _- yes: Cancel the operation (if possible) and proceed with deletion
- no: Exit without making changes
- wait: Wait for the current operation to complete, then delete
With --force flag, the command automatically cancels the operation and proceeds with deletion:
# Force mode - automatically cancels and deletes
$ idp-cli delete --stack-name test-stack --force
Force mode: Canceling operation and proceeding with deletion...
✓ Stack reached stable state: ROLLBACK_COMPLETE
Proceeding with stack deletion...Note: CREATE operations cannot be cancelled directly - they must complete or roll back naturally. UPDATE operations can be cancelled immediately.
delete --terraform removes a Terraform deployment, keeping the resources CloudFormation would retain. See Terraform deployments below and Terraform Deployment.
deploy --terraform and delete --terraform deploy and remove the accelerator with Terraform (or OpenTofu) from a module publish --terraform produced. The terraform (or tofu) binary must be on PATH, or named with --terraform-bin; the command refuses to start without it.
idp-cli deploy --terraform --stack-name my-idp --admin-email user@example.com \
--region us-west-2 --parameters "MaxConcurrentWorkflows=50"
idp-cli delete --terraform --stack-name my-idp --region us-west-2 --force
# Build and deploy with Terraform from a local checkout
idp-cli deploy --terraform --from-code . --stack-name my-idp \
--admin-email user@example.com --region us-west-2With --from-code, deploy --terraform first builds and publishes the templates and the Terraform module from that checkout (as publish --source-dir <dir> --terraform does, honouring --bucket-basename, --prefix, --public, --build-max-workers, --clean-build and --no-validate-template), then deploys that module. With --headless or --govcloud it builds and deploys that variant's module. It cannot be combined with --terraform-module-url.
deploy --terraform options: --stack-name (at most 25 characters; it names the deployment's CloudFormation label stack, so no CloudFormation stack may already use it), --admin-email (required on the first deploy), --parameters (CloudFormation parameter names, mapped to module variables through the module's idp-terraform.json; an unknown name is an error, Number values become numbers and list-typed values are split on commas), --terraform-module-url (default: the moduleUrl in <prefix>/terraform/idp-terraform-latest.json beside the region's default template), --terraform-dir (default ./idp-terraform/<stack-name>), --terraform-bin, --terraform-assume-role-arn (an IAM role both the aws and awscc providers assume, written into each provider block of main.tf in that provider's own assume_role syntax; the resources are then created as that role, and you need sts:AssumeRole on it), --terraform-allow-replace-stateful (see below; --terraform-allow-replace-retained is the earlier name), --headless or --govcloud (deploy the latest module of that variant, from idp-terraform-<variant>-latest.json; the two cannot be combined with each other or with --terraform-module-url, and --admin-email is ignored with --headless, whose module has no Cognito), and --region. --wait is accepted and always implied. CloudFormation-only options (--from-code, --template-url, --template-file, --tags, --custom-config and the like) are refused with --terraform.
When the module comes from the pointer, idp-cli downloads the zip to read its idp-terraform.json and refuses it if its SHA-256 differs from the pointer's sha256. That is a consistency check of the copy idp-cli reads, against the manifest beside it: Terraform downloads the module again itself, and that download is not checked. A module named with --terraform-module-url is not checked at all, because there is no pointer to compare it with; to verify an archive, see Verifying the archive.
The command writes main.tf (the aws and awscc providers in the region and module "idp"), terraform.tfvars and idp-cli-parameters.json (both created readable by their owner only, as they can hold NoEcho values) into the directory. It then runs init (with -upgrade only when the module source changed since the last deploy from that directory, so providers do not float otherwise), plan -out, and show -json on the saved plan, and refuses the plan, applying nothing and listing the addresses, if it deletes or replaces a resource that holds data or that CloudFormation would retain (the statefulAddresses and retainedAddresses of the installed and the new module). --terraform-allow-replace-stateful applies such a plan anyway, losing those resources' data. Otherwise it applies exactly the saved plan. Output streams to stderr. State is local in that directory unless you add a backend block in another .tf file there. main.tf carries an idp-cli marker comment, and a main.tf without it is never overwritten. On a later deploy, the values of the earlier deploy, remembered by CloudFormation parameter name in idp-cli-parameters.json, are kept and the new --parameters applied over them, as a CloudFormation update keeps parameters it is not given. An S3 module URL is written as an s3:: source so Terraform downloads it with your AWS credentials.
delete --terraform (options --stack-name, --terraform-dir, --terraform-bin, --terraform-assume-role-arn, --region, --force) uses the role the deploy wrote into main.tf, or, with --terraform-assume-role-arn, rewrites the two provider blocks to assume that one first; then it runs init, empties every bucket in the state that destroy will delete, removes the resources CloudFormation would retain (the module's retainedAddresses, under module.idp.) from the state so they stay, then runs destroy -auto-approve, and lists what was retained. --force-delete-all does not apply.
Plain deploy and delete (without --terraform) refuse a Terraform deployment's label stack.
Moves a deployment of the separate
awslabs/genai-idp-terraform
implementation onto this repository's Terraform module in place: a new deployment
adopts the old one's buckets, tables, KMS key and Cognito user pool, so documents,
configuration, users and passwords stay, and recreates everything else. It applies
only to that implementation; it is not a CloudFormation or a configuration migration.
The procedure, what each phase checks and how to back out of it are in
Migrating from genai-idp-terraform.
Run the phases in this order. Each takes --stack-name (the new deployment's
name), --region, --terraform-dir (the new deployment's root configuration
directory, default ./idp-terraform/<stack-name>) and --json (the phase's result
as JSON on stdout; progress and the human-readable report go to stderr). A phase that
changes something only lists the changes unless given --apply, and every phase exits
1 when it finds something that blocks the next step.
| Phase | Changes | Options |
|---|---|---|
plan |
Local files only: main.tf, terraform.tfvars, adopt.tf and genai-idp-terraform-migration/; runs terraform init |
--old-state (the old terraform.tfstate or terraform state pull output), --admin-email (an existing user of exactly this name in the old pool is adopted), --parameters, --terraform-module-url (default: the region's latest module), --terraform-bin, --terraform-assume-role-arn |
check-retention |
Nothing (lists every object, and every object version of a versioned bucket, in each kept bucket) | --accept-expiry KIND (repeatable) |
freeze |
Disables the old deployment's EventBridge upload rule | --apply, --wait, --timeout (seconds, default 1800), --ignore-document KEY (repeatable) |
fix-config |
Backs up, then removes LatestRevision and PublishedRevision from the old Config# rows |
--apply, --restore <backup> |
adopt-user-pool |
terraform import of the user pool, then a backed-up rewrite of its schema in the new state |
--apply, --terraform-bin |
apply |
Applies the checked adoption plan | --apply, --accept-expiry KIND, --accept-change ADDRESS:RULE, --ignore-document KEY (all repeatable), --terraform-bin |
restore-config |
Restores, as a new revision, each kept configuration profile the first deploy replaced with the module preset | --apply |
detach |
Backs up the old state, then removes the adopted and kept resources from it | --old-terraform-dir (required), --apply, --terraform-bin |
check-destroy |
Nothing | --destroy-plan (terraform show -json of the old deployment's terraform plan -destroy -out=<file>) |
idp-cli terraform migrate-from-genai-idp-terraform plan --stack-name my-idp \
--region us-west-2 --old-state old-idp.tfstate --admin-email admin@example.com
idp-cli terraform migrate-from-genai-idp-terraform check-retention --stack-name my-idp --region us-west-2
idp-cli terraform migrate-from-genai-idp-terraform freeze --stack-name my-idp --region us-west-2 --apply --wait
idp-cli terraform migrate-from-genai-idp-terraform fix-config --stack-name my-idp --region us-west-2 --apply
idp-cli terraform migrate-from-genai-idp-terraform adopt-user-pool --stack-name my-idp --region us-west-2 --apply
idp-cli terraform migrate-from-genai-idp-terraform apply --stack-name my-idp --region us-west-2 --apply
idp-cli terraform migrate-from-genai-idp-terraform detach --stack-name my-idp --region us-west-2 \
--old-terraform-dir ./old-idp --apply
idp-cli terraform migrate-from-genai-idp-terraform check-destroy --stack-name my-idp \
--region us-west-2 --destroy-plan destroy-plan.jsonplan refuses, writing no adoption files, when the old state holds a resource that
holds data and has no verdict, or a resource of a type it does not know is safe to
destroy, and names each one. apply re-checks the earlier phases, then refuses a plan
in which any adopted resource is not imported in place, the user pool is not yet in
state or would lose a schema item, anything at all is replaced, destroyed or
forgotten, or an adopted resource (or a setting on a kept bucket) would lose a
protection such as point-in-time recovery, versioning or its KMS key. It lists every
other in-place change to those resources for you to read. check-destroy refuses
anything but the old deployment's destroy plan.
When plan was given --terraform-assume-role-arn, every later phase's own AWS calls
assume the same role (read back from the providers in main.tf), so they act in the
account Terraform deploys to.
Bootstrap a configuration (and optional synthetic test set) from a plain-language description — the scriptable equivalent of the web UI Quick Start widget. Authors a document-class schema from your prompt (reusing a catalog match when one fits), creates a config profile, and — when the document generator is available — generates a small labeled synthetic test set attached to it.
Usage:
idp-cli bootstrap --prompt "<description>" [--stack-name <stack>] [OPTIONS]Example:
idp-cli bootstrap \
--prompt "Invoices with vendor name, invoice number, date, and total amount" \
--stack-name my-idp-stackLocal mode — omit --stack-name to author and print the schema as JSON without
saving anything to a stack:
idp-cli bootstrap --prompt "Bank statements with account holder and transactions"Options:
--prompt,-p: (required) Natural-language description of the document type.--stack-name: Target CloudFormation stack. Omit for local mode (print schema, no save).--class-name: Document class name to use as the schema$id/ document type.--field-hint: A field the schema must include. Repeatable:--field-hint X --field-hint Y.--config-profile(alias:--config-version): Existing configuration profile to source catalog classes from / merge the new class into.--target-profile(alias:--target-version): Name of the configuration profile to create (default:bootstrap-<class>).--count,-c: Number of synthetic documents to generate (default:3).--threshold: Generation quality threshold, 1–10 (default:7).--augment: Apply scan/fax-style image augmentation to generated documents.--model-id: Bedrock model id override for schema authoring / generation.--region: AWS region (optional).
Note: The created profile is not activated automatically (unlike the web UI
Quick Start). Activate it from Configuration › View/Edit Configuration in the UI
when you're ready to process documents with it. Synthetic generation is optional and
requires the Test Set Generator extension (deployed stack) or
pip install -e "lib/idp_common_pkg[synthesis]" (local); without it, the config is still
created and you can upload your own documents to build a test set. See the
Quick Start guide for the full workflow.
Process a batch of documents.
Usage:
idp-cli process [OPTIONS]
# or (deprecated alias)
idp-cli run-inference [OPTIONS]Document Source (choose ONE):
--manifest: Path to manifest file (CSV or JSON)--dir: Local directory containing documents--s3-uri: S3 URI in InputBucket--test-set: Test set ID from test set bucket
Options:
--stack-name(required): CloudFormation stack name--batch-id: Custom batch ID (auto-generated if omitted, ignored with --test-set)--batch-prefix: Prefix for auto-generated batch ID (default:cli-batch)--file-pattern: File pattern for directory/S3 scanning (default:*.pdf)--recursive/--no-recursive: Include subdirectories (default: recursive)--number-of-files: Limit number of files to process--config: Refused. A configuration file is not applied to a batch submission, so passing one exits non-zero rather than running under the stack's existing configuration. Upload the file as a profile withconfig-upload, then pass--config-profile.--config-profile(alias:--config-version): Configuration profile to use for processing (e.g.,default,lending)--context: Context description for test run (used with --test-set, e.g., "Model v2.1", "Production validation")--monitor: Monitor progress until completion--refresh-interval: Seconds between status checks (default: 5)--region: AWS region (optional)
A document shows as Queued before it has a tracking row.
A document's tracking row is not written by the upload. The upload puts the object in the input bucket, and EventBridge then invokes the queue sender, which writes the row — asynchronously, so for the first seconds of a batch the status lookup has no record of a document that is about to process normally.
Monitoring reports such a document as Queued for up to 60 seconds after the
batch was submitted. Past that it is reported as failed, with
Document not found in tracking table, which at that point means the document
really did not reach the pipeline — check the input bucket's EventBridge
notifications and the queue sender's logs. Set IDP_NOT_FOUND_GRACE_SECONDS to
widen or narrow the window.
status for a single document, which
answers immediately: a document id that is not in the table is reported missing
straight away rather than after a minute, so a typo or a document whose data
retention has elapsed does not look like a slow query.
A document waiting for human review ends the monitor.
With review queues on (hitl.enabled: true), a document that raises a
confidence alert ends its first processing run as HITL_IN_PROGRESS and waits
for a reviewer. Nothing more happens to it until someone works it, so
--monitor counts it under Awaiting Review rather than Running, and
stops once no document is queued or running. The final summary says how many
documents are awaiting review, lists them, and says where to work them: the
web UI's review queues (Home › Queues, or Review Studio › Review
Management for an Admin). Completing or rejecting a review continues the
document; run idp-cli status --batch-id <id> afterwards to see the outcome.
Test Set Integration: For test runs to appear properly in the Test Studio UI, use either:
--test-set: Process test set directly by ID (recommended for test sets)--manifest: Use manifest file with populated baseline_source column for evaluation tracking
Other options (--dir, --s3-uri) are for general document processing but won't integrate with test studio tracking.
Examples:
# Process from local directory
idp-cli process \
--stack-name my-stack \
--dir ./documents/ \
--monitor
# Process from manifest with baselines (enables evaluation)
idp-cli process \
--stack-name my-stack \
--manifest documents-with-baselines.csv \
--monitor
# Process from manifest with limited files
idp-cli process \
--stack-name my-stack \
--manifest documents-with-baselines.csv \
--number-of-files 10 \
--monitor
# Process test set (integrates with Test Studio UI - use test set ID)
idp-cli process \
--stack-name my-stack \
--test-set fcc-example-test \
--monitor
# Process test set with limited files for quick testing
idp-cli process \
--stack-name my-stack \
--test-set fcc-example-test \
--number-of-files 5 \
--monitor
# Process test set with custom context (for tracking in Test Studio)
idp-cli process \
--stack-name my-stack \
--test-set fcc-example-test \
--context "Model v2.1 - improved prompts" \
--monitor
# Process S3 URI
idp-cli process \
--stack-name my-stack \
--s3-uri archive/2024/ \
--monitor
# Process with specific configuration profile
idp-cli process \
--stack-name my-stack \
--dir ./documents/ \
--config-profile lending \
--monitor
# Process test set with configuration profile
idp-cli process \
--stack-name my-stack \
--test-set fcc-example-test \
--config-profile lending \
--context "Testing with the lending profile" \
--monitorReprocess existing documents from a specific pipeline step.
Usage:
idp-cli reprocess [OPTIONS]
# or (deprecated alias)
idp-cli rerun-inference [OPTIONS]Use Cases:
- Test different classification or extraction configurations without re-running OCR
- Fix classification errors and reprocess extraction
- Iterate on prompt engineering rapidly
Options:
--stack-name(required): CloudFormation stack name--step(required): Pipeline step to rerun from (classificationorextraction)- Document Source (choose ONE):
--document-ids: Comma-separated document IDs--batch-id: Batch ID to get all documents from
--force: Skip confirmation prompt (useful for automation)--monitor: Monitor progress until completion--refresh-interval: Seconds between status checks (default: 5)--region: AWS region (optional)
Step Behavior:
classification: Clears page classifications and sections, reruns classification → extraction → assessmentextraction: Keeps classifications, clears extraction data, reruns extraction → assessment
Examples:
# Rerun classification for specific documents
idp-cli reprocess \
--stack-name my-stack \
--step classification \
--document-ids "batch-123/doc1.pdf,batch-123/doc2.pdf" \
--monitor
# Rerun extraction for entire batch
idp-cli reprocess \
--stack-name my-stack \
--step extraction \
--batch-id cli-batch-20251015-143000 \
--monitor
# Automated rerun (skip confirmation - perfect for CI/CD)
idp-cli reprocess \
--stack-name my-stack \
--step classification \
--batch-id test-set \
--force \
--monitorWhat Gets Cleared:
| Step | Clears | Keeps |
|---|---|---|
classification |
Page classifications, sections, extraction results | OCR data (pages, images, text) |
extraction |
Section extraction results, attributes | OCR data, page classifications, section structure |
Benefits:
- Leverages existing OCR data (saves time and cost)
- Rapid iteration on classification/extraction configurations
- Perfect for prompt engineering experiments
Demo:
RerunInference.mp4
Check status of documents by batch ID, document ID, or search criteria.
Usage:
idp-cli status [OPTIONS]Document Source (choose ONE):
--batch-id: Batch identifier or PK substring to search for (searches tracking table)--document-id: Single document ID (check individual document)
Optional Filters and Display:
--object-status: Filter by status (COMPLETED, FAILED, QUEUED, RUNNING, PROCESSING)--show-details: Show detailed document information in table format--get-time: Calculate and display timing statistics (processing time, queue time, total time)--include-metering: Include Lambda metering statistics (GB-seconds by stage) - requires--get-time
Other Options:
--stack-name(required): CloudFormation stack name--wait: Wait for all documents to complete--refresh-interval: Seconds between status checks (default: 5)--format: Output format -table(default) orjson--region: AWS region (optional)
How --batch-id Works:
The --batch-id option performs a PK substring search in the DynamoDB tracking table. This means:
- It searches for all documents where the PK (Primary Key) contains your search string
- You can search for exact batch IDs:
cli-batch-20251015-143000 - You can search for partial matches:
batch-123finds all documents with "batch-123" in their path - You can search across multiple batches:
invoicefinds all documents with "invoice" in their name
Examples:
# Search for all documents in a batch (PK substring search)
idp-cli status \
--stack-name my-stack \
--batch-id cli-batch-20251015-143000
# Search for documents across batches with partial match
idp-cli status \
--stack-name my-stack \
--batch-id batch-123
# Search for completed documents only
idp-cli status \
--stack-name my-stack \
--batch-id batch-123 \
--object-status COMPLETED
# Search for failed documents with details
idp-cli status \
--stack-name my-stack \
--batch-id batch-123 \
--object-status FAILED \
--show-details
# Search with timing statistics
idp-cli status \
--stack-name my-stack \
--batch-id batch-123 \
--object-status COMPLETED \
--get-time
# Search with timing and Lambda metering data
idp-cli status \
--stack-name my-stack \
--batch-id test \
--object-status COMPLETED \
--get-time \
--include-metering
# Check single document status
idp-cli status \
--stack-name my-stack \
--document-id batch-123/invoice.pdf
# Monitor documents until completion
idp-cli status \
--stack-name my-stack \
--batch-id batch-123 \
--wait
# Get JSON output for scripting
idp-cli status \
--stack-name my-stack \
--batch-id batch-123 \
--format jsonTiming Statistics:
When using --get-time, the command calculates:
- Processing Time: WorkflowStartTime → CompletionTime (actual processing duration)
- Queue Time: QueuedTime → WorkflowStartTime (time waiting in queue)
- Total Time: QueuedTime → CompletionTime (end-to-end duration)
For each metric, you'll see:
- Average, Median, Min, Max, Standard Deviation, Total
- ObjectKey for min/max values (helps identify outliers)
Lambda Metering:
When using --include-metering with --get-time, you'll see GB-seconds usage by stage:
- Assessment, OCR, Classification, Extraction, Summarization
- Statistics: Average, Median, Min, Max, Std Dev, Total
- Cost estimates based on AWS Lambda pricing ($0.0000166667 per GB-second)
Example Output with Timing:
$ idp-cli status --stack-name my-stack --batch-id test-batch --object-status COMPLETED --get-time
Searching for documents with PK containing 'test-batch'...
✓ Found 25 matching documents
Timing Statistics:
Valid documents: 25
Processing Time (WorkflowStartTime → CompletionTime):
┏━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ Metric ┃ Value ┃ ObjectKey ┃
┡━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┩
│ Average │ 45.23s │ │
│ Median │ 43.10s │ │
│ Minimum │ 32.45s │ test-batch/small-doc.pdf │
│ Maximum │ 78.90s │ test-batch/large-doc.pdf │
│ Std Dev │ 12.34s │ │
│ Total │ 18m 50.75s │ │
└──────────┴─────────────┴──────────────────────────────┘Programmatic Use:
The command returns exit codes for scripting:
0- Document(s) completed successfully1- Document(s) failed2- Document(s) still processing, waiting for human review, or the outcome could not be established
--wait returns once no document is queued or running, so a batch that is only
waiting on reviewers ends the wait. It exits 2 rather than 0 in that case,
because "every document completed" is not yet true; a batch with a failure still
exits 1. --format json reports the count as awaiting_review, and the table
output's final line reads AWAITING REVIEW (<n> awaiting review).
--wait, the polled table form and --format json all derive their code from the
same place. A
batch that finished with failures exits 1 whether you polled it or waited on it; it
used to exit 0 when waited on. For a single document the code comes from the progress
bucket the SDK put it in, so the three forms cannot disagree about it. They did for four
document statuses — a redacted-and-superseded original, a document with no tracking row
past the window in which one could still arrive, and the two the malware gate refuses —
because --format json matched the status string against COMPLETED, FAILED and
ABORTED and answered 2 for anything else. Those four are terminal, and 2 is this
CLI's code for "still processing", so a script waiting for the code to stop being 2
waited forever. Before this change
--wait exited 0 on
that batch while the poll exited 1, which meant idp-cli status --wait && deploy
proceeded after a batch in which every document failed. If you have a script that
relied on --wait always exiting 0, it will now stop on a failed batch — that is
the intended behaviour, but it is a change.
--wait also exits 2 when the watch ended without a verdict: a monitoring error, or
Ctrl-C. Nothing about the batch was measured on those paths, so 0 would assert a
success and 1 would report failures that may not exist.
process --monitor and rerun --monitor deliberately still exit 0 regardless of
what the monitored batch did. Their work is the submission, which succeeded; exiting
non-zero because 1 of 100 documents failed would stop process --monitor && download-results from collecting the 99 that worked. Ask for the batch's verdict with
idp-cli status --batch-id <id>, which answers exactly that.
JSON Output Format:
# Single document
$ idp-cli status --stack-name my-stack --document-id batch-123/invoice.pdf --format json
{
"document_id": "batch-123/invoice.pdf",
"status": "COMPLETED",
"duration": 125.4,
"start_time": "2025-01-01T10:30:45Z",
"end_time": "2025-01-01T10:32:50Z",
"num_sections": 2,
"exit_code": 0
}
# Table output includes final status summary
$ idp-cli status --stack-name my-stack --document-id batch-123/invoice.pdf
[status table]
FINAL STATUS: COMPLETED | Duration: 125.4s | Exit Code: 0Scripting Examples:
#!/bin/bash
# Wait for document completion and check result
idp-cli status --stack-name prod --document-id batch-001/invoice.pdf --wait
exit_code=$?
if [ $exit_code -eq 0 ]; then
echo "Document processed successfully"
# Proceed with downstream processing
else
echo "Document processing failed"
exit 1
fi#!/bin/bash
# Poll document status in script.
#
# Branch on `.exit_code`, not on `.status`. The code is 0 done, 1 failed, 2 not
# established yet, and it is decided from the progress bucket — so it covers every
# way a document can finish, including the terminal ones whose status string is
# neither COMPLETED nor FAILED (a redacted copy superseding the original, a refusal
# at the malware gate, a document that never acquired a tracking row). A loop
# comparing `.status` against those two literals never exits on any of them.
while true; do
status=$(idp-cli status --stack-name prod --document-id batch-001/invoice.pdf --format json)
code=$(echo "$status" | jq -r '.exit_code')
if [ "$code" = "0" ]; then
echo "Processing complete!"
break
elif [ "$code" = "1" ]; then
# `error` is always present on a failure but can be an empty string: the
# reason is the pipeline's, and some terminal states record none. jq's `//`
# falls back on null and false only, so test for emptiness in the shell.
reason=$(echo "$status" | jq -r '.error // ""')
[ -n "$reason" ] || reason="no reason recorded"
echo "Processing failed: $(echo "$status" | jq -r '.status') — $reason"
exit 1
fi
sleep 5
doneDownload processing results to local directory.
Usage:
idp-cli download-results [OPTIONS]Options:
--stack-name(required): CloudFormation stack name--batch-id: Batch identifier (mutually exclusive with--document-id/--run-id)--document-id: Document object key — required with--run-idto download a specific document version--run-id: Version run id (fromidp-cli list-versions). Downloads the exact pinned S3 bytes of that processing run. Requires--document-id.--output-dir(required): Local directory to download to--file-types: File types to download (default:all)- Options:
pages,sections,summary,evaluation, orall
- Options:
--region: AWS region (optional)
Examples:
# Download all results
idp-cli download-results \
--stack-name my-stack \
--batch-id cli-batch-20251015-143000 \
--output-dir ./results/
# Download only extraction results
idp-cli download-results \
--stack-name my-stack \
--batch-id cli-batch-20251015-143000 \
--output-dir ./results/ \
--file-types sections
# Download evaluation results only
idp-cli download-results \
--stack-name my-stack \
--batch-id eval-batch-20251015 \
--output-dir ./eval-results/ \
--file-types evaluation
# Download a specific document VERSION (exact bytes of one processing run)
idp-cli download-results \
--stack-name my-stack \
--document-id loan-12345/package.pdf \
--run-id 20250707T141530Z-exec-abc \
--output-dir ./results/Output Structure:
./results/
└── cli-batch-20251015-143000/
└── invoice.pdf/
├── pages/
│ └── 1/
│ ├── image.jpg
│ ├── rawText.json
│ └── result.json
├── sections/
│ └── 1/
│ ├── result.json # Extracted structured data
│ └── summary.json
├── summary/
│ ├── fulltext.txt
│ └── summary.json
└── evaluation/ # Only present if baseline provided
├── report.json # Detailed metrics
└── report.md # Human-readable report
Promote a processed document's output to the evaluation baseline — the
scriptable equivalent of the web UI's More Actions › Use As Evaluation Baseline (Admin, Author).
Copies every output object for the document into the evaluation baseline bucket
and sets the document's EvaluationStatus to BASELINE_AVAILABLE. Runs
synchronously (returns once the copy is complete).
Use this to capture a manually validated result as the "ground truth" that future re-runs of the same document are evaluated against.
Usage:
idp-cli use-as-baseline [OPTIONS]Options:
--stack-name(required): CloudFormation stack name--document-id(required): Document object key (S3 key) of a processed document, e.g.loan-12345/package.pdf--region: AWS region (optional)
Example:
idp-cli use-as-baseline \
--stack-name my-stack \
--document-id loan-12345/package.pdfThe document must have finished processing (its output prefix must exist); the caller's IAM credentials need read on the output bucket and write on the evaluation baseline bucket and tracking table.
List the retained processing-run versions of a document, newest first. Each successful run of a document is retained as a version whose output bytes are pinned by S3 object version; use a version's Run ID with download-results --run-id to fetch that exact version.
Usage:
idp-cli list-versions [OPTIONS]Options:
--stack-name(required): CloudFormation stack name--document-id(required): Document object key (its tracking id)--region: AWS region (optional)
Example:
idp-cli list-versions \
--stack-name my-stack \
--document-id loan-12345/package.pdfOutput is a table of Run ID, Completed, Config Profile, Pages, and Files. See the Document Versions guide for how versioning works and the Web UI / API surfaces.
Delete documents and all associated data from the IDP system.
Usage:
idp-cli delete-documents [OPTIONS]Document Selection (choose ONE):
--document-ids: Comma-separated list of document IDs (S3 object keys) to delete--batch-id: Delete all documents in this batch--pattern: Wildcard pattern to match document keys (e.g."batch-123/*.pdf","*invoice*")
Options:
--stack-name(required): CloudFormation stack name--status-filter: Only delete documents with this status (use with --batch-id or --pattern)- Options:
FAILED,COMPLETED,PROCESSING,QUEUED
- Options:
--dry-run: Show what would be deleted without actually deleting--force,-y: Skip confirmation prompt--region: AWS region (optional)
What Gets Deleted:
- Source files from input bucket
- Processed outputs from output bucket
- DynamoDB tracking records
- List entries in tracking table
Exit codes: No documents found for batch … with exit 0 means the selector matched
nothing, and that is all it means. A failure while finding the documents — a throttled or
rejected table scan, a table that is not there — prints the cause and exits 1 instead of
reporting that there was nothing to delete.
A delete that could not finish says so. Each document's cleanup has several steps — the input object, every output version, the tracking-list row, the run records, the tracking record — and a step that fails is reported per document rather than being reported as a completed delete. A transient throttle is retried first — four attempts, and no further attempt starts once five seconds of retrying is spent — so what is reported is a sustained failure rather than a momentary one.
Two things to expect when a delete is reported as failed:
- The document's tracking record is kept on purpose if its tracking-list row could not be cleared, because that record is what a retry needs to find the row. The document therefore still appears in the document list, while its input file and outputs may already be gone — so the entry can open a document whose content is no longer there. Run the same delete again: the retry picks up where the first attempt stopped.
--dry-runis unaffected and still issues no delete of any kind.
A run in which every deletion failed also exits 1; it used to exit 0 after printing "Deleted 0/2 document(s)", so an automated cleanup step reported success having deleted nothing.
Examples:
# Delete specific documents by ID
idp-cli delete-documents \
--stack-name my-stack \
--document-ids "batch-123/doc1.pdf,batch-123/doc2.pdf"
# Delete all documents in a batch
idp-cli delete-documents \
--stack-name my-stack \
--batch-id cli-batch-20250123
# Delete only failed documents in a batch
idp-cli delete-documents \
--stack-name my-stack \
--batch-id cli-batch-20250123 \
--status-filter FAILED
# Dry run to see what would be deleted
idp-cli delete-documents \
--stack-name my-stack \
--batch-id cli-batch-20250123 \
--dry-run
# Delete documents matching a wildcard pattern
idp-cli delete-documents \
--stack-name my-stack \
--pattern "batch-123/*.pdf"
# Delete all failed invoice documents across batches
idp-cli delete-documents \
--stack-name my-stack \
--pattern "*invoice*" \
--status-filter FAILED
# Dry run with pattern to preview matches
idp-cli delete-documents \
--stack-name my-stack \
--pattern "*2024*" \
--dry-run
# Force delete without confirmation
idp-cli delete-documents \
--stack-name my-stack \
--document-ids "batch-123/doc1.pdf" \
--forceOutput Example:
Connecting to stack: my-stack
Getting documents for batch: cli-batch-20250123
Found 15 document(s) in batch
(filtered by status: FAILED)
⚠️ Documents to be deleted:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
• cli-batch-20250123/doc1.pdf
• cli-batch-20250123/doc2.pdf
• cli-batch-20250123/doc3.pdf
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Delete 3 document(s) permanently? [y/N]: y
✓ Successfully deleted 3 document(s)
Use Cases:
- Clean up failed documents after fixing issues
- Remove test documents from a batch
- Free up storage by removing old processed documents
- Prepare for reprocessing by removing previous results
Generate a manifest file from directory or S3 URI, or create a test set in the test set bucket.
Usage:
idp-cli generate-manifest [OPTIONS]Options:
- Source (choose ONE):
--dir: Local directory to scan--s3-uri: S3 URI to scan
--baseline-dir: Baseline directory for automatic matching (only with --dir)--output: Output manifest file path (CSV) - optional when using --test-set--file-pattern: File pattern (default:*.pdf)--case-sensitive/--no-case-sensitive: Match--file-patternexactly as written (default:--no-case-sensitive)--recursive/--no-recursive: Include subdirectories (default: recursive)--region: AWS region (optional)- Test Set Creation:
--test-set: Test set name - creates folder in test set bucket and uploads files--stack-name: CloudFormation stack name (required with --test-set)--force/-y: Overwrite an existing test set without the confirmation prompt
Examples:
# Generate from directory
idp-cli generate-manifest \
--dir ./documents/ \
--output manifest.csv
# Generate with automatic baseline matching
idp-cli generate-manifest \
--dir ./documents/ \
--baseline-dir ./validated-baselines/ \
--output manifest-with-baselines.csv
# Create test set and upload files (no manifest needed - use test set name)
idp-cli generate-manifest \
--dir ./documents/ \
--baseline-dir ./baselines/ \
--test-set "fcc example test" \
--stack-name IDP
# Create test set with manifest output
idp-cli generate-manifest \
--dir ./documents/ \
--baseline-dir ./baselines/ \
--test-set "fcc example test" \
--stack-name IDP \
--output test-manifest.csv
# Refresh an existing test set from a script or CI job (asks nothing)
idp-cli generate-manifest \
--dir ./documents/ \
--baseline-dir ./baselines/ \
--test-set "fcc example test" \
--stack-name IDP \
--forceHow --file-pattern selects documents: the pattern is matched against each
file's base name, on both the --dir and the --s3-uri path, and the match
ignores case. So the default *.pdf selects statement.PDF and Statement.Pdf
as well, which is the point — a corpus exported from a system that uppercases
extensions used to produce a valid-looking manifest with no rows in it, at exit 0.
Case folding covers the whole pattern rather than an extension picked out of it, so
Invoice*.pdf also selects INVOICE01.PDF. Pass --case-sensitive for a pattern
whose case is deliberate — distinguishing an Invoice-*.pdf family from an
invoice-*.pdf one, say. A pattern containing a directory component
(--file-pattern "sub/*.pdf") is refused: point --dir or --s3-uri at the
directory and use --recursive / --no-recursive to choose the depth.
On the --dir path, hidden files follow the usual shell rule and are excluded unless
the pattern itself starts with a dot. The --s3-uri path has no such rule — it
filters keys by base name only — so --file-pattern "*" against a test-set prefix
selects the .uploading marker object as though it were a document. Name the extension
you want rather than relying on * when scanning a bucket.
--file-pattern on process and run-inference is a different scan, in
idp_sdk, and it is still case-sensitive. Pass the extension's actual case there, or
generate a manifest with this command and process that.
How --baseline-dir is matched: a baseline sub-directory must be named after the
document file it labels, extension included (invoice.pdf/, not invoice/). The
match uses the same rule as --file-pattern, so it ignores case unless
--case-sensitive is given, and W2-A.PDF is therefore labelled by w2-a.pdf/. Two
baseline directories differing only in case are refused as ambiguous. A baseline
directory matching no document is named and skipped rather than uploaded; if no
document matched a baseline, --test-set refuses before anything is cleared or
uploaded, and a manifest-only run warns and leaves baseline_source empty for you to
fill in.
Test Set Creation:
When using --test-set, the command:
- Requires
--stack-name,--baseline-dir, and--dir - Uploads input files to
s3://test-set-bucket/{test-set-id}/input/ - Uploads baseline files to
s3://test-set-bucket/{test-set-id}/baseline/ - Creates proper test set structure for evaluation workflows
- Test set will be auto-detected by the Test Studio UI
If the upload fails partway through, the .uploading marker object the command
places under the test set's prefix is removed before it exits. That marker is what
stops the Test Studio resolver registering a folder that is still being filled, so a
marker left behind makes a folder invisible to the backend — and re-running the upload
does not clear it, because the new run writes it again. The command exits non-zero and
names what went wrong.
Be precise about what that leaves behind, because it is not nothing: the objects uploaded before the failure stay, and with the marker gone the resolver may register them as a partial, unlabeled set (a set with documents and no baselines is a legitimate shape, so the backend cannot tell the two apart). That is deliberate — re-running clears the prefix first, so the state is recoverable, whereas a surviving marker made the folder invisible permanently. If you do not want the partial set visible, delete the prefix before retrying.
In the one case where the marker itself cannot be deleted (an IAM policy with
s3:PutObject but not s3:DeleteObject on the test set bucket) the command fails
rather than reporting success, and the error names the S3 object to delete by hand.
Overwriting an existing test set: if the test set name already exists, everything
under its prefix — including the baselines a previous evaluation was scored against —
is deleted before the new files are uploaded, so the command asks for confirmation
first. Answer y to proceed, anything else to abort. Run non-interactively (a CI job,
a make target, stdin from /dev/null) there is no answer to read, and the command
aborts with exit 1 and changes nothing; pass --force to overwrite without the
prompt. Baselines cleared this way are not recoverable from the CLI.
Process the created test set:
# Using test set ID (from UI or after creation)
idp-cli process --stack-name IDP --test-set fcc-example-test --monitor
# Or using S3 URI to process input files directly
idp-cli run-inference --stack-name IDP --s3-uri s3://test-set-bucket/fcc-example-test/input/
# Or using manifest if generated
idp-cli run-inference --stack-name IDP --manifest test-manifest.csvValidate a manifest file without processing.
Usage:
idp-cli validate-manifest [OPTIONS]Options:
--manifest(required): Path to manifest file to validate (CSV or JSON)
Examples:
# Validate a CSV manifest
idp-cli validate-manifest --manifest documents.csv
# Validate a JSON manifest
idp-cli validate-manifest --manifest documents.jsonList recent batch processing jobs.
Usage:
idp-cli list-batches [OPTIONS]Options:
--stack-name(required): CloudFormation stack name--limit: Maximum number of batches to list (default: 10)--region: AWS region (optional)
Examples:
# List last 10 batches (default)
idp-cli list-batches --stack-name my-stack
# List last 5 batches
idp-cli list-batches --stack-name my-stack --limit 5
# List with specific region
idp-cli list-batches --stack-name my-stack --limit 20 --region us-west-2This workflow demonstrates how to process documents, manually validate results, and then reprocess with evaluation to measure accuracy.
Deploy an IDP stack if you haven't already:
idp-cli deploy \
--stack-name eval-testing \
--admin-email your.email@example.com \
--max-concurrent 50 \
--waitWhat happens: CloudFormation creates ~120 resources including S3 buckets, Lambda functions, Step Functions, and DynamoDB tables. This takes 10-15 minutes.
Process your test documents to generate initial extraction results:
# Prepare test documents
mkdir -p ~/test-documents
cp /path/to/your/invoice.pdf ~/test-documents/
cp /path/to/your/w2.pdf ~/test-documents/
cp /path/to/your/paystub.pdf ~/test-documents/
# Process documents
idp-cli run-inference \
--stack-name eval-testing \
--dir ~/test-documents/ \
--batch-id initial-run \
--monitorWhat happens: Documents are uploaded to S3, processed through OCR, classification, extraction, assessment, and summarization. Results are stored in OutputBucket.
Monitor output:
✓ Uploaded 3 documents to InputBucket
✓ Sent 3 messages to processing queue
Monitoring Batch: initial-run
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Status Summary
─────────────────────────────────────
✓ Completed 3 100%
⟳ Running 0 0%
⏸ Queued 0 0%
✗ Failed 0 0%
◐ Awaiting Review 0 0%
Download the extraction results (sections) for manual review:
idp-cli download-results \
--stack-name eval-testing \
--batch-id initial-run \
--output-dir ~/initial-results/ \
--file-types sectionsResult structure:
~/initial-results/initial-run/
├── invoice.pdf/
│ └── sections/
│ └── 1/
│ └── result.json # Extracted data to validate
├── w2.pdf/
│ └── sections/
│ └── 1/
│ └── result.json
└── paystub.pdf/
└── sections/
└── 1/
└── result.json
Review and correct the extraction results to create validated baselines.
4.1 Review extraction results:
# View extracted data for invoice
cat ~/initial-results/initial-run/invoice.pdf/sections/1/result.json | jq .
# Example output:
{
"attributes": {
"Invoice Number": "INV-2024-001",
"Invoice Date": "2024-01-15",
"Total Amount": "$1,250.00",
"Vendor Name": "Acme Corp"
}
}4.2 Validate and correct:
Compare extracted values against the actual documents. If you find errors, create corrected baseline files:
# Create baseline directory structure
mkdir -p ~/validated-baselines/invoice.pdf/sections/1/
mkdir -p ~/validated-baselines/w2.pdf/sections/1/
mkdir -p ~/validated-baselines/paystub.pdf/sections/1/
# Copy and edit result files
cp ~/initial-results/initial-run/invoice.pdf/sections/1/result.json \
~/validated-baselines/invoice.pdf/sections/1/result.json
# Edit the baseline to correct any errors
vi ~/validated-baselines/invoice.pdf/sections/1/result.json
# Repeat for other documents...Baseline directory structure:
~/validated-baselines/
├── invoice.pdf/
│ └── sections/
│ └── 1/
│ └── result.json # Corrected/validated data
├── w2.pdf/
│ └── sections/
│ └── 1/
│ └── result.json
└── paystub.pdf/
└── sections/
└── 1/
└── result.json
Create a manifest that links each document to its validated baseline:
cat > ~/evaluation-manifest.csv << EOF
document_path,baseline_source
/home/user/test-documents/invoice.pdf,/home/user/validated-baselines/invoice.pdf/
/home/user/test-documents/w2.pdf,/home/user/validated-baselines/w2.pdf/
/home/user/test-documents/paystub.pdf,/home/user/validated-baselines/paystub.pdf/
EOFManifest format:
document_path: Path to original documentbaseline_source: Path to directory containing validated sections
Alternative using auto-matching:
# Generate manifest with automatic baseline matching
idp-cli generate-manifest \
--dir ~/test-documents/ \
--baseline-dir ~/validated-baselines/ \
--output ~/evaluation-manifest.csvReprocess documents with the baseline-enabled manifest. The accelerator will automatically run evaluation:
idp-cli run-inference \
--stack-name eval-testing \
--manifest ~/evaluation-manifest.csv \
--batch-id eval-run-001 \
--monitorWhat happens:
- Documents are processed through the pipeline as before
- Evaluation step is automatically triggered because baselines are provided
- The evaluation module compares extracted values against baseline values
- Detailed metrics are calculated per attribute and per document
Processing time: Similar to initial run, plus ~5-10 seconds per document for evaluation.
Download the evaluation results to analyze accuracy:
✓ Synchronous Evaluation: Evaluation runs as the final step in the workflow before completion. When a document shows status "COMPLETE", all processing including evaluation is finished - results are immediately available for download.
# Download evaluation results (no waiting needed)
idp-cli download-results \
--stack-name eval-testing \
--batch-id eval-run-001 \
--output-dir ~/eval-results/ \
--file-types evaluation
# Verify evaluation data is present
ls -la ~/eval-results/eval-run-001/invoice.pdf/evaluation/
# Should show: report.json and report.mdReview evaluation report:
# View detailed evaluation metrics
cat ~/eval-results/eval-run-001/invoice.pdf/evaluation/report.json | jq .View human-readable report:
# Markdown report with visual formatting
cat ~/eval-results/eval-run-001/invoice.pdf/evaluation/report.mdThe IDP Accelerator provides multiple ways to analyze evaluation results across batches and at scale.
The accelerator automatically stores evaluation metrics in Athena tables for SQL-based analysis.
Available Tables:
evaluation_results- Per-document evaluation metricsevaluation_attributes- Per-attribute scoresevaluation_summary- Aggregated statistics
Example Queries:
-- Overall accuracy across all batches
SELECT
AVG(overall_accuracy) as avg_accuracy,
COUNT(*) as total_documents,
SUM(CASE WHEN overall_accuracy >= 0.95 THEN 1 ELSE 0 END) as high_accuracy_count
FROM evaluation_results
WHERE batch_id LIKE 'eval-run-%';
-- Attribute-level accuracy
SELECT
attribute_name,
AVG(score) as avg_score,
COUNT(*) as total_occurrences,
SUM(CASE WHEN match = true THEN 1 ELSE 0 END) as correct_count
FROM evaluation_attributes
GROUP BY attribute_name
ORDER BY avg_score DESC;
-- Compare accuracy across different configurations
SELECT
batch_id,
AVG(overall_accuracy) as accuracy,
COUNT(*) as doc_count
FROM evaluation_results
WHERE batch_id IN ('config-v1', 'config-v2', 'config-v3')
GROUP BY batch_id;Access Athena:
# Get Athena database name from stack outputs
aws cloudformation describe-stacks \
--stack-name eval-testing \
--query 'Stacks[0].Outputs[?OutputKey==`ReportingDatabase`].OutputValue' \
--output text
# Query via AWS Console or CLI
aws athena start-query-execution \
--query-string "SELECT * FROM evaluation_results LIMIT 10" \
--result-configuration OutputLocation=s3://your-results-bucket/For detailed Athena table schemas and query examples, see:
../docs/reporting-database.md- Complete Athena table reference../docs/evaluation.md- Evaluation methodology and metrics
The IDP web UI provides an Agent Analytics feature for visual analysis of evaluation results.
Access the UI:
- Get web UI URL from stack outputs:
aws cloudformation describe-stacks \
--stack-name eval-testing \
--query 'Stacks[0].Outputs[?OutputKey==`ApplicationWebURL`].OutputValue' \
--output text-
Login with admin credentials (from deployment email)
-
Navigate to Analytics → Agent Analytics
Available Analytics:
- Accuracy Trends - Track accuracy over time across batches
- Attribute Heatmaps - Visualize which attributes perform best/worst
- Batch Comparisons - Compare different configurations side-by-side
- Error Analysis - Identify common error patterns
- Confidence Correlation - Analyze relationship between assessment confidence and accuracy
Key Features:
- Interactive charts and visualizations
- Filter by batch, date range, document type, or attribute
- Export results to CSV for further analysis
- Drill-down to individual document details
For complete Agent Analytics documentation, see:
../docs/agent-analysis.md- Agent Analytics user guide
Required Field:
document_path: Local file path or full S3 URI (s3://bucket/key)
Optional Field:
baseline_source: Path or S3 URI to validated baseline for evaluation
Note: Document IDs are auto-generated from filenames (e.g., invoice.pdf → invoice)
Examples:
document_path
/home/user/docs/invoice.pdf
/home/user/docs/w2.pdf
s3://external-bucket/statement.pdfdocument_path,baseline_source
/local/invoice.pdf,s3://baselines/invoice/
/local/w2.pdf,/local/validated-baselines/w2/
s3://docs/statement.pdf,s3://baselines/statement/[
{
"document_path": "/local/invoice.pdf",
"baseline_source": "s3://baselines/invoice/"
},
{
"document_path": "s3://bucket/w2.pdf",
"baseline_source": "/local/baselines/w2/"
}
]Document Type (Auto-detected):
s3://...→ S3 file (copied to InputBucket)- Absolute/relative path → Local file (uploaded to InputBucket)
Document ID (Auto-generated):
- From filename without extension
- Example:
invoice-2024.pdf→invoice-2024 - Subdirectories preserved:
W2s/john.pdf→W2s/john
Baseline Source Type (Auto-detected):
- Local path → the directory is uploaded recursively
s3://bucket/prefix/→ every object under the prefix is copied, keeping its directory shape. This is the formgenerate-manifest --test-setwrites
baseline_source naming a single S3 object copies nothing. When
idp-cli process --manifest consumes the manifest, an s3:// value is treated as a
prefix — a trailing / is appended if absent — so s3://bucket/gt/invoice.json
becomes a prefix that matches no object and the document is processed with no baseline
to score against. Point baseline_source at the directory holding the baseline files,
not at one of them.
Important:
⚠️ Duplicate filenames not allowed- ✅ Use directory structure for organization (e.g.,
clientA/invoice.pdf,clientB/invoice.pdf) - ✅ S3 URIs can reference any bucket (automatically copied)
Test different extraction prompts or configurations without redeploying the stack. Upload each attempt to one profile — every upload cuts a new revision — and process under that revision (Iterating on one profile instead of many):
# Attempt 1: upload, then process under the revision it printed (here r1)
idp-cli config-upload --stack-name my-stack --config-file ./attempt-1.yaml \
--config-profile tuning --revision-notes "baseline prompts"
idp-cli run-inference --stack-name my-stack --dir ./test-set/ --batch-id tuning-r1 \
--config-profile tuning --config-revision 1 --monitor
# Download and analyze results
idp-cli download-results --stack-name my-stack --batch-id tuning-r1 --output-dir ./results-r1/
# Attempt 2: upload the change to the same profile, then process under its revision (here r2)
idp-cli config-upload --stack-name my-stack --config-file ./attempt-2.yaml \
--config-profile tuning --revision-notes "improved extraction prompt"
idp-cli run-inference --stack-name my-stack --dir ./test-set/ --batch-id tuning-r2 \
--config-profile tuning --config-revision 2 --monitor
# Compare in Athena
# SELECT batch_id, AVG(overall_accuracy) FROM evaluation_results
# WHERE batch_id IN ('tuning-r1', 'tuning-r2') GROUP BY batch_id;Process thousands of documents efficiently:
# Generate manifest for large dataset
idp-cli generate-manifest \
--dir ./production-documents/ \
--output large-batch-manifest.csv
# Validate before processing
idp-cli validate-manifest --manifest large-batch-manifest.csv
# Process in background (no --monitor flag)
idp-cli run-inference \
--stack-name production-stack \
--manifest large-batch-manifest.csv \
--batch-id production-batch-001
# Check status later
idp-cli status \
--stack-name production-stack \
--batch-id production-batch-001Integrate into automated pipelines:
#!/bin/bash
# ci-test.sh - Automated accuracy testing
# Run processing with evaluation
idp-cli run-inference \
--stack-name ci-stack \
--manifest test-suite-with-baselines.csv \
--batch-id ci-test-$BUILD_ID \
--monitor
# Download evaluation results
idp-cli download-results \
--stack-name ci-stack \
--batch-id ci-test-$BUILD_ID \
--output-dir ./ci-results/ \
--file-types evaluation
# Parse results and fail if accuracy below threshold
python check_accuracy.py ./ci-results/ --min-accuracy 0.90
# Exit code 0 if passed, 1 if failed
exit $?Stop all running workflows for a stack. Useful for halting processing during development or when issues are detected.
Usage:
idp-cli stop-workflows [OPTIONS]Options:
--stack-name(required): CloudFormation stack name--skip-purge: Skip purging the SQS queue--skip-stop: Skip stopping Step Function executions--region: AWS region (optional)
Examples:
# Stop all workflows (purge queue + stop executions)
idp-cli stop-workflows --stack-name my-stack
# Only purge the queue (don't stop running executions)
idp-cli stop-workflows --stack-name my-stack --skip-stop
# Only stop executions (don't purge queue)
idp-cli stop-workflows --stack-name my-stack --skip-purgeRun load tests by copying files to the input bucket at specified rates.
Usage:
idp-cli load-test [OPTIONS]Options:
--stack-name(required): CloudFormation stack name--source-file(required): Source file to copy (local path or s3://bucket/key)--rate: Files per minute (default: 100)--duration: Duration in minutes (default: 1)--schedule: CSV schedule file (minute,count) - overrides --rate and --duration--dest-prefix: Destination prefix in input bucket (default: load-test)--config-profile(alias:--config-version): Configuration profile to use for processing (default: active profile)--region: AWS region (optional)
Examples:
# Constant rate: 100 files/minute for 5 minutes
idp-cli load-test --stack-name my-stack --source-file samples/invoice.pdf --rate 100 --duration 5
# High volume: 2500 files/minute for 1 minute
idp-cli load-test --stack-name my-stack --source-file samples/invoice.pdf --rate 2500
# Use schedule file for variable rates
idp-cli load-test --stack-name my-stack --source-file samples/invoice.pdf --schedule schedule.csv
# Use S3 source file
idp-cli load-test --stack-name my-stack --source-file s3://my-bucket/test.pdf --rate 500
# Load test with a specific config profile
idp-cli load-test --stack-name my-stack --source-file samples/invoice.pdf --rate 100 --config-profile v2Schedule File Format (CSV):
minute,count
1,100
2,200
3,500
4,1000
5,500See lib/idp_cli_pkg/examples/load-test-schedule.csv for a sample schedule file.
Remove residual AWS resources left behind from deleted IDP CloudFormation stacks.
--dry-run first.
Intended Use: This command is designed for development and test accounts where IDP stacks are frequently created and deleted, and where the consequences of accidentally deleting resources or data are low. Do not use this command in production accounts where data retention is critical. For production cleanup, manually review and delete resources through the AWS Console.
Usage:
idp-cli remove-deleted-stack-resources [OPTIONS]How It Works:
This command safely identifies and removes ONLY resources belonging to IDP stacks that have been deleted:
- Multi-region Stack Discovery - Scans CloudFormation in multiple regions.
⚠️ The CLI's own default isus-east-1,us-west-2,eu-central-1only — narrower than the SDK'sIDP_REGIONSlist, whichdiscover_idp_stacksfalls back to when called directly — so a stack deployed anywhere else,ca-central-1included, is not scanned unless you name it in--check-stack-regions - IDP Stack Identification - Identifies IDP stacks by their Description ("AWS GenAI IDP Accelerator") or naming patterns (IDP-*, PATTERN1/2/3)
- Active Stack Protection - Tracks both ACTIVE and DELETED stacks; resources from active stacks are NEVER touched
- Safe Cleanup - Only targets resources belonging to stacks in DELETE_COMPLETE state
- Global Resource Check - CloudFront distributions, response headers policies and IAM policies are global, and the S3 bucket listing is account-wide; all of them carry only the stack's name. So before deleting one the command asks every Region enabled for the account whether a stack of that name still exists there. This protects a stack deleted in a scanned Region and recreated under the same name in one
--check-stack-regionsdoes not name. If any Region cannot be read, the resource is left in place and reported under Errors with the reason, and the command exits 1. A resource left because its stack still exists in another Region is a deliberate skip: it is recorded in the SDK result'sskippedlist and listed under Skipped in the CLI summary
Safety Features:
- Resources from ACTIVE stacks are protected and skipped
- Resources from UNKNOWN stacks (not verified as IDP) are skipped
- Interactive confirmation for each resource (unless --yes)
- Options: y=yes, n=no, a=yes to all of type, s=skip all of type
- --dry-run mode shows exactly what would be deleted
Resources Cleaned:
- CloudFront distributions and response header policies
- CloudWatch log groups
- AppSync APIs (leftovers from stacks created before AppSync was removed; current stacks create none)
- IAM policies
- CloudWatch Logs resource policy entries
- S3 buckets (automatically emptied before deletion)
- DynamoDB tables (PITR disabled before deletion)
Note: This command targets resources that remain in AWS after IDP stacks have already been deleted. These are typically resources with RetainOnDelete policies or non-empty S3 buckets that CloudFormation couldn't delete. All resources are identified by their naming pattern and verified against the deleted stack registry before deletion.
Options:
--region: Primary AWS region for regional resources (default: us-west-2)--profile: AWS profile to use--dry-run: Preview changes without making them (RECOMMENDED first step)--yes,-y: Auto-approve all deletions (skip confirmations)--check-stack-regions: Comma-separated regions to check for stacks (default:us-east-1,us-west-2,eu-central-1). Passca-central-1explicitly to sweep a Canada deployment
Permissions: besides delete permissions for the resources it cleans, the global resource check needs ec2:DescribeRegions and cloudformation:DescribeStacks in every enabled Region. A DescribeStacks grant scoped to one Region counts as missing for the others. Without them, every CloudFront distribution, response headers policy, IAM policy and S3 bucket is left in place and reported as an error, and the command exits 1.
Examples:
# RECOMMENDED: Always dry-run first to see what would be deleted
idp-cli remove-deleted-stack-resources --dry-run
# Interactive cleanup with confirmations for each resource
idp-cli remove-deleted-stack-resources
# Use specific AWS profile
idp-cli remove-deleted-stack-resources --profile my-profile
# Auto-approve all deletions (USE WITH CAUTION)
idp-cli remove-deleted-stack-resources --yes
# Check additional regions for stacks
idp-cli remove-deleted-stack-resources --check-stack-regions us-east-1,us-west-2,eu-central-1,eu-west-1CloudFront Two-Phase Cleanup:
CloudFront requires distributions to be disabled before deletion:
- First run: Disables orphaned distributions (you confirm each)
- Wait 15-20 minutes for CloudFront global propagation
- Second run: Deletes the previously disabled distributions
Interactive Confirmation:
Delete orphaned CloudFront distribution?
Resource: E1H6W47Z36CQE2 (exists in AWS)
Originally from stack: IDP-P2-DevTest1
Stack status: DELETE_COMPLETE (stack no longer exists)
Stack was in region: us-west-2
Options: y=yes, n=no, a=yes to all CloudFront distribution, s=skip all CloudFront distribution
Delete? [y/n/a/s]:
Important Limitation - 90-Day Window:
CloudFormation only retains deleted stack information for approximately 90 days. After this period, stacks in DELETE_COMPLETE status are removed from the CloudFormation API.
This means:
- Resources from stacks deleted within the past 90 days → Identified and offered for cleanup
- Resources from stacks deleted more than 90 days ago → Not identified (silently skipped)
Best Practice: Run remove-deleted-stack-resources promptly after deleting IDP stacks to ensure complete cleanup. For maximum effectiveness, run this command within 90 days of stack deletion.
Generate an IDP configuration template from system defaults.
Usage:
idp-cli config-create [OPTIONS]Options:
--features: Feature set (default:min)min: classification and extraction only (simplest). It also writesenabled: falsefor summarization and evaluation, so those are switched off. It leaves confidence scoring on: confidence is configured underextraction.confidence, which comes withextractionat its defaultmode: separate. To turn it off, setextraction.confidence.mode: offin the generated filecore: the sections of min plus ocr. Unlike min, it does not turn summarization or evaluation off. Confidence scoring is configured underextraction.confidence, so it comes withextractionall: every section at its default values, excepthitl,chatandrule_validation.⚠️ On upload the file is merged over the stack's preset or the profile's current configuration, so every setting it writes is reset to its default there:summarizationandevaluationare writtenenabled: true, for example, and switch back on where a preset turned them off. Without--include-promptsthe long prompt fields are not written, so those keep the deployment's own values. Edit the file before uploading, or ask for only the sections you mean to change. The three sections are left out so that the reset does not also switch human review, the chat companion or rule validation on or off. Name them explicitly to include them (--features hitl, or a comma-separated list of section names); a feature set cannot be combined with section names, soallplus one of them means listing every section- Or one or more section names, comma-separated:
summarization, or"classification,extraction,summarization". A single name and a list are read the same way. A name that is not a section of the configuration is refused with exit 1 and the list of valid sections, rather than dropped, and a feature set cannot be combined with section names
- No feature set writes a
classessection. The system defaults define no document classes, and an emptyclasses: []uploaded over a profile would replace the profile's classes with none, so the key is left out and the profile keeps its own.--features classeson its own is refused for the same reason. Add your classes under a top-levelclasses:key - The header comment of the generated file states which sections it copies, which it switches off, which it leaves out, and whether confidence scoring is on
--output,-o: Output file path (default: stdout)--include-prompts: Include full prompt templates. By default the longsystem_prompt,task_promptanduser_promptvalues are left out of the file, so they take whatever value the configuration is merged over: the system defaults when validating or deploying, or the stack's default (or the profile's current) configuration when uploading--no-comments: Omit explanatory header comments
Examples:
# Generate minimal config to stdout
idp-cli config-create
# Generate a config with every section except hitl, chat and rule_validation
idp-cli config-create --features all --output full-config.yaml
# Custom section selection
idp-cli config-create --features "classification,extraction,summarization" --output config.yaml
# A single section
idp-cli config-create --features summarizationValidate a configuration file against system defaults and Pydantic models. Catches common configuration errors that cause silent pipeline failures.
Validation Checks:
- YAML/JSON Syntax - Ensures file is well-formed
- Schema Validation - Validates against Pydantic models
- Model IDs - Verifies every Bedrock model ID the configuration names is in
config_library/pricing.yaml: each section's model, the confidence, escalation, review-agent, chat, agent, discovery, rule-validation and evaluation models, the per-classx-aws-idp-extraction-model,x-aws-idp-extraction-escalation-modelandx-aws-idp-confidence-escalation-modeloverrides, and themodelin thex-aws-idp-evaluation-method-configof any field evaluated withLLM.config-uploadruns the same check unless given--no-validate. An ID is compared as written, so one with surrounding spaces or in upper case is reported like any other typo. Embedding-model fields are not checked, becausepricing.yamlprices no embedding model. A model named by an ARN whose underlying model cannot be determined offline is a warning rather than an error - Placeholder Validation - Ensures required placeholders are present in custom task_prompts:
ocr.task_prompt(bedrock only): Requires{DOCUMENT_IMAGE}- The task prompt of each classification call the strategy makes (for example
classification.combined.page.task_prompt): Requires{DOCUMENT_TEXT}OR{DOCUMENT_IMAGE} extraction.task_prompt: Requires{DOCUMENT_TEXT}OR{DOCUMENT_IMAGE}extraction.confidence.task_prompt(unlessextraction.confidence.modeisoff): Requires{EXTRACTION_RESULTS}, and at least one of{DOCUMENT_TEXT},{OCR_TEXT_CONFIDENCE}or{DOCUMENT_IMAGE}.{DOCUMENT_IMAGE}is not required on its own, because leaving it out is how the confidence pass is made text-only. A pre-0.6assessment.task_promptis checked at this path, where the migration moves it.{DOCUMENT_IMAGE}may appear at most once in the prompt the confidence pass sends — this prompt with comments removed, plusextraction.geometry.task_prompt_bboxwhenextraction.geometry.modeisllmorllm_grounded— because the pass refuses the document otherwise. An explicit empty prompt (task_prompt: "", or one holding only a comment) is refused for the same reason. Both checks run inintegratedmode too, where the separate pass still runs for some sections; leave the key out to use the system defaultsummarization.task_prompt: Requires{DOCUMENT_TEXT},{EXTRACTION_RESULTS}
- JSON Schema Fields - Warns about non-standard fields (e.g.,
data_type) - Pre-0.9 classification layout - Warns when the file still uses the flat
classification keys, naming the ones it converted, so you can save the nested
form and keep it. A second warning appears for the one conversion that
changes behaviour: a
maxPagesForClassificationpage limit combined with boundary detection is converted tosectionSplitting: disabled, after which a short packet becomes one section where it used to be split. Both are warnings, not errors — the file is valid and will run. - OpenAI GPT-5.x compatibility - Errors if an
openai.gpt-5.*model is paired with agentic extraction, either asextraction.modelor as a per-classx-aws-idp-extraction-modeloverride. The escalation model is checked the same way —extraction.validation.escalation_modelor a per-classx-aws-idp-extraction-escalation-model— when escalation can run, meaning validation is enabled andextraction.validation.fail_actionisescalate. Extraction is agentic whenextraction.modeisadvanced, or, in a configuration that sets nomode, whenextraction.agentic.enabledistrue;modewins when both are set. It also errors if one is used for Discovery (discovery.*.model_id/discovery.rules.model). These models run on thebedrock-mantleResponses API and cannot accept the Strands agent loop or whole-PDF document blocks. See OpenAI GPT-5.x Models. - xAI Grok compatibility - Errors if
us.xai.grok-4.6/global.xai.grok-4.6is used for Discovery (discovery.*.model_id/discovery.rules.model), because Grok rejects whole-PDFdocumentblocks. Unlike GPT-5.x, Grok is valid for agentic extraction — it reaches Converse and supports tool use, so that pairing is deliberately allowed. See xAI Grok Models. - OpenAI GPT-6 Astra compatibility - Errors if
us.openai.gpt-6-astra/global.openai.gpt-6-astrais used for Discovery (discovery.*.model_id/discovery.rules.model), because Astra rejects whole-PDFdocumentblocks. Like Grok — and unlike its GPT-5.x stablemates — Astra is valid for agentic extraction, since it reaches Converse and emitstoolUse. The agentic check keys on the bedrock-mantle route rather than theopenai.prefix, so one OpenAI model passes it and the others do not. See OpenAI Models.
Usage:
idp-cli config-validate [OPTIONS]Options:
--config-file,-f(required): Path to configuration file to validate--show-merged: Show the full merged configuration--strict: Fail validation if config contains unknown or deprecated fields--emit-migrated <path>: Write the file converted to the current configuration format to<path>, headed by a comment naming the format version and the source file
Examples:
# Validate a config file
idp-cli config-validate --config-file ./my-config.yaml
# Show full merged config
idp-cli config-validate --config-file ./config.yaml --show-merged
# Strict mode (fails if config has unknown or deprecated fields — useful for CI/CD)
idp-cli config-validate --config-file ./config.yaml --strictNotes:
- The
config-uploadcommand runs validation by default before uploading to protect production stacks. - The command calls the SDK's
client.config.validate()and, for--emit-migrated,client.config.migrate(), so the CLI and the SDK give the same verdict on a file. See config.validate(). - Model ID validation requires
config_library/pricing.yaml. Ensureidp-cliis run from the repository root, or set theIDP_PROJECT_ROOTenvironment variable to the repo root for validation to work correctly.
Download configuration from a deployed IDP stack.
Usage:
idp-cli config-download [OPTIONS]Options:
--stack-name(required): CloudFormation stack name--output,-o: Output file path (default: stdout)--format: Output format -full(default) orminimal(only differences from defaults)--config-profile(alias:--config-version): Configuration profile to download (e.g.,default,lending). If not specified, downloads the active profile--config-revision: Download an exact revision of that profile instead of its current configuration (e.g.7). Requires--config-profile. Fails if the revision is no longer retained, or if its body has expired underDataRetentionInDays, unless it is the profile's current published revision and the profile still provably holds that configuration, in which case it is rebuilt from the profile (Retention), rather than silently returning the current configuration. The file is written in the current configuration format whichever release cut the revision, so it has the same shape as a download of the profile's current configuration (Revision History)--region: AWS region (optional)
Examples:
# Download full config from the active profile
idp-cli config-download --stack-name my-stack --output config.yaml
# Download a specific profile
idp-cli config-download --stack-name my-stack --config-profile lending --output config.yaml
# Download minimal config (only customizations)
idp-cli config-download --stack-name my-stack --format minimal --output config.yaml
# Print to stdout
idp-cli config-download --stack-name my-stack
# Download an exact revision — what an earlier run actually used
idp-cli config-download --stack-name my-stack --config-profile lending \
--config-revision 7 --output r7.yaml--config-profile used to exit 0 having written the YAML null document — so
config-download --config-profile lendnig > config.yaml left a file every downstream
step reads as an empty configuration, under an exit code that said it worked. All
three spellings (stdout, --output, --format minimal) now exit 1, name the profile
they could not find, and write no file. A script that swallowed the exit code and
carried on with the downloaded file will now be handed nothing instead of an empty
configuration.
Upload a configuration file to a deployed IDP stack.
Note: Configurations with
managed: truecannot be uploaded via CLI. Managed configurations are stack-controlled and automatically overwritten during stack updates. Removemanaged: truefrom your configuration file before uploading.
Usage:
idp-cli config-upload [OPTIONS]Options:
--stack-name(required): CloudFormation stack name--config-file,-f(required): Path to configuration file (YAML or JSON)--validate/--no-validate: Validate config before uploading (default: validate)--config-profile(alias:--config-version) (required): Configuration profile to update (e.g.,default,Production). If the profile doesn't exist, it will be created automatically. An empty value is refused with exit 1 and nothing is written; it used to land the configuration on a key no profile listing can see, reported as "Configuration is now active!" with exit 0--version-description: Description for the configuration profile (persisted on the profile and overwritten by every save)--revision-notes: What this upload changed, recorded on the revision it cuts and shown as Notes in the revision history (e.g.'raised topK to 20'). Per-revision and immutable, unlike--version-description--region: AWS region (optional)
Examples:
# Upload config to the active profile
idp-cli config-upload --stack-name my-stack --config-file ./config.yaml --config-profile default
# Update an existing profile
idp-cli config-upload --stack-name my-stack --config-file ./config.yaml --config-profile Production
# Create a new profile with a description
idp-cli config-upload --stack-name my-stack --config-file ./config.yaml --config-profile NewProfile --version-description "Test configuration for new feature"
# Skip validation (use with caution)
idp-cli config-upload --stack-name my-stack --config-file ./config.yaml --no-validateOutput: on success the command prints the revision the upload produced, with the flags to process under it:
✓ Configuration uploaded successfully
Configuration profile 'lending' updated!
Revision: r7
Process under it with: --config-profile lending --config-revision 7
Nothing is printed when the stack has no revision history. A save that changes nothing records no new revision, so the number printed is then the revision already current — which is still the correct one to pin.
List all configuration profiles in a deployed IDP stack.
Usage:
idp-cli config-list [OPTIONS]Options:
--stack-name(required): CloudFormation stack name--region: AWS region (optional)
Examples:
# List all configuration profiles
idp-cli config-list --stack-name my-stackOutput:
Shows a table with profile names, active status, the current revision (Rev),
creation/update timestamps, and descriptions. Rev is the revision each profile's
configuration currently reflects — the value to pass to --config-revision. It is
blank for a profile with no history.
List the revision history of a Configuration Profile.
Every save of a profile cuts an immutable revision. This shows the ones still retained: the last 20, plus anything labeled, pinned by a test run, or currently in use. See configuration-profiles.md.
A listed revision is not necessarily one you can still download. A label or a
test-run pin keeps a revision listed, but every revision's body expires
DataRetentionInDays after it was cut, and after that only the profile's current
published revision can still be downloaded, and only while the profile provably
still holds that configuration (Retention).
Usage:
idp-cli config-revisions [OPTIONS]Options:
--stack-name(required): CloudFormation stack name--config-profile(alias:--config-version) (required): Profile whose history to list--json: Emit JSON instead of a table, for scripting--region: AWS region (optional)
Examples:
# Revision history of a profile
idp-cli config-revisions --stack-name my-stack --config-profile lending
# Machine-readable — e.g. pick the current revision
idp-cli config-revisions --stack-name my-stack --config-profile lending --json \
| jq -r '.revisions[] | select(.published) | .revision'Output: one row per retained revision, newest first, with Rev, which one the
profile currently reflects, when and by whom it was cut, its label and notes, and
why it is exempt from pruning (labeled / test run). A profile with no
history reports that and exits 0 — a profile untouched since the stack was
upgraded genuinely has none.
config-upload (which prints the revision it produced), config-revisions, and
--config-revision on config-download / process / run-inference together
let an automated tuning loop keep one profile and track its attempts as
revisions:
# Upload attempt N and capture the revision it became
rev=$(idp-cli config-upload --stack-name my-stack --config-file attempt.yaml \
--config-profile tuning-run-42 --revision-notes "raised topK to 20" \
| sed -n 's/^Revision: r//p')
# Score exactly that revision
idp-cli run-inference --stack-name my-stack --test-set my-tests \
--config-profile tuning-run-42 --config-revision "$rev" --monitor
# Later: retrieve the configuration that produced the best score
idp-cli config-download --stack-name my-stack --config-profile tuning-run-42 \
--config-revision 7 --output best.yamlNaming a new profile per attempt also works, but every one of them then appears in the profile pickers and access-control scope lists of the whole deployment.
Activate a configuration profile in a deployed IDP stack.
Automatic BDA Sync: If the configuration profile has use_bda enabled, this command will automatically sync the configuration to BDA (Bedrock Data Automation) before activation. This ensures BDA blueprints are up-to-date and matches the UI behavior.
Usage:
idp-cli config-activate [OPTIONS]Options:
--stack-name(required): CloudFormation stack name--config-profile(alias:--config-version) (required): Configuration profile to activate--region: AWS region (optional)
Examples:
# Activate a specific profile
idp-cli config-activate --stack-name my-stack --config-profile v2
# Activate the default profile
idp-cli config-activate --stack-name my-stack --config-profile defaultBehavior:
- Validates the configuration profile exists
- If
use_bdais enabled in the configuration:- Syncs IDP document classes to BDA blueprints
- Creates a new BDA project if none exists
- Updates BDA sync status
- Activates the configuration profile
- All new document processing will use this configuration
Note: If BDA sync fails (when use_bda is enabled), the activation will be aborted to prevent processing errors.
An aborted activation can still have left a blueprint behind. The BDA sync this
command runs is the same replace-mode sync as
config-sync-bda, so the same thing can happen: a blueprint removed
from the BDA project that could not then be deleted. The deletes happen whatever became
of the document classes, which makes an aborted activation the outcome most likely to
have left one. Those ARNs are printed whether the activation succeeded or failed, and
they are not counted as failed classes — the remedy is the orphaned-blueprint cleanup,
config-sync-bda --direction cleanup-orphaned, not a
re-run of this command. See the config-sync-bda section for the full explanation.
Notes:
- Sets the specified profile as active for all new document processing
- Profile must exist (use
config-listto see available profiles)
Delete a configuration profile from a deployed IDP stack.
Usage:
idp-cli config-delete [OPTIONS]Options:
--stack-name(required): CloudFormation stack name--config-profile(alias:--config-version) (required): Configuration profile to delete--force: Skip confirmation prompt--region: AWS region (optional)
Examples:
# Delete a profile with confirmation
idp-cli config-delete --stack-name my-stack --config-profile old-profile
# Delete without confirmation prompt
idp-cli config-delete --stack-name my-stack --config-profile old-profile --forceRestrictions:
- Cannot delete the 'default' configuration profile
- Cannot delete the currently active profile (activate another profile first)
- Includes confirmation prompt unless
--forceis used
What Happens:
- Loads and parses your YAML or JSON config file
- Validates against system defaults (unless
--no-validate) - If the profile exists: Updates it with the uploaded configuration (saved as a complete snapshot)
- If the profile doesn't exist: Creates a new profile with the uploaded configuration
- Uploads to the stack's ConfigurationTable in DynamoDB
- Configuration is immediately available for document processing
Configuration Profiles:
- Existing profile: Saves the uploaded configuration as the full profile snapshot
- New profile: Creates a new independent profile with the uploaded configuration
- Profile descriptions: Can be added to new profiles for better organization
--config-revision <n> pins an exact revision of --config-profile on process
and run-inference; omit it to process under the profile's current configuration.
For full details on configuration profiles and their revisions, see configuration-profiles.md.
This uses the same mechanism as the Web UI configuration management system.
Get test results for a specific Test Studio test run with automatic evaluation triggering.
Usage:
idp-cli test-result [OPTIONS]Options:
--stack-name(required): CloudFormation stack name--test-run-id(required): Test run ID to retrieve results for--wait: Wait for evaluation to complete (polls until metrics are calculated)--timeout: Timeout in seconds when using--wait(default: 600)--output-dir: Directory to save results as JSON file--region: AWS region (optional)
Examples:
# Get results immediately (may show "EVALUATING" status if metrics not ready)
idp-cli test-result \
--stack-name my-stack \
--test-run-id fake-w2-20260409-123456
# Wait for evaluation to complete (recommended for CI/CD)
idp-cli test-result \
--stack-name my-stack \
--test-run-id fake-w2-20260409-123456 \
--wait --timeout 900
# Save results to JSON file
idp-cli test-result \
--stack-name my-stack \
--test-run-id fake-w2-20260409-123456 \
--wait --output-dir ./resultsExit codes: 1 when the run's status is FAILED, or when any file failed
whatever the status is (a PARTIAL_COMPLETE run with failures exits 1). 0
otherwise, which includes the in-flight states — EVALUATING, IN_PROGRESS, QUEUED —
and, for now, ABORTED with no failed files. Read the printed Status: line rather
than the exit code if you need to distinguish "passed" from "has not finished". The
results are printed before the exit either way, so you still get the accuracy figures
for a failed run.
0 for a run with status="FAILED" and every file
failed, exactly like a clean pass, so idp-cli test-result ... && deploy proceeded on a
failed evaluation. A CI job that relied on that will now stop, which is the point.
Output:
- Overall accuracy, precision, recall, F1 score
- Total cost
- Files completed/failed
- Created/completed timestamps
- JSON file:
<test-run-id>-result.json(when--output-dirspecified)
Behavior:
- Triggers lazy evaluation if metrics not yet calculated (first call after test run completes)
- Polls Lambda every 10 seconds when
--waitis used - Returns complete test run data including field-level metrics and cost breakdown
Abort one or more running Test Studio test runs. Stops all document processing workflows and preserves results from completed documents.
Usage:
idp-cli abort-test-run [OPTIONS]Options:
--stack-name(required): CloudFormation stack name--test-run-ids(required): Comma-separated list of test run IDs to abort--force/-y: Skip confirmation prompt--region: AWS region (optional)
Examples:
# Abort a single test run
idp-cli abort-test-run \
--stack-name my-stack \
--test-run-ids "fake-w2-20260409-123456"
# Abort multiple test runs
idp-cli abort-test-run \
--stack-name my-stack \
--test-run-ids "run1,run2,run3"
# Skip confirmation prompt
idp-cli abort-test-run \
--stack-name my-stack \
--test-run-ids "fake-w2-20260409-123456" \
--forceOutput:
- Success/failure count for each test run
- Error details for failed aborts (e.g., test run not found, already completed)
- Confirmation prompt unless
--forceis used
Behavior:
- Only test runs with status QUEUED or RUNNING can be aborted
- Completed documents are preserved with their evaluation results
- Test run status is updated to ABORTED
- Metrics are calculated for any completed documents
- Document workflows are stopped via Step Functions execution abort
Limitations:
- Cannot abort test runs with status EVALUATING, COMPLETE, PARTIAL_COMPLETE, or FAILED
- The abort operation waits up to 25 seconds for documents to reach terminal state
Compare metrics and configurations from multiple Test Studio test runs.
Usage:
idp-cli test-compare [OPTIONS]Options:
--stack-name(required): CloudFormation stack name--test-run-ids(required): Comma-separated list of test run IDs to compare (minimum 2)--output-dir: Directory to save comparison as JSON and CSV files--region: AWS region (optional)
Examples:
# Compare two test runs
idp-cli test-compare \
--stack-name my-stack \
--test-run-ids "fake-w2-20260409-123456,fake-w2-20260409-234567"
# Compare multiple runs and export to files
idp-cli test-compare \
--stack-name my-stack \
--test-run-ids "run1,run2,run3" \
--output-dir ./comparisonsOutput:
- Console: Side-by-side table with accuracy, precision, recall, F1 score, and cost for each test run
- JSON file:
comparison-<timestamp>.json- Complete comparison data with full test results and config differences - CSV file:
comparison-<timestamp>.csv- Metrics table suitable for spreadsheets
Configuration Differences:
- Automatically detects and displays configuration differences between test runs
- Shows nested config paths (e.g.,
classification.combined.llm.model,extraction.temperature) - Highlights values that differ across test runs
Requirements:
- All test runs must be in
COMPLETEorPARTIAL_COMPLETEstatus - Minimum 2 test runs required for comparison
Manage MCP service credentials: client-credentials apps that an MCP client signing
in as a service (a shared Amazon Quick workflow, the MCP Connector, a scheduled
job) uses instead of a person's sign-in. Each credential has its own role,
configuration profiles, expiry and secret; the rules are the ones the MCP
Configuration page applies (see MCP Server).
Every subcommand takes --stack-name and an optional --region, and needs IAM
access to the stack's credential table, user pool, Secrets Manager and gateway.
| Subcommand | Purpose |
|---|---|
list [--json] |
Print the credentials. Secrets are never listed |
create --name N --role ROLE [--scope PROFILE …] [--write-tools] [--expires-in-days D] |
Create one and print, as JSON on stdout, its client ID and secret (shown once; also stored in Secrets Manager at secretArn), the token URL and the MCP server endpoint. --scope repeats; --expires-in-days 0 means never (default 90) |
update --client-id ID [--role ROLE] [--scope PROFILE …] [--clear-scope] [--write-tools/--no-write-tools] [--enable/--disable] |
Change role, profiles, write tools or enabled; never the secret |
rotate --client-id ID [--overlap-days 0-7] |
Issue a replacement with the same access, print its secret once, and keep the old one working for the overlap (default 7 days) |
revoke --client-id ID [--yes] |
Refuse the credential from its next request and delete its app client and secret |
set-tenant --client-id ID [--tenant-id T] |
Grant one tenant, or none; only when TenantGrantSource is Api |
Manage tenant declarations and check tenant isolation on a deployed stack. A tenant
is a set of Input-bucket prefixes plus a Configuration Profile pattern; see
Multi-Tenant Isolation for what declaring one enforces and what
TenantIsolationMode=Strict adds. Every subcommand takes --stack-name and an
optional --region.
| Subcommand | Purpose |
|---|---|
list [--json] |
Print the declared tenants |
put --tenant-id ID --prefix P [--prefix P2 …] --profile-pattern PAT [--granted-by V …] [--description D] |
Create or replace one tenant. Refused when it overlaps another tenant or an existing prefix mapping would cross it — the same checks the web UI applies. --granted-by (repeatable, at most three) names the claim or entitlement values that grant the tenant; a value may grant only one tenant |
set-user-tenant --email ADDR [--tenant ID] [--json] |
Grant one existing user one tenant, or none when --tenant is omitted. Only when the stack's TenantGrantSource is Api; an Admin is refused. See Who grants a tenant |
delete --tenant-id ID [--yes] |
Remove one declaration. Mappings, profiles and documents are untouched |
apply --file tenants.yaml [--prune] [--yes] [--json] |
Make the declarations match a file. Prints the plan and writes nothing unless --yes is given; --prune also deletes tenants the file does not declare. The whole plan is validated before anything is written |
generate-policies --principals principals.yaml [--output FILE] [--deploy [--access-point-stack NAME] [--json]] |
Write a CloudFormation template creating one Input-bucket and one Output-bucket S3 Access Point per tenant, confined to that tenant's prefixes. --deploy also deploys it as its own stack (default <stack>-tenant-access-points, created or updated) and prints each tenant's access point alias and ARN; see the procedure |
audit [--principals principals.yaml] [--sample N] [--json] |
Read-only report; exits 1 when there are findings |
verify --document FILE [--tenant A --tenant B] [--timeout S] [--email-domain D] [--json] |
Live cross-tenant check; exits 0 pass, 1 fail, 2 error, 3 skipped |
tenants.yaml:
tenants:
- tenantId: acme
prefixes: [clients/acme/]
profilePattern: acme-*
grantedBy: [tenant.acme] # optional; read when TenantGrantSource is IdpClaim
description: ACME Corp
- tenantId: globex
prefixes: [clients/globex/, feeds/globex/]
profilePattern: globex-*principals.yaml names, per tenant, the IAM roles or users that write its
documents and read its results. An account-root ARN is refused, because an access
point cannot confine a whole account to one tenant:
acme:
uploaders: [arn:aws:iam::123456789012:role/acme-uploader]
consumers: [arn:aws:iam::123456789012:role/acme-results-reader]
kmsKeyArn: arn:aws:kms:us-east-1:123456789012:key/EXAMPLE # optionalIn AWS GovCloud (US) the ARNs use the aws-us-gov partition (arn:aws-us-gov:iam::…).
Output-bucket keys begin with the document's input key, so a tenant's output access
point is confined to the same prefixes as its input one. The generated template
creates only the access points. The stack's bucket policies delegate to access points
in its own account when TenantIsolationMode=Strict. Each access point policy confines
with explicit denies — no other principal may use it, and nothing outside the tenant's
prefixes may be reached through it by any action — so the confinement holds whatever the
roles' own policies allow. A role in the stack's account needs no S3 permission of its
own; a role in another account needs an identity policy allowing its own access point.
Every role needs use of the stack's KMS key. The command prints what is needed.
audit reports the isolation mode, prefix mappings that cross a tenant boundary,
tenant prefixes no mapping of the tenant's own covers, non-admin users with no scope
or one spanning tenants, a sample of tracked documents whose key and profile belong
to different tenants, and — with --principals — deployed access point policies that
differ from what generate-policies would produce.
verify creates a throwaway Admin and two throwaway Viewer users scoped to two
tenants (through the API's own createUser, so the scope check runs), uploads the
given document under each tenant's prefix, and checks that each user can list, read
and download its own tenant's document and is refused the other's. On a Strict
stack it also uploads a document outside every tenant, which must be refused at
ingest, and one under the first tenant's prefix whose metadata names the second
tenant's profile, which must be refused or processed under the first tenant's own
profile, never the second's. It temporarily enables
admin password authentication on the app client and restores the original setting,
and it removes every user and object it created; a cleanup failure is reported as an
error. Each tenant needs an enabled prefix mapping to one of its own profiles before
it runs. The throwaway users' addresses use the stack's first AllowedSignUpEmailDomain
when that parameter is set, because the API refuses to create a user outside it, and
example.com otherwise; --email-domain overrides both.
Discover document class schemas from sample documents using Amazon Bedrock.
Two modes:
- Stack-connected (
--stack-name): Uses stack's discovery config and saves schema to DynamoDB configuration - Local (no
--stack-name): Uses system default Bedrock settings, prints schema to stdout without saving
Ground truth matching: Ground truth files (-g) are auto-matched to documents (-d) by filename stem. For example, invoice.pdf matches invoice.json. Unmatched documents run without ground truth.
- Single document + single ground truth: When exactly one
-dand one-gare provided, they are paired by position regardless of filename stem. This supports the common case where ground truth files have generic names (e.g.,baseline/<doc>/sections/1/result.json). - Batch mode (multiple
-dor multiple-g): Files are matched by stem. If any-gfile cannot be matched to a document,discoverexits non-zero with a clear error. This prevents silently running without-GT discovery when the user explicitly requested ground-truth-guided discovery. See issue #310.
Output behavior:
- Single document:
-owrites the schema to the specified file - Batch +
-ois a directory (or has no extension): writes one{class_name}.jsonper schema - Batch +
-ois a file: writes all schemas as a JSON array
# Single document (local mode — no stack needed)
idp-cli discover -d ./invoice.pdf
# With ground truth (matched by filename stem)
idp-cli discover -d ./invoice.pdf -g ./invoice.json
# Save schema to file
idp-cli discover -d ./form.pdf -o ./form-schema.json
# With class name hint (guides LLM to use specific class name)
idp-cli discover -d ./form.pdf --class-hint "W2 Tax Form"
# Batch with auto-matched ground truth
idp-cli discover -d ./invoice.pdf -d ./w2.pdf -g ./invoice.json -g ./w2.json
# Batch output to directory (one file per schema)
idp-cli discover -d ./invoice.pdf -d ./w2.pdf -o ./schemas/
# Batch output to single file (JSON array)
idp-cli discover -d ./invoice.pdf -d ./w2.pdf -o ./all-schemas.json
# Multi-section: discover specific page ranges from a single PDF
idp-cli discover -d ./lending_package.pdf \
--page-range "1-2" --page-label "Cover Letter" \
--page-range "3-5" --page-label "W2 Form" \
--page-range "6-8" --page-label "Bank Statement" \
-o ./schemas/
# Auto-detect sections then discover each
idp-cli discover -d ./lending_package.pdf --auto-detect -o ./schemas/
# Only detect section boundaries (no discovery)
idp-cli discover -d ./lending_package.pdf --auto-detect --detect-only
# Auto-detect with output to file
idp-cli discover -d ./lending_package.pdf --auto-detect --detect-only -o sections.json
# Stack mode (saves to config)
idp-cli discover --stack-name my-stack -d ./invoice.pdf --config-profile v2
# Override the Bedrock model (e.g. use Claude Opus instead of the configured default)
idp-cli discover -d ./invoice.pdf -g ./invoice.json \
--model-id us.anthropic.claude-opus-4-6-v1| Option | Description |
|---|---|
--stack-name |
CloudFormation stack name (optional — omit for local mode) |
-d, --document |
Path to document file (required, repeatable for batch) |
-g, --ground-truth |
Path to JSON ground truth file(s) (repeatable, auto-matched by filename stem) |
--config-profile (alias: --config-version) |
Configuration profile to save to (stack mode only) |
-o, --output |
Output path: file (single/JSON array) or directory (one file per schema) |
--class-hint |
Hint for the document class name (e.g., "W2 Form"). The LLM will use this as $id. |
--page-range |
Page range to discover (e.g., "1-3"). Repeatable for multi-section. Requires PDF. |
--page-label |
Label for corresponding --page-range (e.g., "W2 Form"). Used as class name hint per range. Optional per range; a label with no range is refused. |
--auto-detect |
Auto-detect document section boundaries using AI, then discover each section. Cannot be combined with --page-range, --page-label, -g or --class-hint. |
--detect-only |
Only detect section boundaries. Requires --auto-detect. Prints boundaries without running discovery. |
--model-id |
Override the Bedrock model ID used for discovery (e.g., us.anthropic.claude-opus-4-6-v1). When omitted, the discovery model from the stack config (stack mode) or system defaults (local mode) is used. Applies to with-ground-truth, without-ground-truth, --auto-detect, and --page-range modes. |
--region |
AWS region |
Combinations that are refused. Discovery is paid work and its output is written to disk and consumed as configuration, so a combination that cannot be honoured exits non-zero before any Bedrock call rather than proceeding with part of what was asked for:
| Given | Why it is refused |
|---|---|
--auto-detect with -g / --class-hint |
This mode infers one class per detected section and applies neither, so the run would cost the same and disregard them. Drop --auto-detect to discover the whole document, where both apply. |
--auto-detect with --page-range |
Both decide where the sections are, and there is no basis on which to prefer one. Give one. |
--detect-only without --auto-detect |
Without it, a full schema inference ran instead of boundary detection — a more expensive operation than the one asked for. |
More --page-label than --page-range |
Labels pair with ranges in order, so the extras had no range and their class-name hints were lost. Fewer labels than ranges is fine: a label is optional per range. Under --auto-detect the remedy is to drop the label, not to add a range — that mode names each section itself. |
Filenames in directory mode. When -o names a directory, each schema is
written as <class id>.json, where the class id is the schema's $id (falling
back to x-aws-idp-document-type, then to unknown) reduced to the character
set every consumer of a class id accepts — [a-zA-Z0-9_-]. Anything else becomes
a hyphen, runs of hyphens collapse to one, and leading and trailing hyphens are
trimmed, so a class id of Bank Statement is written as Bank-Statement.json
and one with no usable character in it at all as unknown.json. That is the same
rule the discovered class gets when it is saved to a configuration profile, so
the two agree on the name.
The class id is generated by the model from the content of the document, and a filename is not the right place to trust it: without that reduction a class id spelling a path would name a file somewhere else entirely. The directory each file is written into is therefore checked to be the one you named, and the command errors rather than writing outside it. A symlink you placed at a target filename inside that directory is followed as usual — what is checked is the directory, not the file.
The command prints the path it wrote to, and prints a note when a class id had to be rewritten to produce it.
Discover document classes from a collection of documents using embedding-based clustering and agentic analysis.
Unlike discover (which analyzes one document at a time), discover-multidoc analyzes a directory of mixed documents to automatically identify document types, cluster similar documents, and generate JSON Schemas for each discovered class.
Requires: pip install -e "lib/idp_common_pkg[multi_document_discovery]" (scikit-learn, scipy, numpy, strands-agents)
Note: Requires at least 2 documents per expected class. Clusters with fewer than 2 documents are filtered as noise. For discovering schemas from individual documents, use discover instead.
Usage:
idp-cli discover-multidoc [OPTIONS]Options:
| Option | Description |
|---|---|
--dir |
Directory containing documents to analyze (recursive scan) |
-d, --document |
Individual document files (repeatable: -d doc1.pdf -d doc2.png) |
--embedding-model |
Bedrock embedding model ID (default: us.cohere.embed-v4:0; eu.cohere.embed-v4:0 in EU Regions, global.cohere.embed-v4:0 in Canada) |
--analysis-model |
Bedrock LLM for cluster analysis (default: us.anthropic.claude-sonnet-4-6; eu.anthropic.claude-sonnet-4-6 in EU Regions) |
-o, --output |
Output directory for discovered JSON schemas |
--stack-name |
CloudFormation stack name (required for --save-to-config) |
--config-profile (alias: --config-version) |
Configuration profile to save schemas to |
--save-to-config |
Save discovered schemas to the stack's configuration |
--region |
AWS region |
Examples:
# Discover from a directory of documents
idp-cli discover-multidoc --dir ./samples/
# Discover with explicit files
idp-cli discover-multidoc -d doc1.pdf -d doc2.png -d doc3.jpg
# Save schemas to output directory
idp-cli discover-multidoc --dir ./samples/ -o ./schemas/
# Save to stack configuration
idp-cli discover-multidoc --dir ./samples/ --save-to-config \
--stack-name IDP --config-profile v2
# Use custom models
idp-cli discover-multidoc --dir ./samples/ \
--embedding-model us.amazon.titan-embed-image-v1 \
--analysis-model us.anthropic.claude-sonnet-4-6Pipeline stages (shown in Rich progress output):
- Document scan — Finds PDF, PNG, JPG, TIFF files in the directory
- Embedding — Generates image embeddings via Bedrock (Cohere Embed v4)
- Clustering — KMeans + silhouette analysis to find optimal number of clusters
- Analysis — Strands agent analyzes each cluster to identify the document class and generate a JSON Schema
- Reflection — Agent generates a summary report of all discovered classes
Output: Results table showing cluster ID, classification, document count, field count, and status. Optionally writes individual JSON schema files and a reflection report.
Synchronize IDP document class schemas with BDA (Bedrock Data Automation) blueprints.
Usage:
idp-cli config-sync-bda [OPTIONS]Options:
--stack-name(required): CloudFormation stack name--direction: Sync direction —bidirectional(default),bda-to-idp,idp-to-bda, orcleanup-orphaned(not a sync — see below)--mode: Sync mode —replace(default, full alignment) ormerge(additive, don't delete). Not read bycleanup-orphaned--config-profile(alias:--config-version): Configuration profile to sync (default: active profile)--force: Skip the confirmation prompt. Onlycleanup-orphanedprompts--region: AWS region (optional)
Examples:
# Bidirectional sync (default)
idp-cli config-sync-bda --stack-name my-stack
# Import BDA blueprints into IDP config
idp-cli config-sync-bda --stack-name my-stack --direction bda-to-idp
# Push IDP classes to BDA blueprints
idp-cli config-sync-bda --stack-name my-stack --direction idp-to-bda
# Merge mode (additive — don't remove existing items)
idp-cli config-sync-bda --stack-name my-stack --direction bda-to-idp --mode merge
# Sync specific config profile
idp-cli config-sync-bda --stack-name my-stack --config-profile v2What a failing sync reports. A sync that cannot read the BDA project fails with an
error instead of proceeding. That matters most in replace mode, where the side being
read is the source of truth: an unreadable project is not an empty one, and treating it
as empty would remove every document class from the profile (bda-to-idp) or create a
second blueprint for every class (idp-to-bda). A transient error — a throttle, or a
missing permission — is therefore safe to retry rather than something to recover from.
A class is also reported failed when its blueprint was created but could not be associated with the project: the blueprint exists, but BDA does not recognise that document type until it is in the project's blueprint list.
Properties BDA cannot represent are reported as warnings. BDA supports neither
objects nested inside objects nor arrays whose items nest further, so those properties
are dropped from the blueprint and each one is named in the sync's warnings, with the
class it belongs to. A property whose value is not a schema object at all — null, or a
string where an object was meant — is reported the same way. Read the warnings on every
sync: a class can succeed with a whole line-items section missing from what it extracts.
To keep such a section, flatten the schema so the nested structure sits in a top-level
$defs definition referenced by $ref.
A blueprint that could not be deleted is reported separately from the classes. In
replace mode the sync removes blueprints the profile no longer describes. The order is
forced — BDA refuses to delete a blueprint a project still associates, so the project's
blueprint list is rewritten first and the deletes follow — which means a delete that
fails leaves a blueprint that is already out of the project. It is invisible to
everything that reads the project, it still counts against the account's blueprint
limit, and a name-prefix match can still pick it up. Those ARNs are printed beside the
result, and they do not count as failed classes: the classes may all have synced,
and the outstanding work is a cleanup rather than a re-sync. Remove them with
--direction cleanup-orphaned, described next.
Not a sync. It deletes every BDA blueprint carrying the stack's name prefix that the named configuration profile's classes do not account for. That is an account-wide scan rather than a project-scoped one, which is exactly why it is the only thing that can reach a blueprint a replace-mode sync disassociated but could not delete — such a blueprint is invisible to every read that goes through the project.
idp-cli config-sync-bda --stack-name my-stack \
--direction cleanup-orphaned --config-profile v2It prompts for confirmation and will not proceed on an empty answer; --force skips the
prompt, which is what a script wants. --mode is not read. The command reports how many
blueprints it deleted and exits non-zero if any deletion failed, naming the ARNs that
are still orphaned afterwards.
--config-profile, or a whitespace-only value — is refused with exit 1 and
nothing is deleted, and so is having no profile at all (you named none and none is
active). Either way the set of classes to keep would come out empty, which is
indistinguishable from "keep nothing", so every prefixed blueprint in the account would
be deleted — and the typo is the worse of the two, because those include the live
blueprints of the profile you meant. Name the profile explicitly on a stack with no
active configuration.
A profile that exists and declares no classes is not refused: keeping nothing is a real instruction, and every prefixed blueprint is deleted. That is the one case where the account-wide sweep is the whole point.
The same operation is available as config.sync_bda(direction="cleanup_orphaned") in
the SDK and as the syncBdaIdp API operation with direction cleanup_orphaned. The Web
UI has no control for it.
Interactive Agent Companion Chat from the terminal. Provides access to the full multi-agent orchestrator including Analytics, Error Analyzer, Code Intelligence, and any configured External MCP Agents.
The chat command runs the same orchestrator as the Web UI's Agent Companion Chat, but locally in your terminal — with real-time streaming and multi-turn conversation support.
Agents wrap their private reasoning in <thinking>...</thinking> and only the answer is printed. Because the response arrives as a stream of small pieces whose boundaries the service chooses, a reasoning block can be split across two of them; the terminal output is the same either way, and if a response ends mid-thought the incomplete reasoning is discarded rather than shown.
Usage:
idp-cli chat [OPTIONS]Options:
--stack-name(required): CloudFormation stack name--region: AWS region (optional)--prompt: Single-shot prompt — sends one message, prints the response, and exits. Useful for scripts and CI/CD.--enable-code-intelligence: Enable the Code Intelligence Agent (disabled by default because it uses external third-party services)
Examples:
# Interactive mode — multi-turn conversation
idp-cli chat --stack-name my-stack
# Single-shot mode — for scripts and automation
idp-cli chat --stack-name my-stack --prompt "What is the avg accuracy for the last test run?"
# With Code Intelligence enabled
idp-cli chat --stack-name my-stack --enable-code-intelligence
# Pipe output in scripts
idp-cli chat --stack-name my-stack --prompt "How many documents failed today?" 2>/dev/nullInteractive session example:
IDP Agent Chat
Stack: my-stack
✓ Ready Agents: Analytics Agent · Error Analyzer Agent · Code Intelligence Agent
Type /quit to exit
You: What is the avg accuracy for test run Fake-W2-Tax-Forms-20260320?
⟶ Analytics Agent
The average accuracy for test run Fake-W2-Tax-Forms-20260320 is 0.867 (86.7%) across 95 documents.
You: Break that down by document type
⟶ Analytics Agent
...
You: /quit
Goodbye.
SDK usage:
from idp_sdk import IDPClient
client = IDPClient(stack_name="my-stack")
# Single message
resp = client.chat.send_message("How many documents were processed today?")
print(resp.response)
# Multi-turn conversation
resp2 = client.chat.send_message("Break down by type", session_id=resp.session_id)
print(resp2.response)Prerequisites:
- Requires
idp_common[agents]to be installed:pip install -e 'lib/idp_common_pkg[agents]' - Requires Amazon Bedrock model access (Claude or Nova models)
- Stack must be deployed with Agent Companion Chat resources (DynamoDB tables, Athena database)
Error: Stack 'my-stack' is not in a valid state
Solution:
# Verify stack exists
aws cloudformation describe-stacks --stack-name my-stackError: Access Denied when uploading files
Solution: Ensure AWS credentials have permissions for:
- S3: PutObject, GetObject on InputBucket/OutputBucket
- SQS: SendMessage on DocumentQueue
- Lambda: InvokeFunction on LookupFunction
- CloudFormation: DescribeStacks, ListStackResources
Error: Duplicate filenames found
Solution: Ensure unique filenames or use directory structure:
document_path
./clientA/invoice.pdf
./clientB/invoice.pdfIssue: Evaluation results missing even with baselines
Checklist:
- Verify
baseline_sourcecolumn exists in manifest - Confirm baseline paths are correct and accessible
- Check baseline directory has correct structure (
sections/1/result.json) - Review CloudWatch logs for EvaluationFunction
Issue: Cannot retrieve document status
Solution:
# Verify LookupFunction exists
aws lambda get-function --function-name <LookupFunctionName>
# Check CloudWatch logs
aws logs tail /aws/lambda/<LookupFunctionName> --followRun the test suite:
cd lib/idp_cli_pkg
pytestRun specific tests:
pytest tests/test_manifest_parser.py -vFor issues or questions:
- Check CloudWatch logs for Lambda functions
- Review AWS Console for resource status
- Open an issue on GitHub