Skip to content

Latest commit

 

History

History
821 lines (670 loc) · 35 KB

File metadata and controls

821 lines (670 loc) · 35 KB
title Pattern 2: Bedrock Classification and Extraction

Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. SPDX-License-Identifier: MIT-0

Pattern 2: Bedrock Classification and Extraction

⚠️ DEPRECATED: Pattern 2 has been superseded by the Unified Pattern, which combines both BDA and pipeline processing modes into a single deployment. The use_bda configuration flag (set via the UI) controls whether documents are processed via BDA or the step-by-step pipeline. See architecture.md for details on the unified architecture.

This document is retained as a reference for pipeline-specific concepts (OCR, classification, extraction, assessment, rule validation) that still apply when use_bda: false is set in the unified pattern (the default mode).

This pattern implements an intelligent document processing workflow that uses Amazon Bedrock with Nova or Claude models for both page classification/grouping and information extraction.

For the current architecture diagram, see the unified architecture in architecture.md.

Table of Contents

Architecture Overview

The workflow consists of three main processing steps with an optional assessment step:

  1. OCR processing using Amazon Textract
  2. Document classification using Amazon Bedrock. classification.strategy chooses between two strategies, and the page-reading call then works at one of two granularities:
    • combined (Split and classify together (one call) in the configuration editor): one call returns a page's or packet's section boundaries and classes together
    • split_then_classify: boundaries first, with no class vocabulary in the prompt, then one call per detected section to classify it
    • Within either, granularity: page makes one call per page and granularity: packet one call per packet or window of window_pages pages
  3. Field extraction using Claude via Amazon Bedrock
  4. Assessment (optional): Confidence evaluation of extraction results using LLMs

State Machine Workflow

The Step Functions state machine (workflow.asl.json) orchestrates the following flow:

OCRStep → ClassificationStep → ProcessPageGroups (Map State for Extraction) → ProcessResultsStep

Each step includes comprehensive retry logic for handling transient errors:

  • Initial retry after 2 seconds
  • Exponential backoff with rate of 2
  • Maximum of 8-10 retry attempts depending on the step

Lambda Functions

OCR Function

  • Purpose: Processes input PDFs using Amazon Textract or Amazon Bedrock
  • Key Features:
    • Supports two OCR backends:
      • Amazon Textract (default)
      • Amazon Bedrock LLMs (Claude, Nova)
    • Concurrent page processing with ThreadPoolExecutor
    • Configurable Image Processing: Enhanced image resizing with aspect-ratio preservation
    • Configurable DPI: Adjustable DPI for PDF-to-image conversion
    • Dual Image Strategy: Stores original high-DPI images while using resized images for OCR processing
    • Smart Resizing: Only downsizes images when necessary (scale factor < 1.0)
    • Image preprocessing and optimization
    • Comprehensive error handling and retries
    • Detailed metrics tracking
  • Input:
    {
      "execution_arn": "<ARN>",
      "output_bucket": "<BUCKET>",
      "input": {
        "detail": {
          "bucket": { "name": "<BUCKET>" },
          "object": { "key": "<KEY>" }
        }
      }
    }
  • Output:
    {
      "metadata": {
        "input_bucket": "<BUCKET>",
        "object_key": "<KEY>",
        "output_bucket": "<BUCKET>",
        "output_prefix": "<PREFIX>",
        "num_pages": "<NUMBER OF PAGES>"
      },
      "pages": {
        "<PAGE_NUMBER>": {
          "rawTextUri": "<S3_URI>",
          "parsedTextUri": "<S3_URI>",
          "imageUri": "<S3_URI>"
        }
      }
    }

Classification Function

  • Purpose: Classifies pages or packets using Claude via Bedrock and splits the document into sections
  • Key Features:
    • Two classification granularities, set by classification.combined.granularity under the default combined strategy and by classification.split_then_classify.splitting.llm.granularity under split_then_classify (see Classification):
      • Page granularity (the default): one call per page
      • Packet granularity: one call per packet
    • RVL-CDIP dataset categories for classification
    • Concurrent page processing
    • Automatic image resizing and optimization
    • Robust error handling with exponential backoff
    • Few shot example support for improved accuracy
  • Input: Output from OCR function
  • Output:
    {
      "metadata": "<FROM_OCR>",
      "sections": [
        {
          "id": "<GROUP_ID>",
          "class": "<CLASS>",
          "pages": [...]
        }
      ]
    }

Extraction Function

  • Purpose: Extracts fields using Claude via Bedrock
  • Key Features:
    • Document class-specific attribute extraction
    • Configurable extraction attributes
    • Comprehensive error handling
    • Token usage tracking
    • Few shot example support for improved accuracy
  • Input: Individual section from Classification output
  • Output:
    {
      "section": {
        "id": "<ID>",
        "class": "<CLASS>",
        "page_ids": ["<PAGEID>", ...],
        "outputJSONUri": "<S3_URI>"
      },
      "pages": [...]
    }

ProcessResults Function

  • Purpose: Consolidates results from all sections
  • Output: Standardized format for GenAIIDP parent stack:
    {
      "Sections": [{
        "Id": "<ID>",
        "PageIds": ["<PAGEID>", ...],
        "Class": "<CLASS>",
        "OutputJSONUri": "<S3_URI>"
      }],
      "Pages": [{
        "Id": "<ID>",
        "Class": "<CLASS>",
        "TextUri": "<S3_URI>",
        "ImageUri": "<S3_URI>"
      }],
      "PageCount": "<TOTAL_PAGES>"
    }

Human-in-the-Loop (HITL)

Human review is built into the solution's Web UI; it does not use Amazon SageMaker Augmented AI (A2I). Reviewers validate and correct extracted information when the system's confidence falls below a specified threshold.

Configuration: HITL is enabled through the configuration (Web UI Configuration → Human-in-the-Loop (HITL) Review (Optional), or the hitl.enabled key in the config YAML), not a CloudFormation parameter — the EnableHITL parameter was removed in v0.4.11.

For comprehensive HITL documentation including workflow details, configuration steps, best practices, and troubleshooting, see the Human-in-the-Loop Review Guide.

Monitoring and Metrics

The pattern includes a comprehensive CloudWatch dashboard with:

Performance Metrics

  • Document and page throughput
  • Token usage (input/output/total)
  • Bedrock request statistics
  • Processing latencies
  • Throttling and retry metrics

Error Tracking

  • Lambda function errors
  • Long-running invocations
  • Classification/extraction failures
  • Throttling events

Lambda Function Metrics

  • Duration
  • Memory usage
  • Error rates
  • Concurrent executions

Template Outputs

The pattern exports these outputs to the parent stack:

  • StateMachineName: Name of Step Functions state machine
  • StateMachineArn: ARN of Step Functions state machine
  • StateMachineLogGroup: CloudWatch log group for state machine
  • DashboardName: Name of pattern-specific dashboard
  • DashboardArn: ARN of pattern-specific dashboard

Configuration

Stack Deployment Parameters:

Pipeline mode has no deployment parameters of its own. The stack is deployed with the main template's parameters, among them:

  • ConfigurationPreset: the starting configuration, a directory under config_library/unified/
  • CustomConfigPath: the S3 URI of your own configuration file, which overrides the preset
  • MaxConcurrentWorkflows: Workflow concurrency limit
  • LogRetentionDays: CloudWatch log retention period
  • ExecutionTimeThresholdMs: Latency threshold for alerts

Summarization is controlled by the configuration's summarization.enabled property.

Configuration Management:

  • Model selection is now handled through configuration files rather than CloudFormation parameters
  • The presets are the directories under config_library/unified/
  • Configuration can be updated through the Web UI without stack redeployment
  • Model choices are constrained through enum constraints in the configuration schema

Bedrock OCR Feature

Overview

Pipeline mode supports Amazon Bedrock LLMs (Claude, Nova) as an alternative OCR backend alongside the traditional Amazon Textract service. This feature enables multimodal document processing where large language models can extract text from document images using their vision capabilities.

Configuration

Bedrock OCR is configured through the pattern's configuration files. The OCR backend can be selected using the backend parameter:

ocr:
  backend: "bedrock"  # Options: "textract", "bedrock", "none"
  model_id: "us.anthropic.claude-sonnet-4-6"
  system_prompt: "You are an expert OCR system. Extract all text from the provided image accurately, preserving layout where possible."
  task_prompt: "Extract all text from this document image. Preserve the layout, including paragraphs, tables, and formatting."

Enabling Bedrock OCR

To use Bedrock OCR:

  1. Set the backend: Configure backend: "bedrock" in your OCR configuration

  2. Choose a model: Pick from the Model dropdown under OCR in the configuration editor. It is generated from the ocr.model_id enum in patterns/unified/template.yaml, and spans the us., eu. and global. Claude and Nova families, qwen.qwen3-vl-235b-a22b, Grok and the OpenAI models. Commonly used: us.amazon.nova-lite-v1:0 for clean text-heavy scans, us.amazon.nova-pro-v1:0 for mixed layouts, us.anthropic.claude-sonnet-4-6 for handwriting and dense tables. Select LambdaHook to route OCR to your own Lambda function instead.

  3. Configure prompts: Customize system and task prompts for your specific use case

  4. Deploy: The configuration can be updated through the Web UI without stack redeployment

Benefits

Advantages of Bedrock OCR:

  • Multimodal Understanding: LLMs can understand both visual layout and textual content
  • Context Awareness: Better handling of complex document structures and relationships
  • Flexibility: Customizable prompts for domain-specific terminology and formats
  • Advanced Reasoning: Can handle challenging cases like handwritten text, poor quality scans, or complex layouts
  • Unified Processing: Same models used for OCR, classification, and extraction provide consistency

⚠️ Measured, Bedrock OCR is the one backend that gets values wrong. On the v0.6.14 benchmark (typed synthetic bank statements, the default OCR model Amazon Nova 2 Lite) it returned every row at cell accuracy 0.990, where Textract TABLES, Textract LAYOUT and BDA were all at 1.000, and it perturbed fixed-width identifiers such as account numbers. Do not use it where identifiers or amounts matter (config-guidance §2a, finding 4). Handwritten and low-quality documents were not measured.

When to Use Bedrock OCR:

  • Documents with complex layouts or mixed content types
  • Handwritten or low-quality documents where Textract struggles (not measured; test on your own documents first)
  • Domain-specific documents requiring contextual understanding
  • When you want unified processing across the entire pipeline
  • For experimental or specialized use cases requiring prompt customization

Cost Considerations

Bedrock OCR Pricing:

  • Charged per input/output token rather than per page
  • Not necessarily dearer than Textract: on the v0.6.14 benchmark statements the default OCR model (Amazon Nova 2 Lite) cost about $0.004 per page for OCR, the same as Textract LAYOUT and about a quarter of Textract TABLES' $0.015, and the Bedrock OCR configuration was the cheapest whole-document cell of the grid ($0.3821 against $0.5238 for TABLES). The per-page figures are the OCR phase of benchmarks/results/v0.6.14/core/summary.json
  • Cost varies significantly by model (Nova Lite < Nova Pro < Claude models)
  • Image tokens are more expensive than text tokens

Cost Optimization Tips:

  1. Model Selection: Use Nova Lite for cost-sensitive applications, Claude for quality-critical use cases
  2. Image Preprocessing: Enable image resizing and preprocessing to reduce token consumption
  3. Prompt Optimization: Use concise, focused prompts to minimize token usage
  4. Hybrid Approach: Use Textract for standard documents, Bedrock for complex cases
  5. Batch Processing: Process multiple pages efficiently with concurrent processing

OCR Configuration

The OCR service in pipeline mode supports enhanced image processing capabilities for optimal text extraction:

DPI Configuration

Configure DPI (Dots Per Inch) for PDF-to-image conversion:

# Example OCR service initialization with custom DPI
ocr_service = OcrService(
    dpi=400,  # Higher DPI for better quality (default: 300)
    resize_config={
        'target_width': 1200,
        'target_height': 1600
    }
)

Image Resizing Configuration

The OCR service supports optional image resizing for processing optimization:

# OCR configuration example
ocr:
  image:
    target_width: 951     # Target width for processing
    target_height: 1268   # Target height for processing
    dpi: 400              # PDF-to-image rasterization DPI
    preprocessing: true   # Enable adaptive binarization preprocessing

All four settings sit directly under ocr.image, and there is no resize_config level in the configuration — the service builds that dictionary itself from target_width and target_height. The resize_config= and dpi= constructor keywords in the Python example above are the deprecated back-compatibility path; pass the configuration instead and set the values there. When neither dimension is set, OcrService applies its own 2600×3600 ceiling.

OCR Image Processing Features

  • Configurable DPI: Higher DPI (400+) for better quality, standard DPI (300) for balanced performance
  • Dual Image Strategy:
    • Stores original high-DPI images in S3 for archival and downstream processing
    • Uses resized images for OCR processing to optimize performance
  • Aspect Ratio Preservation: Images are resized proportionally without distortion
  • Smart Scaling: Only downsizes images when necessary (scale factor < 1.0)
  • Image Preprocessing: Optional adaptive binarization to improve OCR accuracy on challenging documents
  • Enhanced Logging: Detailed logging for DPI and resize operations

Image Preprocessing Configuration

The OCR service supports optional adaptive binarization preprocessing to improve OCR accuracy on challenging documents:

# Enable preprocessing for improved OCR accuracy
ocr:
  image:
    preprocessing: true  # Enable adaptive binarization

Adaptive Binarization Benefits:

  • Improved OCR Accuracy: Significantly enhances text extraction on documents with uneven lighting, shadows, or low contrast
  • Background Noise Reduction: Removes background gradients and noise that can interfere with OCR
  • Enhanced Edge Detection: Sharpens text boundaries for better character recognition
  • Robust Processing: Handles challenging document conditions like poor scans or faded text

When to Enable Preprocessing:

  • Documents with uneven lighting or shadows
  • Low contrast text or faded documents
  • Scanned documents with background noise
  • Handwritten or mixed content documents
  • When standard OCR accuracy is insufficient

Configuration Benefits

  • Quality Control: Higher DPI settings improve OCR accuracy for complex documents
  • Performance Optimization: Resized images reduce processing time and memory usage
  • Storage Efficiency: Dual strategy balances quality preservation with processing efficiency
  • Preprocessing Enhancement: Adaptive binarization improves OCR accuracy on challenging documents
  • Flexibility: Runtime configuration allows adjustment without code changes
  • Backward Compatibility: Default values maintain existing behavior

Best Practices for OCR

  1. DPI Selection:

    • Use 300 DPI for standard documents
    • Use 400+ DPI for documents with small text or complex layouts
    • Consider processing costs when using higher DPI settings
  2. Image Resizing:

    • Enable resizing for large documents to improve processing speed
    • Maintain aspect ratios to preserve text readability
    • Test different dimensions based on document types
  3. Performance Tuning:

    • Monitor processing times and adjust DPI/resize settings accordingly
    • Use concurrent processing for multi-page documents
    • Balance quality requirements with processing costs

Customizing Classification

Classification runs at one of two granularities, set in the configuration by classification.combined.granularity under the default combined strategy. Under split_then_classify it is set by classification.split_then_classify.splitting.llm.granularity instead, and applies to the splitting call, which returns section boundaries without classes; each section is then classified by a call of its own. The two descriptions below are of the combined call:

  1. Page granularity (page): This is the default. It classifies each page based on its visual layout and textual content, and outputs a JSON object with a class label (and a start/continue boundary marker) per page. Its prompts are classification.combined.page.system_prompt and classification.combined.page.task_prompt.

  2. Packet granularity (packet): This granularity examines the document as a whole to identify the boundaries between its sections. It detects section boundaries and identifies each section's class in the context of the whole document. This is especially useful for packets where individual pages may not be clearly classifiable on their own. It outputs a JSON format that identifies the sections with start and end page numbers. Its prompts are classification.combined.packet.system_prompt and classification.combined.packet.task_prompt.

The granularity is a configuration setting, not a stack parameter: change it in the configuration (for example in the web UI's configuration editor). Classification covers both granularities, the split-then-classify strategy, and how to choose between them.

The classification system uses RVL-CDIP dataset categories and can be customized through the configuration files. Classification models and prompts are now managed through the configuration library rather than CloudFormation parameters.

Available categories:

  • letter
  • form
  • email
  • handwritten
  • advertisement
  • scientific_report
  • scientific_publication
  • specification
  • file_folder
  • news_article
  • budget
  • invoice
  • presentation
  • questionnaire
  • resume
  • memo

Few Shot Example Feature

Pipeline mode supports few shot learning through example-based prompting to significantly improve classification and extraction accuracy. This feature allows you to provide concrete examples of documents with their expected classifications and attribute extractions.

Overview

Few shot examples work by including reference documents with known classifications and expected attribute values in the prompts sent to the AI model. This helps the model understand the expected format and accuracy requirements for your specific use case.

Configuration

Few shot examples are configured using JSON Schema format in the configuration files in the config_library/unified/ directory. The rvl-cdip-with-few-shot-examples preset demonstrates how to set up examples:

classes:
  - $schema: "https://json-schema.org/draft/2020-12/schema"
    $id: Letter
    x-aws-idp-document-type: Letter
    type: object
    description: "A formal written correspondence..."
    properties:
      SenderName:
        type: string
        description: "The name of the person who wrote the letter..."
    x-aws-idp-examples:
      - x-aws-idp-class-prompt: "This is an example of the class 'Letter'"
        name: "Letter1"
        x-aws-idp-attributes-prompt: |
          expected attributes are:
              "SenderName": "Will E. Clark",
              "SenderAddress": "206 Maple Street P.O. Box 1056 Murray Kentucky 42071-1056",
              "RecipientName": "The Honorable Wendell H. Ford"
        x-aws-idp-image-path: "config_library/unified/rvl-cdip-with-few-shot-examples/example-images/letter1.jpg"
  - $schema: "https://json-schema.org/draft/2020-12/schema"
    $id: Email
    x-aws-idp-document-type: Email
    type: object
    description: "A digital message with email headers..."
    x-aws-idp-examples:
      - x-aws-idp-class-prompt: "This is an example of the class 'Email'"
        name: "Email1"
        x-aws-idp-attributes-prompt: |
          expected attributes are: 
             "FromAddress": "Kelahan, Ben",
             "ToAddress": "TI New York: 'TI Minnesota",
             "Subject": "FW: Morning Team Notes 4/20"
        x-aws-idp-image-path: "config_library/unified/rvl-cdip-with-few-shot-examples/example-images/email1.jpg"

Benefits

Using few shot examples provides several advantages:

  1. Improved Accuracy: Models perform better when given concrete examples
  2. Consistent Formatting: Examples help ensure consistent output structure
  3. Domain Adaptation: Examples help models understand domain-specific terminology
  4. Reduced Hallucination: Examples reduce the likelihood of made-up data
  5. Better Edge Case Handling: Examples can demonstrate how to handle unusual cases

Integration with Template Prompts

The few shot examples are automatically integrated into the classification and extraction prompts using the {FEW_SHOT_EXAMPLES} placeholder. You can also use the {DOCUMENT_IMAGE} placeholder for precise image positioning:

Standard Template with Text Only:

# In classification task_prompt
task_prompt: |
  Classify this document into exactly one of these categories:
  {CLASS_NAMES_AND_DESCRIPTIONS}
  
  <few_shot_examples>
  {FEW_SHOT_EXAMPLES}
  </few_shot_examples>
  
  <document_ocr_data>
  {DOCUMENT_TEXT}
  </document_ocr_data>

# In extraction task_prompt  
task_prompt: |
  Extract attributes from this document.
  
  <few_shot_examples>
  {FEW_SHOT_EXAMPLES}
  </few_shot_examples>
  
  
  <document_ocr_data>
  {DOCUMENT_TEXT}
  </document_ocr_data>

Enhanced Template with Image Placement:

# In classification task_prompt with image positioning
task_prompt: |
  Classify this document into exactly one of these categories:
  {CLASS_NAMES_AND_DESCRIPTIONS}
  
  <few_shot_examples>
  {FEW_SHOT_EXAMPLES}
  </few_shot_examples>
  
  Now examine this new document:
  {DOCUMENT_IMAGE}
  
  <document_ocr_data>
  {DOCUMENT_TEXT}
  </document_ocr_data>
  
  Classification:

# In extraction task_prompt with image positioning
task_prompt: |
  Extract attributes from this {DOCUMENT_CLASS} document:
  {ATTRIBUTE_NAMES_AND_DESCRIPTIONS}
  
  <few_shot_examples>
  {FEW_SHOT_EXAMPLES}
  </few_shot_examples>
  
  Analyze this document image:
  {DOCUMENT_IMAGE}
  
  <document_ocr_data>
  {DOCUMENT_TEXT}
  </document_ocr_data>
  
  Extract as JSON:

Available Template Placeholders

Pipeline mode supports several placeholders for building dynamic prompts:

  • {CLASS_NAMES_AND_DESCRIPTIONS}: List of document classes and their descriptions
  • {FEW_SHOT_EXAMPLES}: Examples from the configuration (class-specific for extraction, all classes for classification)
  • {DOCUMENT_TEXT}: OCR-extracted text content from the document
  • {DOCUMENT_IMAGE}: Document image(s) positioned at specific locations in the prompt
  • {DOCUMENT_CLASS}: The class of the section being processed (used in extraction prompts)
  • {ATTRIBUTE_NAMES_AND_DESCRIPTIONS}: List of attributes to extract with their descriptions

Image Placement Benefits:

  • Visual Context: Position images where they provide maximum context for the task
  • Multimodal Understanding: Help models correlate visual and textual information effectively
  • Flexible Design: Create prompts that flow naturally between different content types
  • Enhanced Accuracy: Strategic image placement can improve both classification and extraction performance

Using Few Shot Examples

To use few shot examples in your deployment:

  1. Use the example configuration: Deploy with the ConfigurationPreset stack parameter set to rvl-cdip-with-few-shot-examples, or upload a copy of config_library/unified/rvl-cdip-with-few-shot-examples/config.yaml to S3 and set CustomConfigPath to its S3 URI
  2. Create custom examples: Copy the example configuration and modify it with your own document examples
  3. Provide example images: Place example document images in the appropriate directory and reference them in the x-aws-idp-image-path field

Best Practices

  1. Quality over Quantity: Use 1-3 high-quality examples per document class
  2. Representative Examples: Choose examples that represent typical documents in your use case
  3. Clear Attribution: Ensure examples clearly show expected attribute extractions
  4. Diverse Coverage: Include examples that cover different variations and edge cases

Customizing Extraction

The extraction system can be customized through the configuration files rather than CloudFormation parameters:

  1. Attribute Definitions:

    • Define attributes per document class in the classes section of the configuration
    • Specify descriptions for each attribute
    • Configure the format and structure
  2. Extraction Prompts:

    • Customize system behavior through the system_prompt in configuration
    • Add domain expertise and guidance in the task_prompt
    • Modify output formatting requirements
  3. Model Selection:

    • Model selection is handled through enum constraints in the configuration
    • Available models are defined in the configuration schema
    • Changes can be made through the Web UI without redeployment

Example attribute definition from the configuration using JSON Schema format:

classes:
  - $schema: "https://json-schema.org/draft/2020-12/schema"
    $id: Invoice
    x-aws-idp-document-type: Invoice
    type: object
    description: A commercial document issued by a seller to a buyer relating to a sale
    properties:
      InvoiceNumber:
        type: string
        description: The unique identifier for the invoice. Look for 'invoice no', 'invoice #', or 'bill number', typically near the top of the document.
      InvoiceDate:
        type: string
        description: The date when the invoice was issued. May be labeled as 'date', 'invoice date', or 'billing date'.
      TotalAmount:
        type: string
        description: The final amount to be paid including all charges. Look for 'total', 'grand total', or 'amount due', typically the last figure on the invoice.

Confidence (Assessment)

As of config format 0.6, per-field confidence and geometry are outputs of extraction, not a separate downstream stage. Their settings live under extraction.confidence.* and extraction.geometry.* (there is no top-level assessment.{model, geometry_mode, ...} block anymore). This section is a Pattern 2 (Pipeline mode) summary; the full reference is Extraction & Confidence.

Overview

Confidence assessment produces, for each extracted attribute:

  • Confidence Scores: Per-attribute confidence ratings (0.0-1.0)
  • Explanatory Reasoning: Human-readable explanations for each confidence score
  • UI Integration: Automatic display in the web interface visual editor
  • Cost Optimization: Can be disabled entirely for zero LLM cost
  • Large-list batching: Long list fields (hundreds of rows) are scored in sequential batches so every row is covered

Confidence modes

extraction.confidence.mode controls where per-field confidence is produced:

  • separate (default) — on the Simple (non-agentic) path, confidence runs as the standalone Assessment step; on the Advanced (agentic) path, it runs as a second inference inside each extraction shard and the standalone step auto-skips.
  • integrated — a single extraction inference returns values and inline confidence together (works on both the simple and agentic paths); the service splits the envelope, enriches with thresholds, and emits explainability_info, so the standalone Assessment step auto-skips.
  • off — no confidence scoring (equivalent to enabled: false); zero LLM cost.
extraction:
  confidence:
    enabled: true             # false disables confidence entirely (zero LLM cost)
    mode: separate            # off | separate (default) | integrated
    model: us.amazon.nova-lite-v1:0
    temperature: 0.0
    list_batch_size: 25       # rows per assessment batch for large lists
    system_prompt: "You are an expert document analyst..."
    task_prompt: |
      Assess the confidence of extraction results for this {DOCUMENT_CLASS} document.
      Extraction Results: {EXTRACTION_RESULTS}
      Attributes: {ATTRIBUTE_NAMES_AND_DESCRIPTIONS}
      Document Text: {DOCUMENT_TEXT}
      OCR Confidence: {OCR_TEXT_CONFIDENCE}
      {DOCUMENT_IMAGE}

Behavior when disabled (enabled: false or mode: off): the assessment Lambda is still invoked (minimal overhead) but returns immediately with logging "Assessment is disabled via configuration" — no LLM calls, no S3 operations. Defaults to enabled when the property is missing.

Migration note: The previous IsAssessmentEnabled CloudFormation parameter has been removed in favor of this configuration-based control.

Large lists (list_batch_size)

For documents with large lists (bank statements with hundreds of transactions, line-item tables, brokerage holdings), the standalone Assessment step batches large lists automatically: it slices the largest list field into extraction.confidence.list_batch_size chunks (default 25), assesses each chunk sequentially with the shared scalars/context, then reconciles the results so every list cell gets its own confidence and bounding box. A bounded missing-row retry re-scores any rows the model dropped, so coverage reaches 100%. Lower list_batch_size if a model still struggles to enumerate a full chunk; raise it to reduce the number of inference calls.

Granular assessment is retired. The former "granular assessment" service (a thread-pool fan-out with DynamoDB caching, formerly configured under assessment.granular / extraction.confidence.granular) has been retired and deleted. Large-list batching is its full replacement. Any leftover granular.* keys still validate but are ignored — no config edit required. See Granular Assessment Retirement.

For large documents, prefer Advanced (agentic) extraction

For sections that can hold more than about 400 list rows, and for very large documents, prefer Advanced (agentic) extraction — it shards both extraction and confidence assessment. On the v0.6.14 benchmark, at the study's control extraction model, it returned every row with every row confidence-scored at every size from 25 to 3,200 rows, where Simple mode left 5–20% of rows unscored at 1,200–1,600 rows; at the shipped default model, Simple mode returned part of an 800-row list on 8 of 13 runs and Advanced mode all of it on 6 of 6 (config-guidance §3, §5.5). On real-world forms Simple mode was as accurate at about half the cost.

Comprehensive Documentation

For detailed information about confidence and geometry configuration, output formats, confidence thresholds, UI integration, cost optimization, and troubleshooting, see Extraction & Confidence. That guide covers:

  • Complete configuration examples and placeholders
  • Attribute types and assessment formats (simple, group, list)
  • Confidence threshold configuration and UI integration
  • Confidence modes (off / separate / integrated) and geometry modes
  • Large-list batching and the missing-row retry
  • Cost optimization strategies and token reduction techniques
  • Multimodal assessment with image processing
  • Best practices

Best Practices

  1. Configuration Management:

    • Start from the configuration library preset closest to your use case (the directories under config_library/unified/)
    • Test configuration changes thoroughly before production deployment
    • Leverage the Web UI for configuration updates without redeployment
  2. Throttling Management:

    • Implement exponential backoff with jitter
    • Configure appropriate retry limits
    • Monitor throttling metrics
  3. Error Handling:

    • Comprehensive error logging
    • Graceful degradation
    • Clear error messages
  4. Performance Optimization:

    • Concurrent processing where appropriate
    • Image optimization
    • Resource pooling
  5. Monitoring:

    • Detailed CloudWatch metrics
    • Performance dashboards
    • Error tracking
  6. Security:

    • KMS encryption
    • Least privilege IAM roles
    • Secure configuration management
  7. Few Shot Examples:

    • Use high-quality, representative examples
    • Include examples for all document classes you expect to process
    • Regularly review and update examples based on real-world performance
    • Test configurations with examples before production deployment
  8. Image Processing Optimization:

    • Configure appropriate image dimensions for each service based on document complexity
    • Use higher DPI (400+) for OCR when processing documents with small text or complex layouts
    • Balance image quality with processing performance and costs
    • Test different image configurations with your specific document types
    • Monitor memory usage and processing times when adjusting image settings
    • Leverage the dual image strategy in OCR to preserve quality while optimizing processing