Design Principles

Design Principles

Note

Documentation on this page is still under construction

This page describes the general design principles and guidelines driving the development of the PEACH API.

Table of Contents

• Core Features
• Design Scope
• LRU Cache Strategy
• Merge Algorithm

Core Features

The NR Permitting Exchange, Aggregation and Collection Hub is a microservice designed with the following core features:

• Permitting Exchange: Facilitates the exchange of standardized PIES permitting data between different systems.
• Aggregation: Collects and aggregates PIES permitting data from various sources.
• Collection: Provides a centralized hub for collecting and managing PIES permitting data.

The architecture of PEACH emphasizes a loosely coupled approach, offering flexibility to adapt to evolving data and exchange standards. This approach enables line-of-business (LOB) systems to modernize incrementally at their own pace. Additionally, it provides a reliable and standardizable mechanism to ensure the consumption and provision of consistent, standardized permitting data that can be utilized across various sectors and services. It aims to act as a "record discovery" service, eventually allowing consuming systems to traverse relationships between PIES records and autonomously discover related records as needed.

Design Scope

We envision the PEACH API to be one of the potentially many microservices and systems that will leverage the PIES ecosystem. While PEACH will initially serve as a primary hub for PIES structured data, it will not be the only system to do so. Other systems may also implement PIES data structures and exchange mechanisms, and PEACH may interact with these systems as needed.

Some of the core design principles and boundaries for what PEACH will and will not do include:

• Data Scope
  • PEACH shall behave as a derivative, caching source of truth for transformed standardized PIES formated data.
  • PEACH shall not be a primary source of truth for PIES data, but rather act as a convenient hub that aggregates and collects data from various sources.
  • PEACH will consider all PIES data inserted by originating systems as canonical once it has been transformed and stored in the hub.
  • PEACH will not implement custom data formats, definitions or structures outside of the PIES ecosystem.
• Data Structure
  • PEACH shall only accept transformed standardized PIES formatted data.
  • PEACH shall only provide responses in transformed standardized PIES format.
  • PEACH shall be an idempotent service, meaning that it will actively de-duplicate information within the hub to ensure consistency and reliability.
  • PEACH will not alter the meaning or context of the PIES data it processes.
• Data Organization
  • PEACH shall be responsible for JSON Schema validation and some basic semantic codeword validation of the PIES data it receives.
  • PEACH will assume that all PIES data pushed to it is considered correct and valid by the originating system.
  • Originating systems may require an interim ETL service to pull data from their systems and push it into PEACH.
  • Originating systems or intermediate ETL processes shall act as the authoritative system_id for the data they push to PEACH.
  • PEACH will be able to store and manage the relationships between PIES records, allowing for data discoverability and record traversal based on these relationships.
  • PEACH will not perform complex data transformations, aggregations or validations beyond the scope of PIES data structures.

In essence, think of PEACH as a specialized microservice that acts as a caching and aggregation microservice for PIES data. Its job is to make it easier to transmit and exchange PIES data between systems, but PEACH will not become the primary source of truth for PIES data. Instead, it serves as a convenient hub that collects and aggregates data from various sources, eventually allowing for data discoverability and record traversal based on the relationships of the records stored.

LRU Cache Strategy

We have implemented an abstraction of a reusable and atomic LRU (Least Recently Used) memory cache for arbitrary singleton database table records. This cache is designed to efficiently store and retrieve singleton records while maintaining a fixed size, automatically evicting the least recently used records when the cache is full, or when a record expires.

Services may opt-in to use this cache for frequently accessed records, allowing for faster retrieval and reduced database load. Under certain conditions, the cache can improve response times significantly, as many lookups are highly repetitive in nature and can benefit from in-memory caching. We have intentionally designed our cacheWrapper pattern to be hot-swappable with the underlying repository logic, allowing us to easily switch between cached and non-cached operations without significant code changes.

---
title: General LRU Cache Strategy
---

graph LR
    DB@{ shape: cyl, label: "Database" }
    EN@{ shape: fr-circ, label: "End" }
    LC@{ shape: st-rect, label: "LRU Cache" }
    LG@{ shape: diamond, label: "LRU Get" }
    LS@{ shape: lin-rect, label: "LRU Set" }
    RP@{ shape: fr-rect, label: "Repository Subroutine" }
    ST@{ shape: sm-circ, label: "Start" }

    ST ==> LG
    LC -.-> LG
    LG == Cache Hit ==> EN
    LG == Cache Miss ==> RP
    LS -.-> LC
    RP ==> EN
    RP --> LS
    RP -.- DB

Loading

While the LRU cache implementation can theoretically work with queries that yield an array of responses, we intend for the LRU cache to only be used for singleton records. This is because the complexity of managing cache invalidation becomes significantly higher when dealing with multiple records as each record may have different lifetimes and update frequencies. By focusing on singleton records, we can ensure that the cache remains efficient and easy to manage, while still providing substantial performance benefits for frequently accessed data.

Merge Algorithm

Note

Behavior in this section is not finalized and may change over time

The PEACH API accepts ProcessEventSet records as either an assertion (PUT /process-events) or a merge operation (POST /process-events). The assertion operation takes whatever is provided in the request body and hard replaces all of the content in that record, assuming the content passes validation. On the other hand, the merge operation is designed to intelligently deduce and combine new ProcessEvents with existing ones, ensuring that the record reflects the most up-to-date state of its associated processes.

Depending on the maturity and frequency of updates from originating systems, they may choose to use either the assertion or merge operation. The merge operation is particularly useful for systems that frequently update their ProcessEvents, as it allows them to submit only differential changes without needing to resend the entire set of events. Originating systems may use either operation depending on their needs, but they must ensure that the request body complies with the ProcessEventSet message element definition.

Merge Evaluation Logic

The following pseudocode outlines the high-level logic for evaluating and merging incoming ProcessEvent elements with existing ones:

Let `ExistingPEs` be the set of known Process Events in the record.
Let `IncomingPEs` be the set of Process Events in the incoming request.

FOR each `ExistingPE` in `ExistingPEs`:
  Search for `IncomingPE` in `IncomingPEs` such that:
    `IncomingPE.process.code == ExistingPE.process.code`
    IF no match is found:
      Add `IncomingPE` to `ExistingPEs`
    ELSE:
      Compare `IncomingPE` and `ExistingPE`:
      IF `IncomingPE == ExistingPE`:
        Do nothing
      ELSE IF `ExistingPE` contains more precise information:
        Do nothing
      ELSE IF `IncomingPE` contains more precise information:
        Update `ExistingPE` with `IncomingPE`
      ELSE IF `IncomingPE` and `ExistingPE` contain divergent information:
        Compare the UUIDv7 `transaction_id` timestamp portion of `IncomingPE` and `ExistingPE`:
        IF `IncomingPE.transaction_id > ExistingPE.transaction_id`:
          Overwrite `ExistingPE` with `IncomingPE`
        ELSE:
          Do nothing

---
title: Merge Evaluation Flowchart
---

flowchart LR
  Start@{ shape: sm-circ, label: "Start" } --> CheckExistingPEs[Iterate through ExistingPEs]
  CheckExistingPEs --> MatchIncomingPE{Find matching IncomingPE in IncomingPEs}
  MatchIncomingPE -->|No match found| AddIncomingPE[Add IncomingPE to ExistingPEs]
  AddIncomingPE --> End[End]
  MatchIncomingPE -->|Match found| ComparePEs{Compare IncomingPE with ExistingPE}
  ComparePEs -->|IncomingPE equals ExistingPE| DoNothing1[Do nothing]
  DoNothing1 --> End[End]
  ComparePEs -->|ExistingPE is more precise| DoNothing2[Do nothing]
  DoNothing2 --> End[End]
  ComparePEs -->|IncomingPE is more precise| UpdateExistingPE[Update ExistingPE with IncomingPE]
  UpdateExistingPE --> End[End]
  ComparePEs -->|IncomingPE and ExistingPE diverge| CompareTransactionID{Compare transaction_id timestamps}
  CompareTransactionID -->|IncomingPE is newer| OverwriteExistingPE[Overwrite ExistingPE with IncomingPE]
  OverwriteExistingPE --> End[End]
  CompareTransactionID -->|ExistingPE is newer or equal| DoNothing3[Do nothing]
  DoNothing3 --> End@{ shape: fr-circ, label: "End" }

Loading

This logic ensures that the merge operation reasonably handles incoming ProcessEvents, prioritizing the most precise and up-to-date information while resolving any divergences between existing and incoming data.

Event Matching

The merge operation matches incoming ProcessEvents with existing ones based on their event.id attribute. This allows the system to identify which events are related and should be merged together. If an incoming ProcessEvent has an event.id that matches an existing event, the system will evaluate the two events for potential updates or conflicts. Event matching will run on the following principles:

• Events with datetime resolution will be preferred over those with only date resolution:
  • When an incoming ProcessEvent includes a datetime attribute, it will take precedence over an existing event that only has a date attribute, provided the datetime is more precise and corresponds to the same calendar date.
  • If both the incoming and existing events share the same date, they will be treated as equivalent, and no updates will be performed.
  • If both the incoming and existing events share the same datetime, they will be treated as equivalent, and no updates will be performed.
• Events containing both a start and end value will be preferred over those with only a start value:
  • When an incoming ProcessEvent includes both start and end attributes, it will take precedence over an existing event that only has a start attribute, provided the end attribute is at or after the start attribute, and the start dates are equivalent.
  • When an incoming ProcessEvent only contains a start attribute, whereas the existing event has both start and end attributes, the existing event will be preferred if the start date is equivalent to the incoming event's start date.
  • If both the incoming and existing events share the same start and end values, they will be treated as equivalent, and no updates will be performed.
  • Refer to Divergence Handling for resolving Event conflicts with divergent (non-overlapping or partial overlapping) values.

Process Matching

The merge operation identifies related ProcessEvents by matching their process.code and process.code_system attributes. This ensures that incoming events are evaluated against existing ones for updates or conflicts based on consistent and standardized PIES identifiers.

• ProcessEvents are considered a match if both code_system and code attributes match, provided there is only one unique match on both the Existing and Incoming sides.
• If code_system attributes do not match, the comparison can be short-circuited as no match exists.
• In cases where multiple matches occur due to identical code and code_system attributes, these attributes are treated as logically similar. Refer to Divergence Handling for resolving such conflicts.

Divergence Handling

In situations where there are multiple matching processes in either the existing or incoming ProcessEvents, the merge operation will not be able to determine a single authoritative version of the ProcessEvent. In such cases, the system may need to employ additional heuristics or user input to resolve the conflicts and decide which version of the event should be retained.

Note

We may need to consider implementing merge reconciliation endpoints that require human intervention in order to assess, as some of these conflicts will not be resolvable by the system alone.

Multiple Matches

When comparing between different ProcessEventSets, it is possible to encounter situations where there exists multiple Process matches for a given code and code_system. Under these circumstances, the logic to deterministically resolve which ProcessEvents are to be kept, dropped, or merged can become very complex, as the context will also depend on the attached Event information.

Once we have identified a subset of ProcessEvents within the ProcessEventSet that share the same Process, we anticipate needing to implement some form of sieve algorithm to appropriately identify and match ProcessEvents. On a high level, we would need to do another looped iteration, this time through the shared subset of ProcessEvents which contain the same logical Process.

The following outcomes are possible after this second sieve operation:

• All ProcessEvents were able to be bijectively matched, and the merge operation can proceed as normal.
• Some ProcessEvents were able to be matched, while others were not. In this case, we would require branching logic to handle the matched and unmatched ProcessEvents separately.
• No ProcessEvents were able to be matched, and the merge operation will need to determine if accepting both sets of ProcessEvents is appropriate, or if one set should be dropped in favor of the other.

Warning

For all the ProcessEvent cases where a match is not unable to be established, we will not have enough empirical data available to do a confident merge. We will need to explore the design space for how to handle these cases, as they will have ramifications on how we curate and maintain a canonical set of information within the PEACH API. It is likely that we will need to implement some form of user input or manual intervention to resolve these conflicts, as the system may not be able to determine the correct course of action automatically.

Date Divergences

When comparing between Events, there are multiple scenarios that can occur, independent of date precision:

• The date values match line for line
• The date value durations partially overlap (either the start and end are different, but the duration within shares some common timeframe)
• Both events contain non-overlapping start values, implying that both events may still be valid and ongoing.
• The date value durations do not overlap at all (either the start or end is different, and the duration does not share any common timeframe)

In each of these scenarios, we will handle the divergence based on the following principles:

• For partial overlaps, consider them as different ProcessEvents; both must be retained, and the system should not attempt to merge them.
  • If the partial overlap does not share a common start nor end value, the events are distinct and should be treated separately.
  • If the partial overlap shares a common start or end value, there is a possibility that the two events are mergeable, but this would likely require human intervention to assess.
• For non-overlapping Events, the system should retain both events as they represent different states or periods of the process.
• For open-ended events (where the end value is not defined), the system should retain the event as it represents an ongoing Process.
  • If there are two identical ProcessEvents with different open-ended start values, the system should retain both events as they represent different states of the process at different times.