At this time, LC does not have the resources to offer support for this open source code. While LC will make the code available, the Library does not currently promise to address any issues which are pointed out by the community beyond what is needed for the Library's own usage.
An API serving the Findings Environment for Collected Holdings application
This is a containerized application for both local and deployed environments. Below steps will get you up and running locally. Virtual Environment management is only needed for developers who will be updating the application's dependencies and working with scripts. Otherwise, everything is handled inside Containers managed out of the fetch-local repository.
This project uses containers for both local and deployed environments. You can use either Docker Desktop or Podman Desktop.
Note: The
fetch-localcompose workflow usesdocker composecommands. The standalone build scripts in this directory (dev-build.sh,test-build.sh) usepodmandirectly. Both tools use compatible CLI syntax and compose file formats.
This project's environment and dependencies are managed with Poetry.
Poetry's workflow ties the developer's virtual environment to the project itself, allowing for rapid ramp-up, team standardization, and reduced maintenance divergence. Poetry provides hooks to integrate with pre-commit, further ensuring dependency integrity on commits. This includes synchronization of Poetry's dependency management with the traditional requirements.txt used in Python frameworks. Finally, poetry also streamlines the process of publishing a Python package on PyPy should the need arise.
$ python3 -m ensurepip
$ python3 -m pip install --upgrade pip
$ pip3 install poetry==1.6.1Configure poetry to store .venv inside project structure (this is .gitignored).
$ poetry config virtualenvs.in-project trueFirst install pyenv, and add it to your PATH
$ brew install pyenv
$ nano $HOME/.zshenv
# In .zshenv
export PYENV_ROOT="$HOME/.pyenv"
export PATH="$PYENV_ROOT/bin:$PATH"
eval "$(pyenv init -)"Save your zsh environment, exit, and then reload the shell with $ zsh -l
Now create a local virtual environment.
$ cd path/to/repository/root
$ poetry installActivate the environment and verify everything looks right.
$ poetry shell
$ poetry env info
$ python --version
$ where pythonIf your terminal / shell supports it, the activated virtual environment will also display in your prompt. e.g (api-py3.11) or rather (<project-name>-python version).
If / when you want to get out of the environment, simply $ exit or $ deactivate
Dependency management
$ poetry add <package>
$ poetry remove <package>If you are using VSCode (or any IDE debugger), I'd recommend switching your debug interpretter to this virtual env so that it'll have all dependencies met. There is an included launch profile for debugging in VSCode for convenience.
./helper.sh build local
(Though I recommend instead running the whole project from the fetch-local repo instead).
Simply calling the command again will take care of tear down and rebuild. The application should now be running on your localhost at http://127.0.0.1:8001/
Under the hood, the image takes the necessary steps to generate the API's package requirements from the pyproject.toml and poetry lock files.
Additional helpful commands:
./helper.sh build-db- Rebuilds the inventory-database container../helper.sh rebuild-db- Wipes the inventory-database volume and re-seeds fake data.
Environment variables should be set through the environment. The Pydantic settings system is used to set default values for expected .env keys. The application will automatically override these if matching keys are found in the environment.
For local builds there is no need to have a .env file. For deployed environments, devops should handle seeding this file into the repository root before building.
The container images then move .env into the config directory at build, and python-dotenv is used to ensure parsing occurs prior to building the Pydantic Settings class.
The Python Slim image is used as a base for the container. This image itself is based on Debian:Buster, thus any need to interact with the container system should refer to Debian documentation.
Alembic is used for migration management. The target database for the Inventory Service is inventory_service.
Revision logic has been configured to output with an inverted date format instead of the default random strings Alembic would normally generate. This allows our files to be sequential, and the history command to display reverse chronological order.
2023_10_15_06:52:16 -> 2023_10_15_06:54:53 (head), Add Module table
2023_10_15_06:33:28 -> 2023_10_15_06:52:16, Add Module Number table
<base> -> 2023_10_15_06:33:28, Add building tableAdd models for Alembic tracking in /migrations/models.py. The order of import is initially important when first adding new tables, as to satisfy the dependency of foreign key relationships. After a migration file is generated though, it's safe to allow our pre-commit hooks to sort imports however it likes.
Unlike other common migration tools, Alembic only stores the current version in the database. We should assume that all previous revisions in the dowgrade history have been applied. A future task may be to add a separate tracking table.
Commands:
# Add a new migration file
alembic revision -m "Update some model field" --autogenerate
# Run all migrations up to the latest
alembic upgrade head
# Reverse the last migration
alembic downgrade -1
# Revert to a previous migration
alembic downgrade [target_version]
# Show the current migration version in the database
alembic current
# List version migration scripts in order
alembic historyHelper commands have been created to allow CLI migration management from outside a running container on local only (future support may be added for other environments by simply updating the MIGRATION_URL environment variable for those environments).
./helper.sh makemigrations "Your Message Here"- generates a new migration./helper.sh migrate- runs alembic upgrade head./helper.sh current- Reports the current db migration.
Containerfiles are used in deployment. For deployed environments, Kubernetes / Terraform uses the repository root for the build context. In local environments, we use either a build script or a compose file in the fetch-local repository, which assumes a different build context. This is why pathing is different between the images.
- Configure Python Interpreter Ensure that you have a Python interpreter configured for your project:
Go to File > Settings (or PyCharm > Preferences on macOS). Navigate to Project: [Your Project Name] > Python Interpreter. Select an interpreter or add a new one.
- Install Formatting Tools Install Python code formatting tools like Black, isort, or autopep8. You can do this within PyCharm or using a terminal:
Open the terminal in PyCharm or your regular terminal. Install the tools using pip. For example:
pip install black
pip install isort
pip install autopep8
- Configure Code Style in PyCharm To configure the code style settings:
Go to File > Settings > Editor > Code Style > Python. Here, you can adjust various settings like indentation, spaces, wrapping, etc., to match your preferred style or the style guide of your project. 6. Set Up Black and/or isort To integrate Black or isort:
Go to File > Settings > Tools.
For Black:
Navigate to External Tools and add a new tool.
Set 'Program' to the path of Black (you can find it with which black in the terminal), and 'Arguments' to
Go to File > Settings > Editor > Inspections. Under Python, ensure PEP 8 coding style violation is checked. 8. Format Code on Save To automatically format code on save:
Go to File > Settings > Tools > Actions on Save. Enable Reformat Code to apply PyCharm's code style settings. Optionally, enable Optimize Imports to automatically organize imports on save. 9. Use Code Reformat You can manually reformat code anytime:
Right-click on a file or folder in the Project view and select Reformat Code. Or use the shortcut Ctrl+Alt+L (Windows/Linux) or Cmd+Alt+L (macOS). 10. Additional Plugins Consider installing plugins for enhanced Python development:
Go to File > Settings > Plugins. Search for Python-related plugins like SonarLint, GitToolBox, etc., and install as needed. 11. Test Your Setup Write some Python code and try saving the file or manually reformatting to see the formatting in action.
- Install Visual Studio Code If you haven't already, download and install VS Code from the official website.
Setting up Visual Studio Code (VS Code) for Python code formatting involves installing and configuring extensions and tools that help in linting and formatting your Python code. Here's a step-by-step guide to set up VS Code for Python development with code formatting:
1.1. Install Visual Studio Code If you haven't already, download and install VS Code from the official website.
- Install the Python Extension The Python extension for VS Code provides a rich set of features including linting, debugging, IntelliSense, code navigation, code formatting, etc.
Open VS Code. Go to the Extensions view by clicking on the square icon on the sidebar, or press Ctrl+Shift+X. Search for Python (look for the extension published by Microsoft). Click on the Install button. 3. Install Python Ensure that Python is installed on your system. You can download it from the official Python website. VS Code's Python extension will automatically detect the installed Python interpreters.
- Select Python Interpreter Open a Python file or create a new one. Click on the Python version in the status bar at the bottom or use the command palette (Ctrl+Shift+P) and type Python: Select Interpreter to choose the appropriate interpreter for your project.
- Install Formatting Tools VS Code supports various Python code formatters. The most popular ones are autopep8, black, and yapf. You can install them using pip:
pip install autopep8
pip install black
pip install yapf
- Configure the Formatter in VS Code Open the Command Palette (Ctrl+Shift+P). Type Preferences: Open Settings (JSON). Add or modify the following settings to select your preferred formatter (e.g., black):
{
"python.formatting.provider": "black",
"editor.formatOnSave": true
}
- Optional: Configure Linting VS Code's Python extension supports various linters like pylint, flake8, mypy, etc. To use a linter:
Install the linter, e.g., for pylint:
pip install pylint
- Configure VS Code to use the linter:
{
"python.linting.enabled": true,
"python.linting.pylintEnabled": true
}
- Test Your Setup Create or open a Python file and write some code. Try saving the file to see if it gets formatted automatically.
This application supports two types of authentication. A legacy login endpoint is available for accepting a username and password in a POST request. This is only available in debug, local, dev, and test environments. The other method uses Single-Sign on in SAML communication with an Identity Provider. This method is available over all environments.
The Identity Provider is configured to relay to the acs endpoint on a localhost environment, but may only do so over https. To support this, the local container build uses a self-signed certificate which is generated at build time. Your browser may still give warnings about an untrusted certificate. You can either just bypass it in the browser, or (if on Mac) you can add these local certs to your keychain to stop getting the browser warning. A VSCode launch profile is included in this repository, and that too, makes use of generated self-signed certs for localhost over https.
The API exposes a login route, and a Assertion Consumer Service route for forwarding users to the identity provider, and receiving communication back from the idenity provider. SAML configuration for the environments is slightly different. At the time of this writing, these are separated via config files. At a future date, configuration should move entirely to the environment and be loaded in-memory only.
When the ACS endpoint receives an encrypted re-direction with user Auth data, it then checks and / or creates a user in the FETCH database before issuing its own encrypted token for use with the front-end application. Such a token is necessary for successful ongoing use of the FETCH webapp in an authenticated state.
A table has been created to store the audit log of table updates. The log is updated based on triggers added to each of the model tables. There are three migrations which initially set this up. One to create the table, the second to create the function, and the last to add the trigger to each of the existing tables. There is minimal code changed to support this, namely setting the user based off the user token in the request on post, patch, and deletes.
** It will be necessary to add a trigger to any new table which is added, but it is a simple migration which can be referenced from the third migration add_trigger_to_tables.
Audit log records can be viewed through the Admin panel in the frontend application. The audit trail API is available at /audit-trails/.
The Inventory Service includes an Integrated Library System (ILS) integration framework using an adapter pattern. This allows FETCH2 to communicate with external ILS platforms (e.g., FOLIO) for operations such as check-in synchronization during refile workflows.
The integration is built on a pluggable adapter interface located in app/ils/:
interfaces.py— Defines the abstractILSAdapterinterfacefolio_adapter.py— Production adapter implementing FOLIO API communication via OAuth 2.0 Client Credentialsmock_adapter.py— Mock adapter for local development and testingfactory.py— Factory that selects the appropriate adapter based on environment configurationtasks.py— Background tasks for automated ILS synchronization (e.g., refile check-in hooks)
ILS configuration is managed through the Admin panel in the frontend under Admin > ILS Configuration. The API endpoints are available at /ils-configurations/. Sync errors are tracked at /ils-sync-errors/ and viewable in the Admin UI.
The Inventory Service has a comprehensive pytest-based test suite located in the tests/ directory. Tests are organized by layer:
tests/domain/test_inventory_entities.py— Core SQLAlchemy model and constraint teststests/domain/test_location_hierarchy.py— Location hierarchy cascade and uniqueness teststests/domain/test_job_workflows.py— Full API-level workflow tests covering all 7 job types (Accession, Verification, Shelving, Picklist, Shipping, Refile, Withdrawal)tests/auth/test_rbac.py— Role-based access control security tests
See tests/README.md for detailed documentation on each test module and guidance on writing new tests.
Tests require Docker or Podman to spin up an isolated PostgreSQL container on port 5433:
pytestThe test infrastructure automatically creates a disposable database, runs Alembic migrations, seeds test data, and tears everything down after completion.