The Ultimate Resource for Testing AI/ML Systems - Curated Collection of Frameworks, Tools, Methodologies, and Best Practices
Testing AI/ML systems is crucial yet challenging due to their unique characteristics:
- Non-deterministic behavior: Models may produce different outputs for same inputs
- Data dependencies: Performance heavily relies on training data quality
- Evolving nature: Models degrade over time requiring continuous monitoring
- Complex evaluation: Traditional metrics may not capture real-world performance
This repository provides comprehensive resources to address these challenges and ensure your AI/ML systems are reliable, fair, and production-ready.
Help spread the word about this resource by sharing it with your network:
Testing AI/ML systems presents unique challenges beyond traditional software testing. This curated list provides resources for ensuring the quality, reliability, and fairness of your AI/ML systems throughout their lifecycle.
Whether you're new to ML testing or an experienced practitioner, this list offers valuable resources for implementing effective testing strategies for your AI/ML applications.
- General ML Testing Frameworks
- LLM & Chatbot Testing
- Data Validation & Quality
- Fairness, Bias & Ethics Testing
- Adversarial Testing
- Monitoring & Observability
- CI/CD for ML
- Multi-modal Testing
- Regulatory Compliance Testing
- Generative AI Evaluation
- Hardware-aware ML Testing
- AI Safety Testing
- Benchmarks & Datasets
- Books & Academic Papers
- Courses & Tutorials
- Blogs & Newsletters
- Communities & Conferences
- Case Studies
- Deepchecks - Test suite for data and models, validating a wide range of potential issues.
- MLtest - Testing infrastructure for machine learning with focus on TensorFlow.
- Giskard - Testing framework focused on vulnerability scanning and detection.
- Pytest-ml - Extension of pytest for validating ML code and models.
- Manifold - Visual debugging tool for machine learning models.
- ModelDB - System to version ML models and track experiments.
- MLFlow - Platform for the ML lifecycle, including model testing.
- HyperDrive - Hyperparameter tuning and neural architecture search.
- Kedro - Framework for creating reproducible ML pipelines.
- Seldon Alibi - Algorithms for monitoring and explaining machine learning models.
- Determined AI - Deep learning training platform with built-in validation.
- Ray Tune - Distributed hyperparameter tuning framework.
- LangChain Evaluation Framework - Testing framework for LLM applications.
- HELM - Holistic Evaluation of Language Models.
- LLM-Eval - Framework for evaluating language models.
- Botium - Testing automation for conversational AI.
- BERTScore - Evaluating text generation with BERT.
- ChatEval - Scientific framework for evaluating chatbots.
- ROUGE - Evaluation metric for summarization tasks.
- Rasa Testing Framework - Built-in testing for Rasa conversational agents.
- promptfoo - Testing and evaluation framework for LLM prompts.
- TruLens - LLM evaluation focused on faithfulness, quality and safety.
- Guardrails - Framework to validate and clean language model outputs.
- DeepEval - Evaluation framework for LLMs with human-like metrics.
- Ragas - Framework for evaluation of RAG (Retrieval Augmented Generation) systems.
- Promptimize - A/B testing and optimization for LLM prompts.
- LlamaIndex Evaluation - Evaluation modules for RAG applications.
- TruLens-Eval - Advanced evaluation metrics for LLMs beyond traditional metrics.
- MT-Bench - Multi-turn benchmark for LLM conversation quality.
- Anthropic's Claude Evaluations - Specialized testing framework for Constitutional AI models.
- FairEval - Framework for evaluating fairness in language models.
- AgentBench - Framework for evaluating LLM agents on complex tasks.
- ai-evaluation - Open-source LLM evaluation framework with 50+ metrics, LLM-as-Judge, and guardrail scanners (jailbreak, PII, injection).
- Great Expectations - Framework for data validation and quality.
- TFDV - TensorFlow Data Validation for analyzing and validating ML data.
- Deequ - Data quality validation for large datasets (by AWS).
- Pandera - Statistical data validation for pandas.
- Cerberus - Lightweight and extensible data validation library.
- Data-Linter - Tool to identify common data quality issues.
- DataProfiler - Profiling and detecting sensitive information in data.
- Soda Core - Data quality testing and monitoring.
- PyDeequ - Python wrapper for Deequ library.
- Deepchecks Data Integrity - Specialized tests for data integrity.
- Validio - Data validation pipeline tool.
- Frictionless Data - Data validation framework for tabular data.
- AI Fairness 360 - Toolkit for detecting and mitigating bias in ML models.
- What-If Tool - Visual tool for investigating model behavior.
- FairML - Auditing black-box predictive models for bias.
- Aequitas - Bias and fairness audit toolkit.
- Fairlearn - Tools for assessing and improving fairness of ML systems.
- ML Fairness Gym - Simulation platform for fairness research.
- LiFT - LinkedIn Fairness Toolkit for measuring bias in ML workflows.
- Themis ML - Library for fairness-aware machine learning.
- FairKit - Tools for fair and explainable ML models.
- Bias Buccaneers - Detecting and mitigating bias in NLP models.
- Gender Bias Test - Testing and mitigating gender bias in word embeddings.
- AI Ethics Toolbox - Practical tools for ethical AI development.
- TextAttack - Framework for adversarial attacks on NLP models.
- Adversarial Robustness Toolkit - Testing model robustness against attacks.
- CleverHans - Library for benchmarking vulnerability to adversarial examples.
- FoolBox - Python toolbox to create adversarial examples.
- DeepFool - Simple algorithm to find adversarial perturbations.
- EvalAI - Open source platform for evaluating AI algorithms.
- Counterfit - Command-line tool for security testing of ML systems.
- RAID - Robust AI framework for detecting adversarial inputs.
- Alibi Detect - Tools for outlier, adversarial, and drift detection.
- PGD Attack - Implementation of projected gradient descent attack.
- FGM Attack - Fast gradient sign method for adversarial attacks.
- Threat of Neural Fake News - Tools for detecting neural fake news.
- Evidently AI - Tools for ML model monitoring and evaluation.
- WhyLogs - Profiling library for ML/AI data and model monitoring.
- Seldon Core - Platform for deploying ML models with monitoring.
- Prometheus - Monitoring system adaptable for ML metrics.
- Grafana - Analytics & monitoring solution for metrics visualization.
- Arize AI - ML observability platform.
- NannyML - Post-deployment model performance monitoring.
- BentoML - Model serving with monitoring capabilities.
- Datadog - ML monitoring integrated with infrastructure.
- Fiddler AI - Model monitoring focused on explainability.
- traceAI - Open-source OpenTelemetry-native tracing for LLM and agent apps with 50+ framework integrations.
- Gantry - Production ML monitoring and debugging.
- Aporia - Model observability platform.
- Helicone - Open-source LLM observability platform.
- LangSmith - Observability and evaluation platform for LLM applications.
- Superwise - ML monitoring focused on sensitive data applications.
- Prometheus ML Metrics - Specialized ML metrics for Prometheus.
- MLflow Monitoring Extensions - Advanced monitoring plugins for MLflow.
- Kubeflow - ML toolkit for Kubernetes with CI/CD capabilities.
- GitHub Actions for ML - GitHub Actions workflows for ML.
- DVC - Git for data science, enables versioning of data and models.
- CML - Continuous Machine Learning - CI/CD for ML projects.
- Jenkins ML Plugin - Plugin connecting Jenkins with ML libraries.
- MLReef - GitLab-based collaboration platform for ML.
- Pachyderm - Data lineage and version control for data science.
- ZenML - MLOps framework focusing on reproducibility.
- MLflow CI/CD - Continuous integration features of MLflow.
- Metaflow - Netflix's framework for building data science pipelines.
- TFX - TensorFlow Extended for ML pipelines.
- Flyte - Kubernetes-native workflow orchestration platform.
- MMMU - Massive Multi-discipline Multimodal Understanding benchmark for testing multimodal models across 30+ disciplines.
- MME - Multimodal Evaluation benchmark for testing vision-language models on fine-grained capabilities.
- MultiModal-Eval - Framework for testing multimodal models across diverse tasks and data types.
- LLaVA-Bench - Benchmarking suite for vision-language models on instruction following and reasoning.
- Modular Testing Framework for Multimodal Models - Testing framework that separates modality-specific testing from integration testing.
- AI-RMF - NIST AI Risk Management Framework implementation tools.
- GDPR Compliance Scanner - Tools for checking AI systems against GDPR requirements.
- OpenAI Evals for Legal Compliance - Extensions to OpenAI's evaluation framework focused on legal requirements.
- EU AI Act Compliance Toolkit - Tools specifically designed for testing compliance with the EU AI Act.
- ML Privacy Meter - Framework for measuring privacy leakage in machine learning models.
- GenBench - Framework for comprehensive evaluation of generative AI systems.
- Image Generation Evaluator - Tools for evaluating the quality and diversity of generated images.
- SEGA - Self-Guided Evaluation framework for generative models.
- Music Generation Testing Framework - Testing tools for music generation systems.
- Text2Image-Evaluation - Framework for evaluating text-to-image generation models.
- DeepPerf - Framework for testing ML models across various hardware configurations.
- TVM-Benchmark - Testing suite for model performance on diverse hardware targets.
- MLPerf for Edge Devices - Benchmark suite for ML inference on edge devices.
- PowerInfer - Testing framework for power consumption of ML inference.
- TensorRT Test Automation - Tools for testing hardware acceleration of neural networks.
- AI Safety Gridworlds - Test environments for AI safety research.
- AIS Testbed - Open-source framework for testing AI safety properties.
- Anthropic's Constitutional AI Testing - Testing framework for constitutional AI approaches.
- Alignment Evaluation - Tools for testing goal alignment in agent-based AI systems.
- Robustness Gym - Testing toolkit for evaluating robustness of AI systems.
- MLPerf - Industry standard for measuring ML performance.
- GLUE Benchmark - General Language Understanding Evaluation benchmark.
- SuperGLUE - More challenging extension of GLUE.
- TensorFlow Model Garden - Models and datasets for benchmarking.
- HuggingFace Datasets - Collection of datasets for NLP tasks.
- Papers With Code Benchmarks - Benchmark datasets from research papers.
- BIG-bench - Beyond the Imitation Game benchmark for LLMs.
- MMLU - Massive Multitask Language Understanding benchmark.
- LM Evaluation Harness - Framework for evaluating language models.
- OLID - Offensive Language Identification Dataset.
- SQuAD - Stanford Question Answering Dataset.
- BenchMarks.AI - Comprehensive benchmarking platform for AI.
- ML Test Score Paper - Google's framework for ML testing.
- Testing in AI: From Component to System - Comprehensive overview of AI testing.
- What's your ML Test Score? - Google's checklist for ML system testing.
- Responsible AI Practices - Google's guide for responsible AI development.
- A Survey on Concept Drift Adaptation - Addressing data drift in ML systems.
- Quality Attributes and Trade-offs in ML Systems - Understanding ML quality attributes.
- ML-Ops: Machine Learning as an Engineering Discipline - Overview of ML operations including testing.
- Machine Learning Testing: Survey, Landscapes and Horizons - Comprehensive survey of ML testing techniques.
- Testing and Validating Machine Learning Systems - Practical guide to ML testing.
- Building Machine Learning Powered Applications - O'Reilly book with testing chapters.
- Designing Data-Intensive Applications - Book covering data reliability in ML systems.
- Software Engineering for Machine Learning - Microsoft's framework for ML system development.
- Testing and Monitoring ML Models - Coursera course on ML testing.
- MLOps Specialization - DeepLearning.AI course including testing.
- Practical Machine Learning - Johns Hopkins course with model validation.
- AI Quality Testing Tutorial - Tutorial on ethical ML testing.
- Google ML Testing Guide - Google's guide to ML testing.
- Microsoft Responsible AI Resources - Educational content on responsible AI.
- Fast.ai - Practical courses including model validation and testing.
- Full Stack Deep Learning - Course with emphasis on testing & deployment.
- Udacity ML DevOps Engineer - Nanodegree on ML operations.
- ML Engineering for Production - Andrew Ng's course on MLOps.
- Made With ML MLOps Course - Free MLOps curriculum including testing.
- Weights & Biases Model Testing Tutorial - Practical guide to testing ML models.
- MLOps Community Blog - Articles on ML operations and testing.
- Neptune.ai Blog - Content focused on ML experimentation and testing.
- Deepchecks Blog - Articles on ML testing and validation.
- Chip Huyen's ML Systems Design Newsletter - Newsletter covering ML systems topics.
- TWIML AI Podcast - Podcast with episodes on ML testing and reliability.
- Google AI Blog - Google's AI research blog with testing topics.
- Microsoft AI Blog - Content on responsible AI development.
- Papers with Code Newsletter - Updates on ML research and benchmarks.
- Dair.ai Newsletter - Digest on AI research including testing practices.
- ML Ops Roundup - Newsletter focused on ML operations.
- Applied AI Newsletter - Practical insights for AI applications.
- The Batch - Andrew Ng's weekly newsletter with ML insights.
- ML-Ops Community - Community focused on ML operations including testing.
- Papers with Code - Platform for ML research including benchmarking.
- SciKit-Learn Community - Community for ML in Python with testing practices.
- ICLR - Conference on representation learning with testing tracks.
- NeurIPS - Machine learning conference with ML testing topics.
- ACM FAccT - Conference on Fairness, Accountability, and Transparency.
- CIKM - Conference on Information and Knowledge Management with ML focus.
- MLSys - Conference focused on ML systems including testing.
- ML Safety Workshop - Workshop on safety aspects of ML systems.
- MLOps World - Conference dedicated to ML operations.
- Responsible AI Forum - Forum on ethical AI development.
- AIES Conference - AI, Ethics, and Society conference.
- Uber's ML Testing Framework - How Uber built Manifold for ML testing.
- Netflix's Metaflow Testing - Netflix's approach to ML testing.
- Spotify's ML Testing - How Spotify tests music classification systems.
- Airbnb's ML Model Testing - Airbnb's approach to ML QA.
- Google's Model Cards - Framework for responsible ML model reporting.
- Microsoft's Responsible AI Implementation - Microsoft's approach to AI testing.
- LinkedIn's Fairness Testing - LinkedIn's approach to fairness in recommendations.
- Quantum ML Testing - Tools for testing quantum machine learning models
Automated updates from GitHub and arXiv APIs
Resources submitted by our community
Curated selection of top resources
Featured content from our partners
- Neuro-Symbolic Testing - Combining neural and symbolic approaches for robust testing
- Explainable AI Testing - Frameworks for testing model explainability
- Federated Learning Testing - Tools for testing distributed ML systems
- Causal ML Testing - Testing causal relationships in ML models
- Self-Supervised Testing - Frameworks for testing self-supervised learning approaches
- ML Testing Playground - Interactive environment for testing ML models
- AI Testing Sandbox - Web-based interface for testing AI systems
- ML Testing Notebooks - Jupyter notebooks with testing examples
- AI Testing Challenges - Hands-on testing problems and solutions
- ML Testing Simulator - Simulation environment for testing ML systems
- Data Drift Detection - Tools for identifying and handling data drift
- Model Decay Monitoring - Solutions for tracking model performance degradation
- Edge Case Identification - Frameworks for finding and testing edge cases
- Adversarial Robustness - Testing models against adversarial attacks
- Scalability Testing - Tools for testing model performance at scale
- Open Source ML Testing Projects - Curated list of community-driven testing tools
- ML Testing Hackathons - Community events focused on ML testing
- Testing Case Studies - Real-world examples from the community
- Testing Tool Contributions - Showcase of community contributions
- Testing Research Papers - Community-driven research in ML testing
We gratefully acknowledge the contributions from our community members:
Your name could be here!
Learn how to contribute
Your contributions are always welcome! Please read the contribution guidelines first.