Awesome AI Engineer
A curated roadmap and resource collection for leveling up from AI Engineer to AI Systems Architect β with 18 production-ready portfolio projects, enterprise architecture patterns, and modern 2025β2026 LLM & agent engineering paradigms.
Roadmap β’
Projects β’
Learning β’
Agent Frameworks β’
Structured Outputs β’
Reasoning & DSPy β’
Memory & State β’
AI Gateways β’
MLOps & Serving β’
System Design β’
Guardrails β’
Realtime & Voice β’
Edge AI β’
Books β’
Contributing
The AI engineering landscape is evolving fast. Beyond basic prompting and wrapper scripts, modern AI systems require rigorous architecture, stateful memory, constrained decoding, test-time compute, resilient gateways, and automated evaluation. This repo cuts through the noise with:
A clear career progression from junior AI engineer to enterprise systems architect
18 hands-on portfolio projects covering LLMs, multi-agent systems, RAG, gateways, and protocols
Curated 2025β2026 production stack β structured outputs, prompt compilers, reasoning models, and realtime agents
Enterprise architecture focus β production patterns, latency/cost routing, and safety guardrails
π Click to view Text & Level-by-Level Breakdown
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β AI ENGINEER β SYSTEMS ARCHITECT β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ€
β β
β LEVEL 1: AI Engineer β
β βββ LLM fundamentals (tokenization, generation, fine-tuning) β
β βββ Prompt engineering, structured outputs & JSON schemas β
β βββ RAG pipelines, dense retrieval & vector databases β
β βββ API design for AI services & streaming (SSE/WebSockets) β
β βββ Projects: #1 #2 #5 #7 β
β β
β LEVEL 2: Senior AI Engineer β
β βββ Agent architectures (ReAct, Plan-and-Execute, Code Agents) β
β βββ Multi-agent orchestration patterns (Supervisor, Hierarchical) β
β βββ Agentic frameworks (LangGraph, CrewAI, PydanticAI, ADK) β
β βββ Prompt optimization & compilers (DSPy, TextGrad) β
β βββ Streaming voice & real-time AI systems β
β βββ Projects: #3 #4 #6 #8 β
β β
β LEVEL 3: AI Platform Engineer β
β βββ MLOps & high-throughput serving (vLLM, SGLang, TensorRT-LLM) β
β βββ AI Gateways & smart routing (LiteLLM, RouteLLM, Portkey) β
β βββ Agent memory systems & persistent state (Letta, Mem0, Zep) β
β βββ Kubernetes for AI workloads & GPU orchestration β
β βββ Observability, tracing & evaluation (LangSmith, Phoenix) β
β βββ Protocol design & tool standards (MCP, A2A, UCP) β
β βββ Projects: #9 #10 #11 #12 #13 β
β β
β LEVEL 4: AI Systems Architect β
β βββ Enterprise multi-agent system design at scale β
β βββ Reasoning models, test-time compute & inference scaling β
β βββ Context caching economics & KV-cache optimization β
β βββ Runtime guardrails, security & defense against injection β
β βββ Cross-framework interoperability & model cascading β
β βββ Projects: #14 #15 #16 #17 #18 β
β β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
```
18 production-ready projects covering foundational LLMs to enterprise-scale AI architecture, each equipped with Docker setup, automated tests, CI/CD, and Kubernetes manifests.
Foundation & Agent Workflows (Level 1-2)
#
Project
What You'll Build
Key Skills
Repo
1
LLM Playground
Interactive tokenization & generation explorer
BPE tokenization, sampling strategies, temperature
2
Customer Support Chatbot
Production chatbot with fine-tuning & evaluation
LoRA/PEFT, prompt engineering, evaluation metrics
3
Ask-the-Web Agent
Perplexity-like search & synthesis agent
LangGraph, tool use, citation generation
4
Deep Research
Multi-strategy reasoning engine
Chain-of-Thought, Tree-of-Thought, inference scaling
5
Image Generation
Multi-provider image service
Diffusion models, DALL-E, FLUX, Stable Diffusion
6
Capstone Multi-Agent
Supervisor-pattern agent platform
Multi-agent orchestration, task decomposition
7
Agent RAG
Hierarchical RAG system
Single/multi/hierarchical retrieval, query decomposition
8
MCP & A2A
Protocol implementation
MCP SDK, A2A protocol, AgentCards
Platform, Gateways & Realtime Streaming (Level 3)
#
Project
What You'll Build
Key Skills
Repo
9
UCP Merchant Server
Commerce protocol platform
Checkout state machine, AP2 verification, MCP bindings
10
UCP Shopping Agent
AI shopping assistant
Multi-merchant discovery, comparison, LangGraph
11
Enterprise AI Gateway
Resilient multi-provider routing gateway
LiteLLM, RouteLLM, semantic caching, rate limiting
12
Realtime Voice Agent
Sub-second bidirectional voice & multimodal agent
WebRTC / WebSockets, streaming STT/TTS, tool interrupts
13
Stateful Memory Agent
Tiered memory agent with entity knowledge graph
Letta/Mem0 paradigms, episodic recall, semantic store
Enterprise Systems, Compilers & Governance (Level 4)
#
Project
What You'll Build
Key Skills
Repo
14
DSPy Prompt Optimizer
Self-improving declarative prompt compiler
DSPy MIPROv2, teleprompters, metric optimization
15
AI Guardrails Firewall
Zero-trust GenAI security & injection defense
NeMo Guardrails, Llama Guard 3, PII redaction
16
Compliance Audit Agents
Regulatory compliance audit system
Microsoft Agent Framework, graph workflows, middleware
17
Incident Response ADK
IT incident orchestrator
Google ADK / Genkit, Sequential/Parallel/Loop agents
18
Contract Lifecycle Crew
Contract management platform
CrewAI, role-based agents, Flow orchestration
Quick Start (Any Project)
# Clone any project repository
git clone https://github.com/Skopaq-AI/< project-name> .git
cd < project-name>
# Option 1: Docker (recommended)
docker compose up --build
# Option 2: Local development
pip install -e " .[dev]"
uvicorn src.< package> .main:app --reload --port 8000
# Run test suite
pytest tests/ -v
π Foundational Learning
Prompt Engineering & Optimization
π€ Agent Frameworks & Tools
Framework
Best For
Maintainer
Link
LangGraph
Complex stateful agent workflows with cycles and persistence
LangChain
Docs
PydanticAI
Type-safe, production-grade agent framework built on Pydantic
Pydantic
Docs
CrewAI
Role-based multi-agent collaboration and hierarchical flows
CrewAI
Docs
Smolagents
Lightweight code-agent framework running secure sandboxed Python
Hugging Face
GitHub
Google Genkit / ADK
Agent Development Kit with enterprise workflows and Firebase integration
Google
Docs
Microsoft Agent Framework
Enterprise agents with middleware pipelines and Azure integration
Microsoft
GitHub
OpenAI Agents SDK
Lightweight agent loops with multi-agent handoffs
OpenAI
Docs
LlamaIndex
Data-centric RAG and document-native agent workflows
LlamaIndex
Docs
AutoGen
Event-driven multi-agent conversation patterns
Microsoft
Docs
Semantic Kernel
Enterprise AI orchestration for .NET, Python, and Java
Microsoft
Docs
Agent Protocols & Standards
Protocol
Purpose
Link
MCP (Model Context Protocol)
Standardized protocol for connecting AI models to tools and resources
Spec
A2A (Agent-to-Agent)
Standardized inter-agent communication protocol
Spec
UCP (Universal Commerce Protocol)
Secure agentic commerce and checkout transaction standard
Spec
OpenAI Function Calling
Structured tool use standard adopted across major providers
Docs
Tool
Purpose
Link
LangSmith
LLM observability, step-level tracing, and dataset evaluations
Docs
Arize Phoenix
Open-source AI observability, evals, and OpenTelemetry tracing
GitHub
Weights & Biases Weave
Lightweight AI application tracing, scoring, and evaluation
Docs
Braintrust
Enterprise evaluation platform, logging proxy, and prompt playground
Docs
OpenLLMetry
OpenTelemetry-based instrumentation for LLM applications
GitHub
π Structured Outputs & Constrained Decoding
Deterministic schema enforcement is essential for production agents, APIs, and data extraction pipelines.
Tool
Purpose
Link
Instructor
Structured outputs for LLMs via Pydantic β validation, retries, and streaming
Docs
Outlines
Fast, guided text generation enforcing regex and JSON schemas via CFGs
GitHub
Guidance
Constrained decoding language controlling generation structure and interleaving
GitHub
TypeChat
Schema-guided structured output generation using TypeScript definitions
GitHub
Jsonformer
Bulletproof JSON generation generating only schema-fill tokens
GitHub
π§ Reasoning, Prompt Compilers & Optimization
Modern AI engineering compiles prompts programmatically and leverages test-time search.
Tool / Concept
Purpose
Link
DSPy
Stanford framework for compiling declarative pipelines into optimized prompts & weights
Docs
TextGrad
Automatic "differentiation" via text feedback for prompt and system optimization
GitHub
DeepSeek-R1
Open reasoning models demonstrating reinforcement learning test-time scaling
GitHub
Process Reward Models (PRMs)
Step-by-step verification and tree-search scoring for complex problem solving
Paper
Tree-of-Thought (ToT)
Deliberate exploration and heuristic evaluation of reasoning paths
GitHub
πΎ Agent Memory & State Management
Stateful agents require distinct memory layers: short-term context, working memory, episodic memory, and long-term semantic memory.
Tool
Purpose
Link
Letta (formerly MemGPT)
Stateful agent platform managing working memory and archival storage
GitHub
Mem0
Personalized memory layer for AI agents, extracting entity and user preferences
GitHub
Zep
Long-term memory server providing auto-summarization and knowledge graphs
Docs
LangGraph Store
Built-in key-value and semantic cross-thread persistence for stateful agents
Docs
π AI Gateways, Routing & Cost Optimization
Production AI architectures implement unified gateways for rate-limiting, failover, prompt caching, and cost-aware routing.
Tool
Purpose
Link
LiteLLM
Unified proxy for 100+ LLMs with load balancing, retries, and spend tracking
GitHub
RouteLLM
Framework for serving and evaluating cost-effective LLM routing
GitHub
Portkey Gateway
Blazing fast AI gateway with 50+ integrations, semantic cache, and fallbacks
GitHub
OpenRouter
Multi-provider unified routing and competitive pricing API
Site
Helicone
Developer observability proxy for caching, rate limiting, and cost analytics
Docs
βοΈ MLOps, Serving & Infrastructure
High-Throughput Model Serving
Tool
Purpose
Link
vLLM
High-throughput LLM serving with PagedAttention and prefix caching
GitHub
SGLang
High-performance serving engine for LLMs/VLMs with RadixAttention
GitHub
TensorRT-LLM
NVIDIA-optimized LLM inference execution engine
GitHub
Triton Inference Server
Multi-framework production model serving at scale
GitHub
Ollama
Lightweight CLI to run open-source models locally with one command
GitHub
llama.cpp
Ultra-efficient CPU/GPU LLM inference in C/C++
GitHub
Vector Databases & Retrieval Engines
Database
Strengths
Link
Qdrant
Rust-based vector search engine with payload filtering and quantization
Docs
Pinecone
Serverless managed vector database with hybrid search
Docs
Weaviate
Open-source vector database with multi-modal capabilities and hybrid search
Docs
pgvector
PostgreSQL extension enabling vector embeddings in relational DBs
GitHub
Milvus
Distributed, highly scalable vector database for billion-scale search
Docs
ChromaDB
Lightweight, open-source embedding database for rapid prototyping
Docs
Tool
Purpose
Link
Unsloth
2-5x faster LLM fine-tuning with 80% less VRAM
GitHub
Hugging Face TRL
Transformer Reinforcement Learning (DPO, ORPO, GRPO, SFT)
Docs
Axolotl
Production-ready LLM fine-tuning configured through YAML
GitHub
PEFT
Parameter-efficient fine-tuning (LoRA, QLoRA, Prompt Tuning)
Docs
LitGPT
Hackable 20+ LLM pretraining and fine-tuning engine by Lightning AI
GitHub
ποΈ System Design for AI
Pattern
When to Use
Reference / Project Example
Agentic RAG
Dynamic multi-step retrieval, query routing & self-correction
Agent RAG (#7)
GraphRAG
Complex entity-relationship reasoning across document corpora
Microsoft GraphRAG
ReAct (Reason + Act)
Interleaving thought steps with tool invocations
Ask-the-Web (#3)
Supervisor Pattern
Central orchestrator routing sub-tasks to specialized domain agents
Capstone (#6)
Prompt Caching
Long-context apps to reduce latency and API costs by up to 90%
Anthropic Prompt Caching
Sequential Pipeline
Deterministic ordered multi-step processing
Incident Response (#12)
Parallel Fan-Out
Concurrent independent agent executions
Incident Response (#12)
Human-in-the-Loop
Approval gates, validation interrupts & escalation
Compliance Audit (#11)
Model Cascading
Routing simple queries to SLMs and complex reasoning to frontier models
RouteLLM
State Machine
Transactional, verifiable multi-step commerce workflows
UCP Merchant (#9)
Tool
Purpose
Link
RAGAS
Automated evaluation framework for RAG pipelines (faithfulness, answer relevancy)
Docs
DeepEval
Unit testing framework for LLMs with 14+ production metrics
GitHub
Promptfoo
Automated testing, red-teaming, and regression testing for prompts
Docs
Inspect AI
Framework for large language model evaluations and safety benchmarks by AISI
GitHub
π‘οΈ Runtime Guardrails & Security
Tool / Framework
Purpose
Link
NeMo Guardrails
Programmable rails for dialogue safety, topical control, and hallucination reduction
GitHub
Guardrails AI
Real-time output validation, structured assertions, and schema correction
GitHub
Llama Guard
Fine-tuned safeguard model for input/output safety classification by Meta
Docs
Lakera AI
Real-time defense against prompt injection, jailbreaking, and data leakage
Site
Garak
LLM vulnerability scanner identifying hallucination, injection, and security flaws
GitHub
OWASP Top 10 for LLMs
Standardized threat model for generative AI and LLM application security
Docs
ποΈ Realtime Voice & Multimodal Systems
Tool / Standard
Purpose
Link
LiveKit Agents
Realtime voice, video, and multimodal agent framework with WebRTC
Docs
OpenAI Realtime API
Low-latency, speech-to-speech bidirectional streaming over WebSockets
Docs
Faster-Whisper
Reimplementation of OpenAI's Whisper model using CTranslate2 for ultra-fast transcription
GitHub
ColPali
Efficient vision-language retrieval model indexing documents as images
GitHub
π± Edge AI & On-Device Deployment
Tool / Framework
Target
Link
Apple MLX
Optimized machine learning framework designed for Apple Silicon GPUs/Unified Memory
GitHub
ExecuTorch
PyTorch end-to-end solution for on-device inference on mobile and edge devices
GitHub
ONNX Runtime
Cross-platform accelerator for high-performance ML inference on edge and cloud
Site
Transformers.js
Run state-of-the-art transformer models directly in browsers and Node.js via WebGPU
GitHub
βοΈ Cloud & Kubernetes for AI
Provider
Key Services
Link
Google Cloud
Vertex AI, Gemini APIs, Cloud TPUs, GKE Enterprise for AI
Docs
AWS
Amazon Bedrock, SageMaker HyperPod, Trainium/Inferentia accelerators
Docs
Azure
Azure OpenAI Service, Azure AI Foundry, AKS with GPU operators
Docs
Kubernetes for AI Workloads
Resource
Description
Link
KServe
Serverless model serving standard on Kubernetes
Docs
KubeFlow
Cloud-native end-to-end ML platform on Kubernetes
Docs
Ray on Kubernetes (KubeRay)
Distributed ML compute and batch training on K8s
Docs
NVIDIA GPU Operator
Automated GPU driver, container runtime, and telemetry management in K8s
Docs
Book
Author
Why It Matters
AI Engineering
Chip Huyen
Definitive guide to building reliable LLM applications in production
Designing Machine Learning Systems
Chip Huyen
System design fundamentals from data pipelines to model serving
Build a Large Language Model (From Scratch)
Sebastian Raschka
Deep understanding of transformer internals and tokenizers
Hands-On Large Language Models
Jay Alammar, Maarten Grootendorst
Visual, practical patterns for modern LLM applications
Designing Data-Intensive Applications
Martin Kleppmann
Foundational data architecture principles essential for AI systems architects
π° Blogs, Podcasts & Newsletters
Resource
Type
Focus
Link
Lil'Log
Blog
Deep technical surveys on LLMs and generative models by Lilian Weng
Blog
Eugene Yan
Blog
Applied ML systems design, search, and production patterns
Blog
Latent Space
Podcast / Newsletter
The leading technical podcast and community for AI engineers
Listen
The Batch
Newsletter
Weekly AI news and insights curated by Andrew Ng
Subscribe
Interconnects
Newsletter
Nathan Lambert on RLHF, open models, and reasoning architectures
Subscribe
Contributions are welcome! Please check out CONTRIBUTING.md for contribution guidelines, table schema formats, and PR rules.
This work is licensed under CC0 1.0 Universal .