Multimodality, Protocols, Autonomy, and Swarms
- Vision Transformers and the Patch-Token Primitive
- CLIP and Contrastive Vision-Language Pretraining
- BLIP-2 Q-Former as Modality Bridge
- Flamingo and Gated Cross-Attention
- LLaVA and Visual Instruction Tuning
- Any-Resolution Vision — Patch-n'-Pack and NaFlex
- Open-Weight VLM Recipes: What Actually Matters
- LLaVA-OneVision: Single, Multi, Video
- Qwen-VL Family and Dynamic-FPS Video
- InternVL3 Native Multimodal Pretraining
- Chameleon Early-Fusion Token-Only
- Emu3 Next-Token Prediction for Generation
- Transfusion Autoregressive + Diffusion
- Show-o Discrete-Diffusion Unified
- Janus-Pro Decoupled Encoders
- MIO Any-to-Any Streaming
- Video-Language Temporal Grounding
- Long-Video at Million-Token Context
- Audio-Language Models: Whisper to AF3
- Omni Models: Thinker-Talker Streaming
- Embodied VLAs: RT-2, OpenVLA, π0, GR00T
- Document and Diagram Understanding
- ColPali Vision-Native Document RAG
- Multimodal RAG and Cross-Modal Retrieval
- Multimodal Agents and Computer-Use (Capstone)
- The Tool Interface
- Function Calling Deep Dive
- Parallel and Streaming Tool Calls
- Structured Output
- Tool Schema Design
- MCP Fundamentals: Stateless Requests and JSON-RPC
- Building an MCP Server: Stateless Python and TypeScript
- Building an MCP Client: Discovery, Routing, and Dual-Era Fallback
- MCP Transports: stdio and Stateless Streamable HTTP
- MCP Resources and Prompts: Addressable Context for Stateless Servers
- MCP Model Input: Sampling Migration and Stateless MRTR
- Explicit Scope and Stateless Elicitation
- MCP Tasks Extension: Durable Work on a Stateless Core
- MCP Apps on the Stateless Protocol
- MCP Security: Poisoned Metadata, Routing, and MRTR State
- MCP Authorization: CIMD, Issuer Binding, PKCE, and Step-Up
- Stateless MCP Gateways and Registry Admission
- MCP Auth in Production: Issuer-Bound Enrollment and Tokens
- A2A Protocol
- OpenTelemetry GenAI
- LLM Routing Layer
- Agent Skills: Portable Contract and Runtime Boundary
- Capstone: Stateless Tool Ecosystem
- Skill Discovery and Progressive Disclosure
- Skill Invocation and Routing
- Skill Permissions, Sandboxes, and Trust
- Skill Evals, Packaging, and Portability
- MCP Tool Contracts and Content
- MCP Reliability, Cancellation, and Flow Control
- MCP Registry Supply Chain: Admission, Drift, and Rollback
- MCP Conformance Engineering: Versioning, Evidence, and Operations
- Program Anatomy: An ELF Reader From Scratch
- Disassembly and Control Flow: From Bytes to a Graph You Can Check
- Lifting to Pseudocode: IR, SSA, and Structuring From Scratch
- Evidence, Confidence, and Unknowns in Binary Analysis
- Rebuild and Prove Equal: Differential Tests, Byte Comparison, and the Evidence Loop
- The Reverse Engineering Landscape: History, Tool Families, and How a Decompiler Is Built
- Calling Conventions and Stack Frames: System V, Windows x64, and AArch64
- Compiler Idioms: What Optimization Does to Your Code and How to Read It Back
- Linking, Loading, and Stripped Binaries: PLT, GOT, Imports, Fixups, and Function Identification
- Recovering Data and Types: Structs, Arrays, C++ Objects, and DWARF
- Dynamic Analysis of Your Own Program: Debuggers, Traces, and Confirming Static Claims
- Beyond Native Code: Python, JVM, .NET, and WebAssembly Bytecode
- Binary Diffing and Symbolic Execution: Matching Functions and Solving for Inputs
- AI-Assisted Reverse Engineering: LLM Decompilers, Analysis Agents, and Ground Truth
- The Agent Loop
- ReWOO and Plan-and-Execute
- Reflexion and Verbal Reinforcement Learning
- Tree of Thoughts and LATS
- Self-Refine and CRITIC
- Tool Use and Function Calling
- Agent Memory — Virtual Context and Memory Paging
- Memory Blocks and Sleep-Time Compute
- Hybrid Memory — Vector + Graph + KV
- Skill Libraries and Lifelong Learning (Voyager)
- Planning with HTN and Evolutionary Search
- Anthropic's Workflow Patterns
- Stateful Graph Orchestration — Durable Execution and Checkpoints
- The Actor Model for Agents
- Role-Based Agent Teams — Roles, Tasks, Processes
- OpenAI Agents SDK — Handoffs, Guardrails, Tracing
- The Harness as a Library — Subagents and Session Store
- Production Agent Runtimes
- Benchmarks — SWE-bench, GAIA, AgentBench
- Benchmarks — WebArena and OSWorld
- Computer Use — Claude, OpenAI CUA, Gemini
- Voice Agents — Pipecat and LiveKit
- OpenTelemetry GenAI Semantic Conventions
- Agent Observability — Langfuse, Phoenix, Opik
- Multi-Agent Debate and Collaboration
- Failure Modes — Why Agents Break
- Prompt Injection and the PVE Defense
- Orchestration Patterns — Supervisor, Swarm, Hierarchical
- Production Runtimes — Queue, Event, Cron
- Eval-Driven Agent Development
- Agent Workbench: Why Capable Models Still Fail
- The Minimal Agent Workbench
- Agent Instructions as Executable Constraints
- Repo Memory and Durable State
- Initialization Scripts for Agents
- Scope Contracts and Task Boundaries
- Runtime Feedback Loops
- Verification Gates
- Reviewer Agent: Separate Builder from Marker
- Multi-Session Handoff
- The Workbench on a Real Repo
- Capstone: Ship a Reusable Agent Workbench Pack
- Frame the Task Before the Agent Writes Code
- Build an Evidence-Backed Execution Plan
- Delegate Agent Work with Isolation and Merge Contracts
- Turn Every Agent Correction into a System Improvement
- Define the Outcome Before You Choose the Output
- Discover the Workflow People Actually Perform
- Map Assumptions and Resolve the Riskiest One First
- Choose the Smallest Slice That Can Change the Decision
- Write Specifications That Preserve Judgment
- Design Success Metrics Before the Result Exists
- Choose Prototype, Pilot, or Production Deliberately
- Build a Feedback Ratchet with Ownership and Retirement
- From Chatbots to Long-Horizon Agents (METR)
- STaR, V-STaR, Quiet-STaR: Self-Taught Reasoning
- AlphaEvolve: Evolutionary Coding Agents
- Darwin Gödel Machine: Self-Modifying Agents
- AI Scientist v2: Workshop-Level Research
- Automated Alignment Research (Anthropic AAR)
- Recursive Self-Improvement: Capability vs Alignment
- Bounded Self-Improvement Designs
- Autonomous Coding Agent Landscape (SWE-bench, CodeAct)
- Permission Modes for Autonomous Agents
- Browser Agents and Indirect Prompt Injection
- Durable Execution for Long-Running Agents
- Action Budgets, Iteration Caps, Cost Governors
- Kill Switches, Circuit Breakers, Canary Tokens
- HITL: Propose-Then-Commit
- Checkpoints and Rollback
- Constitutional AI and Rule Overrides
- Llama Guard and Input/Output Classification
- Anthropic Responsible Scaling Policy v3.0
- OpenAI Preparedness Framework and DeepMind FSF
- METR Time Horizons and External Evaluation
- CAIS, CAISI, and Societal-Scale Risk
- Why Multi-Agent
- FIPA-ACL Heritage and Speech Acts
- Communication Protocols
- The Multi-Agent Primitive Model
- Supervisor / Orchestrator-Worker Pattern
- Hierarchical Architecture and Decomposition Drift
- Society of Mind and Multi-Agent Debate
- Role Specialization — Planner / Critic / Executor / Verifier
- Parallel Swarm and Networked Architectures
- Group Chat and Speaker Selection
- Handoffs and Routines (Stateless Orchestration)
- A2A — The Agent-to-Agent Protocol
- Shared Memory and Blackboard Patterns
- Consensus and Byzantine Fault Tolerance
- Voting, Self-Consistency, and Debate Topology
- Negotiation and Bargaining
- Generative Agents and Emergent Simulation
- Theory of Mind and Emergent Coordination
- Swarm Optimization (PSO, ACO)
- MARL — MADDPG, QMIX, MAPPO
- Agent Economies, Token Incentives, Reputation
- Production Scaling — Queues, Checkpoints, Durability
- Failure Modes — MAST, Groupthink, Monoculture
- Evaluation and Coordination Benchmarks
- Case Studies and 2026 State of the Art