Rohit Marathe

Multi-Agent Orchestration — Patterns That Actually Work

After spending the past year building multi-agent systems at Rocket Mortgage, I've learned that the gap between a demo and production is enormous. Here are the patterns that actually survived contact with real traffic.

The Problem

A single LLM prompt can answer simple questions. But when you need an AI system to handle mortgage calculations, search property databases, analyze banker performance, and diagnose system alerts — you need specialization.

That's where multi-agent orchestration comes in: instead of one monolithic prompt, you build a system of specialized agents that collaborate.

Pattern 1: Router-Based Orchestration

The simplest pattern that works. A central router agent classifies incoming requests and delegates to the appropriate specialist.

class OrchestratorRouter:
    def __init__(self, agents: dict[str, Agent]):
        self.agents = agents
        self.classifier = ClassifierAgent()

    async def handle(self, request: UserRequest) -> Response:
        intent = await self.classifier.classify(request)
        agent = self.agents[intent]
        return await agent.execute(request)

When to use it: When your agents have clearly distinct domains and requests rarely need multiple agents.

The catch: Classification accuracy is everything. A misrouted request gives the user a completely wrong answer with full confidence. We found that adding a confidence threshold and falling back to a general agent for low-confidence classifications cut our error rate by 20%.

Pattern 2: Multi-LLM Failover

Production systems can't go down. We built a failover architecture that routes between Claude and GPT-4 based on availability and rate limits.

class MultiLLMClient:
    def __init__(self, providers: list[LLMProvider]):
        self.providers = providers  # ordered by preference

    async def complete(self, prompt: str) -> str:
        for provider in self.providers:
            if provider.is_available():
                try:
                    return await provider.complete(prompt)
                except RateLimitError:
                    provider.mark_limited()
                    continue
        raise AllProvidersExhaustedError()

The key insight: rate-limiting guardrails must be proactive, not reactive. By the time you get a 429 response, you've already wasted latency. We track token consumption in real-time and start routing to the backup provider before hitting the limit.

Pattern 3: Self-Healing Agents

This is the one I'm most proud of. We built an agent that monitors system alerts, diagnoses root causes using RAG over our runbooks, and executes remediation scripts autonomously.

The architecture:

  1. Alert ingestion — System alert comes in via webhook
  2. Context retrieval — RAG searches our knowledge base of past incidents and runbooks
  3. Diagnosis — LLM analyzes the alert + retrieved context to identify root cause
  4. Action planning — Agent proposes remediation steps using function calling
  5. Execution — Approved actions are executed via MCP server integration

The critical guardrail: never auto-execute destructive actions. Our agent can restart services and clear caches autonomously, but anything involving data modification requires human approval.

What I'd Do Differently

If I were starting over, I'd invest more in observability from day one. We eventually built an LLM-as-a-Judge pipeline with Langfuse that continuously evaluates agent responses, but we should have had that from the start. When you have multiple agents in a pipeline, debugging "why did the system give a wrong answer?" becomes a multi-step investigation. Good tracing makes this 10x easier.

Key Takeaways

  1. Start simple — Router-based orchestration handles 80% of use cases
  2. Build for failure — Multi-LLM failover is not optional in production
  3. Observe everything — You can't improve what you can't measure
  4. Guardrails are features — The best agent is one that knows when not to act

I'll dive deeper into each of these patterns in future posts. If you're building something similar, I'd love to hear about your approach — reach out on LinkedIn.