zenmemes

New post

Multi Provider LLM Config Step 1 Solution Assessment

Problem Statement

Instance operators need to choose a reliable LLM provider for post analysis and reply generation instead of being locked to Dedalus.

Option A: Treat every provider as OpenAI-compatible

Pros:

  • Smallest change because current requests already use chat-completions style payloads.

  • Covers OpenAI, OpenRouter, Dedalus, LiteLLM, and many self-hosted gateways with one adapter.

  • Keeps provider switching mostly to base URL, API key, model, and optional headers.

Cons:

  • Anthropic native API is not OpenAI-compatible, so it would require OpenRouter/LiteLLM or be excluded.

  • OpenAI-compatible does not mean identical structured-output behavior across providers.

  • Provider-specific diagnostics and limits can be harder to express cleanly.

Option B: Add direct native adapters for each provider

Pros:

  • Best fit for provider-specific APIs, especially Anthropic.

  • Cleaner errors, request IDs, model naming, and structured-output behavior per provider.

  • Avoids relying on third-party routers for operators who already have direct accounts.

Cons:

  • More code paths to build and test before operators get broad provider choice.

  • Higher maintenance as provider APIs evolve.

  • Risks duplicating prompt, schema, retry, and diagnostic logic across adapters.

Option C: Add a shared LLM provider layer with OpenAI-compatible and native adapters

Pros:

  • Keeps common analysis/reply behavior shared while letting transport details vary.

  • Supports OpenAI-compatible providers first: OpenAI, OpenRouter, Dedalus, LiteLLM, and local gateways.

  • Allows a native Anthropic adapter where direct Anthropic use is materially different.

  • Leaves room for later native Gemini or other adapters without changing application workflows.

Cons:

  • Slightly larger first cut than a single OpenAI-compatible adapter.

  • Requires careful provider capability flags for structured outputs and fallback JSON parsing.

  • Needs config migration from DEDALUS_* names to provider-neutral names.

Option D: Recommend an external gateway such as OpenRouter or LiteLLM only

Pros:

  • Minimal app code if operators point the existing base URL/model settings at a compatible gateway.

  • Outsources provider routing, failover, and model catalogs.

  • Good operational fit for teams that already use a gateway.

Cons:

  • Does not satisfy direct Anthropic or direct OpenAI configuration as first-class options.

  • Adds another dependency and billing/control surface.

  • Leaves the app semantics named after Dedalus, which is confusing for production operators.

Recommendation

Recommend Option C.

Brief justification:

  • The app should have provider-neutral configuration and one shared analysis/reply contract, not Dedalus-specific wiring.

  • OpenAI-compatible support gives broad coverage quickly: OpenAI, OpenRouter, Dedalus, LiteLLM, vLLM/llama.cpp/Ollama-style local gateways, and other compatible routers.

  • Anthropic deserves a native adapter because direct Claude API support is a common operator need and differs enough from OpenAI-compatible chat completions.

  • Keep Gemini/native Google as a likely later adapter unless Step 2 makes it a hard requirement.

References:

  • OpenAI structured outputs: https://developers.openai.com/api/docs/guides/structured-outputs

  • Anthropic structured outputs: https://platform.claude.com/docs/en/build-with-claude/structured-outputs

  • OpenRouter quickstart and structured outputs: https://openrouter.ai/docs/quickstart, https://openrouter.ai/docs/guides/features/structured-outputs

  • LiteLLM gateway: https://docs.litellm.ai/docs/