Open-weight models & the inference-provider economy — Library index

Canonical reference for the second major LLM deployment path — the one that does not route through a single frontier lab’s first-party API. Two intertwined things: (a) the open-weight model families you can download (Mistral, DeepSeek, Qwen, plus pointers to Llama and the Gemma/Phi/OLMo tail) and (b) the inference-provider economy that serves those weights as an API for you (Together AI, Fireworks, Groq, Cerebras, DeepInfra, Replicate, Baseten, Anyscale, Novita, Hyperbolic) — plus the routing/gateway layer (OpenRouter, LiteLLM) that sits in front of all of it. House style and source conventions match the rest of the Compute library; this sub-library is the open-ecosystem counterpart to the single-vendor sub-libraries ([[Claude/_index]], [[OpenAI/_index]], [[GoogleGemini/_index]], [[AWSBedrock/_index]]).

This sub-library exists because the open-weight serving economy is a first-class deployment path that the rest of the Compute library only mentions in catalogs. The [[Compute/Tier3/llm-landscape]] catalog lists the model families and a one-line-per-provider serving table; [[Compute/model-serving-infrastructure]] covers the self-host runtimes (vLLM, SGLang, TGI). Neither answers the live commercial question every team faces: do I call a frontier API, rent a managed open-weight endpoint, or stand up my own GPUs — and which provider, at what token price, at what tokens/sec, under what data policy? That build-vs-buy / self-host-vs-API decision is what this sub-library exists to inform.

It lives inside Math-and-Compute/Compute/ as a sibling of Claude/, OpenAI/, GoogleGemini/, AWSBedrock/, Vercel/, CloudflareWorkers/, and the new MetaLlama/. The open-weight half is deliberately concise — the model-family depth lives in [[Compute/Tier3/llm-landscape]], and Meta’s Llama gets its own sibling sub-library [[Compute/MetaLlama/_index]] rather than being duplicated here. The provider-economy half is the real meat: speed, price, model selection, cold-start, quantization defaults, SLAs, and data policy are exactly the facts that move fastest and are hardest to pin down from a catalog.

Why this matters — the decision this library informs

                       ┌─────────────────────────────────────────────┐
   "I need an LLM       │  1. Frontier first-party API                │  highest quality, simplest,
    in production"  ──► │     (Claude / OpenAI / Gemini)              │  no infra; pay frontier rates
                        ├─────────────────────────────────────────────┤
                        │  2. Hyperscaler model garden                │  one bill, IAM/VPC-native;
                        │     (Bedrock / Azure Foundry / Vertex)      │  small markup, release lag
                        ├─────────────────────────────────────────────┤
                        │  3. Managed open-weight provider            │  open models as an API;
                        │     (Together / Fireworks / Groq / …)       │  cheap, fast, no GPUs to run
                        ├─────────────────────────────────────────────┤
                        │  4. Self-host open weights                  │  full control + data custody;
                        │     (vLLM / SGLang / TGI on your GPUs)      │  you own the ops + utilization risk
                        └─────────────────────────────────────────────┘
                              ▲ a routing/gateway layer (OpenRouter, LiteLLM)
                                can sit in front of 1–3 and fail over between them

Rows 3 and 4 are this library’s subject. Row 1’s depth lives in the vendor sub-libraries; row 2 lives in [[AWSBedrock/_index]]; row 4’s runtime mechanics live in [[Compute/model-serving-infrastructure]]. This library is the connective tissue that lets you reason across all four rows on cost and latency.

See also

• _index — the parent Compute reference hub
• _index — Meta Llama’s own sub-library (referenced here, not duplicated)
• model-serving-infrastructure — the self-host runtimes (vLLM / SGLang / TGI / TensorRT-LLM) behind row 4 and behind every provider in row 3
• llm-landscape — the full model-family catalog this library’s open-weight half is reconciled against (refreshed 2026-05-30)
• inference-optimization — quantization (GPTQ / AWQ / GGUF / FP8), speculative decoding, KV-cache math that determines a provider’s tokens/sec and price
• _index — the hyperscaler model-garden path (row 2)
• _index — the closed-frontier first-party path (row 1), for contrast

Library contents

How to use this library

1. Picking where to run an open model — read the decision framework in part (B) of open-weight-models-and-inference-providers, then the provider comparison table, then cross-check the runtime details in model-serving-infrastructure if you land on self-host.
2. Choosing an open-weight family — start with the family summaries in part (A) here, then go deep in llm-landscape; for Llama specifically use _index.
3. Understanding licensing before you ship — the licensing taxonomy in part (A) tells you which models are safe for commercial / closed-source / on-prem deployment and which carry user-count or research-only restrictions.
4. Optimizing cost at scale — the total-cost reasoning in part (B) plus the per-token / per-tokens-sec tables, then the gateway + cost-fallback pattern (OpenRouter / LiteLLM).
5. Going self-host — this library frames when to; model-serving-infrastructure and inference-optimization are how to.

Conventions used in this library

• All pricing and tokens/sec figures are time-stamped to 2026-05-30 and labelled as such. This is the single fastest-moving corner of the AI stack — per-token prices move weekly and provider throughput leaderboards reshuffle monthly. Treat every number as a dated claim and verify against the provider’s live docs, artificialanalysis.ai, or lmarena.ai before acting on it.
• Pricing is USD per million tokens (MTok); where multiple \$ symbols appear on one line they are backslash-escaped to avoid Obsidian MathJax interpretation.
• Model facts (versions, parameter counts, licenses) are reconciled against [[Compute/Tier3/llm-landscape]] (2026-05-30) — Mistral 3 / Large 3 (Dec 2025), DeepSeek V3.1/V3.2 with DSA sparse attention, Qwen 3 as the current open flagship. Where this library and the catalog could drift, the catalog is canonical for the lineup and this library is canonical for the serving economics.
• Equations and decision trees are in fenced code blocks (not $-delimited) to stay MathJax-safe.
• Named-citation style: “per Artificial Analysis 2026-05”, “per OpenRouter docs 2026-05”, “per DeepSeek-V3.2 paper (arXiv 2512.02556)”, etc.

Sources read while building this library (2026-05-30 pass)

• openrouter.ai/docs — provider routing, model fallbacks, ZDR flag, quantization filtering, FAQ
• docs.litellm.ai — self-hosted proxy: routing, fallbacks, virtual keys, cost tracking, local-model unification
• artificialanalysis.ai — independent quality + speed + price benchmarking (tokens/sec, TTFT, $/MTok)
• console.groq.com/docs + inference-docs.cerebras.ai — LPU / wafer-scale throughput claims
• docs.together.ai, docs.fireworks.ai, deepinfra.com, replicate.com/docs, baseten.co, novita.ai, hyperbolic.xyz — provider model catalogs, pricing, data policies, SLAs
• huggingface.co/Qwen + qwenlm.github.io/blog/qwen3 — Qwen3 lineup + Apache-2.0 license
• github.com/deepseek-ai/DeepSeek-V3.2-Exp + arXiv 2512.02556 (DeepSeek-V3.2) + blog.vllm.ai/2025/09/29/deepseek-v3-2.html (DSA in vLLM)
• docs.mistral.ai + Mistral Large 3 / Ministral 3 release notes (Dec 2025)
• [[Compute/Tier3/llm-landscape]] (2026-05-30) — the in-repo catalog this library is reconciled against

Sections that required inference rather than a direct doc lookup (exact provider quantization defaults, undocumented cold-start behavior) are flagged “verify — fast-moving” inline so a future reader knows to re-check.