Compendium

OpenAI product ecosystem — Library index

5 min read

OpenAI product ecosystem — Library index

Canonical reference for everything OpenAI ships through the platform API: the GPT-5 / GPT-4o / o-series reasoning model line, the Responses API (the new unified primary endpoint), the legacy Chat Completions + Assistants APIs, automatic prompt caching, function calling + structured outputs, vision + audio + image generation, the Realtime API (WebSocket + WebRTC voice), the Agents SDK (openai-agents-python + JS), batch + Files + fine-tuning, and the Codex CLI. House style matches the Claude / Gemini sub-libraries; every non-obvious claim cites either platform.openai.com/docs, developers.openai.com, an openai/* GitHub repo, or an OpenAI blog post. Where the official docs are silent but behavior is observable, the entry is labelled “Undocumented as of 2026-05-25 — observed behavior” so a future reader knows to verify.

This sub-library lives inside Math-and-Compute/Compute/ alongside Claude/, GoogleGemini/, and the broader compute-systems notes. The OpenAI ecosystem moves fast — the Responses API is now the primary surface (Chat Completions is “previous standard but still supported”), the Agents SDK has become the official agentic-workflow toolkit, and the GPT-5 generation supplanted GPT-4o in 2025. Read this index for the map, then drill into the specific note.

See also

Library contents

Foundations

FileWhat it covers
openai-models-and-capabilitiesGPT-5 / GPT-5-mini / GPT-5-nano + GPT-4o / GPT-4o-mini + o-series reasoning models (o3 / o3-mini / o4-mini / o1 / o1-pro) + GPT-4 Turbo legacy + embeddings (text-embedding-3-large/small + ada-002) + Whisper + TTS + DALL-E 3 + gpt-image-1; pricing, context windows, knowledge cutoffs
openai-api-and-sdksThe Responses API surface (parameters, response shape, tools, streaming) + Chat Completions API (legacy default) + Python SDK + Node SDK + Azure OpenAI integration + workload-identity auth
prompt-caching-openaiAutomatic prompt caching (no breakpoints), 1024-token threshold, 50 % discount on cache hits, 5-10 minute window, prompt_cache_key for routing

Tool use & structured output

FileWhat it covers
tool-use-and-function-calling-openaiFunction calling spec (JSON Schema, strict mode), parallel_tool_calls, tool_choice semantics, hosted tools (web_search, file_search, computer_use, code_interpreter) on Responses
structured-outputs-openairesponse_format: json_schema strict mode (additionalProperties false, all required, no recursion), Pydantic .parse() SDK helper, Zod integration in JS, schema constraints

Multimodal & realtime

FileWhat it covers
vision-and-multimodal-openaiImage input (image_url / base64, detail: low/high/auto), audio input/output on GPT-4o, DALL-E 3 + gpt-image-1 generation, image edits + variations
realtime-api-openaiRealtime API (WebSocket + WebRTC), gpt-realtime / gpt-realtime-2 model, bidirectional PCM16 audio, voice activity detection, function calling mid-stream, interruption handling, session config

Agents & agentic infrastructure

FileWhat it covers
agents-sdk-openaiopenai-agents-python (and JS twin) — Agent, Runner, handoffs, @function_tool, sandbox agents, sessions, tracing, guardrails / tripwires, computer use, ChatGPT-side Operator
batch-and-fine-tuning-openaiBatch API (50 % discount, 24h SLA, JSONL input/output via Files API) + fine-tuning (SFT on gpt-4o-mini + gpt-3.5-turbo, reinforcement fine-tuning preview, DPO)
hidden-tricks-and-gotchas-openaiUndocumented patterns, tokenizer quirks (tiktoken cl100k_base vs o200k_base), Responses-vs-Chat-Completions migration, parallel-tool-call ordering, structured-output edge cases, Realtime quirks

How to use this library

  1. Picking a model for a new project — start at openai-models-and-capabilities for the matrix, then prompt-caching-openai for the (automatic) cache mechanics, then tool-use-and-function-calling-openai if you need tools.
  2. Choosing Responses vs Chat Completions — openai-api-and-sdks has the decision tree; new code should use Responses for everything except simple stateless completions.
  3. Building agentic systems — agents-sdk-openai first (the official SDK handles handoffs, sessions, tracing, hosted tools); then tool-use-and-function-calling-openai for the underlying mechanism.
  4. Building voice apps — realtime-api-openai — choose WebSocket vs WebRTC based on whether you’re server-side or client-side, audio formats supported, interruption handling.
  5. Cost optimization — prompt-caching-openai (free, automatic) + batch-and-fine-tuning-openai (50 % discount, 24h SLA) + the pricing tables in openai-models-and-capabilities.
  6. RAG / grounded responses — openai-models-and-capabilities embeddings section + hosted file_search tool documented in tool-use-and-function-calling-openai.

Conventions used in this library

  • Model IDs are quoted verbatim (e.g. gpt-5, gpt-5-mini, gpt-4o-2024-11-20, o3-mini, text-embedding-3-large). OpenAI uses date-suffixed pinned snapshots + dateless aliases that resolve to the latest snapshot.
  • Pricing is in USD per million tokens (MTok). Where multiple \$ symbols appear on the same line they are backslash-escaped to avoid Obsidian MathJax interpretation.
  • Code examples use the unified openai Python SDK and @openai-ai/sdk Node SDK — both default to the Responses API in new docs; legacy code uses Chat Completions.
  • “Per platform.openai.com/docs 2026-05” tags a claim sourced from official documentation as of the file’s created date.
  • “Per openai/openai-python README” or openai/openai-agents-python README tags claims sourced from those repos.
  • The developers.openai.com MCP server (openai-docs skill on this machine — see AGENTS.md) is the authoritative source for live OpenAI docs since the public web pages are behind a 403 firewall for unauthenticated fetchers; this library uses GitHub-mirrored READMEs + the developers.openai.com MCP where direct docs aren’t fetchable.

Sources read while building this library (2026-05-25 pass)

  • github.com/openai/openai-python (README + api.md — current SDK surface)
  • github.com/openai/openai-node (README — JS SDK surface)
  • github.com/openai/openai-agents-python (README + quickstart — Agents SDK)
  • github.com/openai/codex (Codex CLI — install + features)
  • github.com/openai/openai-realtime-api-beta (Realtime client + events + audio formats)
  • developers.openai.com (snapshots via MCP and indirect via dev-doc cross-references in cookbook + community)
  • platform.openai.com/docs/models (model catalog — accessed via reference notes since direct fetch is 403)
  • platform.openai.com/docs/api-reference (Responses + Chat Completions schemas)
  • platform.openai.com/docs/guides/prompt-caching (automatic caching mechanics)
  • platform.openai.com/docs/guides/function-calling (tool use spec + strict mode)
  • platform.openai.com/docs/guides/structured-outputs (json_schema mode)
  • platform.openai.com/docs/guides/realtime (WebSocket + WebRTC)
  • platform.openai.com/docs/guides/agents (Agents SDK overview)
  • platform.openai.com/docs/guides/batch (batch API)
  • platform.openai.com/docs/guides/responses (Responses API)

Sections that required inference (rather than direct doc lookup) are flagged in each file under “Undocumented as of 2026-05-25 — observed behavior” so they can be re-verified later. Where the public docs site returned 403, claims were cross-referenced against the openai-python / openai-agents-python / openai-realtime-api-beta repos and OpenAI’s developers.openai.com (accessible via the MCP server on this machine).

10 items under this folder.