Faster AI Agents: What OpenAI’s WebSocket Case Teaches About Latency

OpenAI shows how persistent connections and per-session caching reduced overhead in Codex agentic workflows. An operational read for designing more responsive AI agents.

Model speed alone is no longer enough

OpenAI has published a technical deep dive on how it accelerated Codex agentic workflows using WebSockets and connection-bound caching in the Responses API. For teams designing enterprise AI solutions, the most interesting point is not only the performance figure: it is the architectural shift behind it.

In the cycle described by OpenAI, when Codex has to fix a bug, the agent analyses the code, identifies relevant files, reads context, proposes changes and runs tests. Each step may require repeated interactions between the client, application services, local tools and the model. According to OpenAI, in these flows overall time does not depend only on inference, but also on request validation, state management, network hops and client-side time.

Historically, LLM inference was the slowest part. With faster models, however, the bottleneck can move elsewhere: if the model generates tokens more quickly, the overhead of surrounding services becomes more visible. This matters directly to anyone bringing AI agents into production: user experience depends on the whole chain, not only on the generative engine.

What OpenAI changed in the Codex loop

In its post, OpenAI explains that the previously cited flagship models, such as GPT-5 and GPT-5.2, ran at around 65 tokens per second in the Responses API. For GPT-5.3-Codex-Spark, the goal was to exceed 1,000 tokens per second, also thanks to specialised Cerebras hardware for LLM inference. At that point, reducing repeated application work became necessary to make the speed improvement perceptible.

Before introducing the new pattern, OpenAI had already worked on critical-path optimisations: in-memory caches for rendered tokens and model configuration, fewer network hops to intermediary services, and improvements in the safety stack to flag some issues more quickly. According to the source, these changes led to an improvement close to 45% in time to first token, the metric that indicates how quickly the first output arrives.

The structural issue that remained was handling each request as if it were independent. In long conversations and agentic rollouts, much of the context does not change, but it was still being reprocessed in subsequent steps. OpenAI therefore introduced a persistent WebSocket connection, maintaining an in-memory cache for the lifetime of the connection.

Why WebSockets matter for AI agents

OpenAI says it evaluated several options, including WebSockets and gRPC bidirectional streaming. It chose WebSockets because they enabled a simple message transport without requiring developers to substantially change input and output formats.

The initial prototype modelled an agentic rollout as a single long-running response: the system could suspend the loop when a tool call was required, send an event to the client, wait for the tool result and then continue. The launched version, however, kept a more familiar shape: developers continue to use response.create and previous_response_id to proceed from the previous context.

Technically, the connection keeps elements in memory such as the previous response, already used inputs and outputs, tool definitions, namespaces and reusable sampling artefacts, including already rendered tokens. This allows some validations and classifications to operate only on new input, avoids restarting tokenisation from scratch every time, and enables part of the model-routing logic to be reused.

Reported results and what businesses should take from them

OpenAI reports that, after a sprint of about two months, the WebSocket mode was tested in alpha with startups building coding agents. The source cites improvements of up to 40% in agentic workflows. In production, Codex is said to have moved the majority of Responses API traffic to this mode, with significant latency benefits.

The article also cites external integrations: Vercel reportedly saw latency reductions of up to 40% in the AI SDK, Cline achieved multi-file workflows that were 39% faster, and Cursor saw improvements of up to 30% with OpenAI models. These figures are attributed to the source and should be read in the specific context of those tools, not as an automatic guarantee for every architecture.

For an organisation introducing AI agents, the lesson is concrete: before looking only for a faster model, it is worth measuring where waiting time accumulates. In projects involving application architecture, integrations and AI-enabled workflows, this means designing policies, monitoring and state control from the beginning, not adding them afterwards. This is closely related to API-first and integration work such as Dedaloop’s custom software and integrations services.

Metrics and risks to assess before adopting the same approach

A persistent connection can reduce repeated work, but it also introduces new operational responsibilities. Before replicating a similar model, a team should observe at least a few key metrics:

There are also architectural risks that should not be underestimated. The connection-bound cache must remain consistent with the real state of the conversation. Reconnections must be handled without losing context or repeating actions that have already been executed. In addition, keeping state in memory can make load balancing, autoscaling and failover more complex.

This is where DevOps work becomes central: observability, distributed tracing and reliable infrastructure management become part of the AI project. That is why the topic naturally connects to cloud managed services and DevOps, especially when agents move from prototype to production.

A practical criterion: optimise the cycle, not the single component

The OpenAI case confirms a direction already visible in Codex-based workflows: agents are not simple isolated calls to a model, but iterative processes that use tools, temporary memory, validations and progressive decisions. The same principle applies to business scenarios with recurring tasks, such as those discussed in Dedaloop’s article on Codex Automations.

The useful question is not “how fast is the model?”, but “how efficient is the agent’s full cycle?”. If each step rebuilds context, repeats checks already completed or multiplies network hops, even a very fast model can create a slow experience.

The approach described by OpenAI is not a universal recipe, but it points to a clear priority: to make AI agents reliable and responsive, organisations need incremental architectures that can reuse validated state and computations while preserving control, security and observability.

Approfondimenti Dedaloop

Fonti