Indirection & caching

Flowgate trades a large static tool list for two fixed tools plus discovery. That trade has a real cost worth naming precisely — and a few reasons the cost stays small in practice. This page is the honest accounting.

Two costs, kept separate

Every tool-using setup pays two different token costs, and they behave very differently:

• Static description cost — tokens spent describing the available tools in the prompt, on every call.
• Discovery cost — round trips the model spends finding and invoking the right capability.

Flowgate moves cost from the first bucket to the second. Whether that’s a good trade comes down to looking at each honestly.

Static cost: caching makes it nearly free — for everyone

The two tool schemas (flowgate.query, flowgate.command) are a stable prompt prefix. They don’t change between calls, so prompt caching on the host serves them from cache after the first request. The marginal per-call cost is close to zero.

Be honest about the comparison: a flat tool list is also a stable prefix, so it caches too. Caching alone doesn’t favor Flowgate. The durable differences are:

• The Flowgate prefix is constant. Two tools whether you wire in five capabilities or five hundred. A flat list grows with every tool you add — and any cache miss (new session, eviction, a host that doesn’t cache) pays the full, growing price.
• Selection reasoning is never cached. A model staring at 50 tool definitions has to reason about which to pick — fresh, every call, uncached. Two tools removes that menu. This is the cost caching can never give back, and it’s the one that grows fastest with tool count.

So caching neutralizes the static cost for both designs. Flowgate’s edge is a prefix that doesn’t grow and a menu the model doesn’t have to re-read.

Discovery cost: the indirection tax

This is the cost Flowgate adds. A direct tool call is one model round trip — the tool is in the list, the model calls it. With two tools, the model first has to find the capability:

Direct tool list        model → tool call → result                   1 round trip

Flowgate, cold path     model → flowgate.query  { query }    (find)
                        model → flowgate.command { ... }     (invoke)  2 round trips

Flowgate, warm (cached) model → flowgate.command { ... }     (invoke)  1 round trip

That extra hop is the indirection tax. Three things keep it small:

• It’s a model round-trip cost, not a gateway cost. The gateway’s search is microseconds. What you pay is inference latency and output tokens for the extra step.
• It bites single capabilities, not workflows. A multi-step workflow was always going to be several round trips, and deterministic chaining already collapses the computable ones. The tax only surfaces on a one-shot capability call.
• It’s a cold-path cost. You pay it the first time the model reaches for something — not every time it uses it.

Context is the cache

The reason the tax stays small: the model’s context is the cache, and it fills lazily.

When the model searches and gets back a prefilled command link, that link is now in the conversation. The next time it wants the same capability this session, it doesn’t search again — it reuses the link it already has. Discovery is paid once per capability per session; reuse after that is free. No prefetching, no server-side cache, no working-set to maintain. The context window already does it.

This is deliberately lazy, not eager. Flowgate never pushes a list of “capabilities you might want” into the prompt — that would just rebuild the flat tool list the two-tool design exists to remove. Capabilities enter context only when the model pulls them in by searching.

Why a cached link is still safe: discovery is contextual

There’s a subtlety worth stating plainly, because it’s exactly where the cache meets the “see only what you need” thesis.

Search results and response links are contextual — the gateway returns the capabilities and moves that are valid in the current state, filtered by guards. A link the model cached three turns ago is therefore a hint, not a guarantee. State may have moved; a guard may now reject it.

That’s fine, by design:

• The gateway is always the authority. Every command is re-validated at execution time against current state and guards. A cached link can’t bypass anything — if it’s no longer legal, the model gets INVALID_TRANSITION or GUARD_REJECTED back, with fresh legal links to recover from.
• A stale cache means re-discovery, which is correctness, not waste. If context has moved far enough that an old link no longer applies, the model searches again — and that re-search is the contextual filtering doing its job. You want the model to see what’s valid now, not what was valid then.

So the cache never fights the contextual model. It accelerates the common case (link still legal → direct call) and falls back to the thing Flowgate would do anyway (re-search) when context has changed. The server stays the single source of truth either way.

The honest floor

You cannot drive this to literal zero, and the reason is the design working as intended:

The first use of a capability in a fresh session costs the search round trip. That’s the floor.

The only way under it is to prefetch capabilities into the prompt before the model asks — which reintroduces the growing, always-present tool list the two-tool design removed. Flowgate doesn’t, on purpose. The honest claim:

Scenario	Cost
Repeat use within a session	Free — cached link, re-validated at command time
First use in a session	One search, then one command — two round trips
Use after context has moved	One re-search — correct behavior, not overhead

That floor is low, it’s bounded, and it buys you a tool surface that stays constant at any scale and a gateway that’s always the authority on what’s legal right now.