LLM-OTA · technical plan

Turn a Grok subscription into an OpenAI- and Anthropic-compatible API

A Cloudflare Worker on llm.e2e.one that accepts standard LLM SDK traffic, authenticates with issued gateway keys, and forwards to xAI using credentials extracted from the official grok-cli OAuth session. An admin UI shows config, token health, and usage including cache reads and writes.

POC provider: Grok / xAI only Host: llm.e2e.one Zone: e2e.one Worker: llm-ota Subscription OAuth · ToS-sensitive

1. Goal

LLM-OTA (Over-The-Air LLM gateway) is a single Cloudflare Worker that looks like OpenAI and Anthropic to clients, and like a Grok Build / SuperGrok subscriber to xAI. The POC implements one provider only: Grok. Later providers (Claude Max, ChatGPT Plus, Gemini, etc.) plug into the same gateway interface.

2. Research findings

2.1 grok-cli does not use grok.com cookies

The official Grok Build CLI (this machine: ~/.grok/bin/grok) authenticates through SpaceXAI OAuth at auth.x.ai, not the unofficial sso / sso-rw cookie scrape used by older grok.com wrappers. Credentials live in ~/.grok/auth.json with mode 0600.

Observed local schema (values redacted):

{
  "https://auth.x.ai::<principal-uuid>": {
    "auth_mode": "...",
    "key": "<access token — Bearer for api.x.ai>",
    "refresh_token": "<rotating refresh token>",
    "expires_at": "<ISO-8601>",
    "oidc_issuer": "https://auth.x.ai",
    "oidc_client_id": "...",
    "user_id": "...",
    "team_id": "...",
    "email": "...",
    "principal_id": "...",
    "principal_type": "..."
  }
}

• key is the access token the CLI sends as Authorization: Bearer to xAI.
• refresh_token is used for silent renewal. Third-party clients (Kilo Code, pi-xai-oauth) document that xAI rotates refresh tokens on every use. Only one refresher may run at a time.
• Session tokens take precedence over XAI_API_KEY. The API key is a pay-per-token console key, not the SuperGrok subscription path.
• Official docs: Grok Build, CLI auth guide under ~/.grok/docs/user-guide/02-authentication.md.
• pi-xai-oauth already reuses ~/.grok/auth.json without rewriting it — same ingest idea.

2.2 xAI already speaks OpenAI; Anthropic native is deprecated

• Primary: https://api.x.ai/v1/chat/completions and /v1/responses.
• OpenAI SDK works with base_url=https://api.x.ai/v1.
• POST /v1/messages still exists and returns Anthropic-shaped cache_creation_input_tokens / cache_read_input_tokens, but xAI marks Anthropic SDK compatibility as fully deprecated and cache_creation_input_tokens as unsupported.
• POC strategy: implement our own Anthropic facade. Translate to xAI Chat Completions or Responses, then map usage back into Anthropic and OpenAI cache fields. Do not rely on xAI's dying /v1/messages as the long-term backend.

2.3 Prompt cache on xAI is automatic prefix cache

From xAI prompt caching:

• Identical leading messages are cached automatically. No cache_control on the wire to xAI.
• Sticky routing via x-grok-conv-id (Chat Completions header) or prompt_cache_key (Responses body) is required for a high hit rate. Cache is per-server.
• OpenAI-shaped usage: usage.prompt_tokens_details.cached_tokens or usage.input_tokens_details.cached_tokens.
• xAI does not publish a first-class cache_write_tokens field. We synthesize writes (see §8).

2.4 What “full” OpenAI / Anthropic cache compat means

Client	Request	Response
OpenAI Chat Completions	Stable prefix; optional prompt_cache_key; pass-through x-grok-conv-id	usage.prompt_tokens_details.cached_tokens, cache_write_tokens (GPT-5.6-style, synthesized)
OpenAI Responses	prompt_cache_key	usage.input_tokens_details.cached_tokens + write counterpart
Anthropic Messages	Block-level cache_control: {type:"ephemeral", ttl:"5m"|"1h"}; top-level automatic cache_control; anthropic-beta: prompt-caching-2024-07-31	cache_creation_input_tokens, cache_read_input_tokens, optional cache_creation.ephemeral_5m_input_tokens / ephemeral_1h_input_tokens

3. Key decisions

Decision	Choice	Why
Runtime	One Cloudflare Worker + D1 + KV + one Durable Object	Already on the e2e.one zone. Custom Domain gives TLS without extra DNS work.
Public hostname	llm.e2e.one	ai.e2e.one is taken by kiro-otai. api.e2e.one points at an origin A record.
POC provider	Grok via grok-cli OAuth against api.x.ai	User request. Official console API keys remain a fallback auth mode.
Not using grok.com cookies	OAuth session only	CLI no longer uses sso cookies; cookie scrapers are brittle and more ToS-hostile.
Anthropic backend	Translate to xAI Chat Completions / Responses	xAI native /v1/messages is deprecated; cache writes are unsupported there.
Refresh lock	Durable Object TokenLease	Refresh tokens rotate. Concurrent refreshes invalidate each other.
Client auth	Gateway-issued keys, hashed in D1	Never expose the grok-cli access token to downstream tools.
Admin auth (POC)	ADMIN_PASSWORD cookie session	Fast. Swap for Cloudflare Access before any shared use.

4. Architecture

  OpenAI SDK / Claude SDK / Cursor / Cline
                 |
                 |  sk-ota-...  or  x-api-key
                 v
        https://llm.e2e.one          Cloudflare Worker  llm-ota
        /v1/chat/completions         - protocol detect
        /v1/responses                - key auth
        /v1/messages                 - model alias
        /v1/models                   - cache-key derivation
        /admin                       - usage log (waitUntil → D1)
                 |
                 |  Bearer <rotated grok-cli access token>
                 |  x-grok-conv-id / prompt_cache_key
                 v
              api.x.ai
                 |
        TokenLease DO <-- refresh_token grant --> auth.x.ai
        KV: current access token + expiry (cache of DO state)
        D1: api_keys, requests, usage_daily, config

Worker routes

Method	Path	Auth	Role
GET	/	public	Status / this plan
GET	/health	public	liveness; no token leak
GET	/v1/models	gateway key	OpenAI model list
POST	/v1/chat/completions	gateway key	OpenAI Chat Completions
POST	/v1/responses	gateway key	OpenAI Responses
POST	/v1/messages	x-api-key or Bearer	Anthropic Messages
POST	/v1/messages/count_tokens	same	Anthropic token count (best-effort)
*	/admin*	admin session	Config + usage UI
POST	/admin/api/keys	admin	Mint / revoke gateway keys
POST	/admin/api/provider/grok/import	admin	Accept pasted auth.json blob (never logged)

5. Extracting grok-cli tokens

5.1 Local importer (preferred)

A repo script scripts/import-grok-auth.mjs reads ~/.grok/auth.json (or $GROK_HOME/auth.json), never prints secrets, and writes Wrangler secrets:

GROK_ACCESS_TOKEN
GROK_REFRESH_TOKEN
GROK_OIDC_ISSUER          # https://auth.x.ai
GROK_OIDC_CLIENT_ID
GROK_TOKEN_EXPIRES_AT     # ISO, optional
GROK_ACCOUNT_EMAIL        # non-secret, also a var

Refresh flow (inside the Durable Object, exclusive):

1. Discover token_endpoint from {issuer}/.well-known/openid-configuration.
2. POST grant_type=refresh_token, client_id, current refresh token. Public client, no secret (CLI is a PKCE public client).
3. Persist the new access token, new expiry, and rotated refresh token via wrangler secret is not available at runtime — store the live pair in DO storage (authoritative) and a short-TTL copy in KV for hot reads.
4. Seed DO storage from the Wrangler secrets only on first boot or when the admin hits “re-import”.

5.2 Admin paste import

The admin UI accepts the raw auth.json object (or the inner record). The Worker extracts key / refresh_token / issuer / client_id, seeds the DO, and returns only { email, expires_at, issuer }. The request body is not written to D1 or logs.

5.3 Fallback: console API key

If no session is loaded, XAI_API_KEY (secret) is used. Usage then bills the xAI console, not the SuperGrok quota. The admin UI labels the active mode: subscription-oauth vs console-api-key.

6. OpenAI-compatible surface

Accept the current OpenAI Chat Completions and Responses request bodies and stream shapes.

Must implement (POC)

• Auth: Authorization: Bearer sk-ota-...
• Fields: model, messages / input, stream, temperature, top_p, max_tokens / max_completion_tokens, stop, tools, tool_choice, response_format, user, seed, n=1
• Streaming: SSE data: {...} then data: [DONE]; stream_options.include_usage
• Tool calls: pass through OpenAI function-tool schema; map xAI tool calls 1:1
• Errors: OpenAI { error: { message, type, code, param } } with matching HTTP status
• Models: alias table, e.g. gpt-4o → grok-4.6 (configurable)

Explicitly later

• Embeddings, images, audio, realtime, batches, assistants, files
• n > 1

7. Anthropic-compatible surface

Clients such as Claude Code, Cline, and the official Anthropic SDK must work unchanged.

Headers

Header	POC behavior
x-api-key	Gateway key. Also accept Bearer.
anthropic-version	Accept 2023-06-01; echo it.
anthropic-beta	Honor prompt-caching-2024-07-31 and extended TTL names. Ignore unknown betas.

Request

• model, required max_tokens, messages, system (string or text-block array), tools, tool_choice, temperature, top_p, stop_sequences, stream, metadata.user_id, thinking (map to xAI reasoning if the mapped model supports it)
• Content blocks: text, image (base64 / url), tool_use, tool_result. Drop or 400 on unimplemented block types with a clear message.
• cache_control on system blocks, message blocks, and the last tool (see §8).
• Top-level cache_control (Anthropic automatic caching, 2026 docs): treat as “cache the whole prefix through the last cacheable block.”

Response

• Non-stream: type=message, role=assistant, content blocks, stop_reason in {end_turn, max_tokens, stop_sequence, tool_use}
• Stream: message_start, content_block_start, content_block_delta, content_block_stop, message_delta, message_stop
• Usage object always includes cache fields, even when zero

Translation to xAI

Anthropic messages + system + tools
        → OpenAI/xAI chat messages
        → POST api.x.ai/v1/chat/completions
        → map choices/tool_calls back to content blocks
        → map usage (see §8)

8. Cache reads and writes

Two different cache languages have to meet in the Worker. xAI only reports reads (cached_tokens). Anthropic and newer OpenAI also expect writes.

8.1 Driving xAI cache

1. Keep the prompt prefix byte-stable. Never rewrite earlier messages.
2. Derive a conversation id: sha256(gateway_key_id + ":" + (client prompt_cache_key || client x-grok-conv-id || hash(stable_prefix)))
3. Send it as x-grok-conv-id on Chat Completions and as prompt_cache_key on Responses.
4. For Anthropic explicit breakpoints: the cached prefix is everything up to and including the last block that carries cache_control. The suffix after that is the “hot” tail. We still send the full prompt to xAI (it only caches matching prefixes), but we use the breakpoint to decide how to report writes vs reads.

8.2 Mapping usage back

Let P = upstream prompt tokens, R = upstream cached_tokens.

Outbound field	Value
OpenAI prompt_tokens / Responses input_tokens	P
OpenAI prompt_tokens_details.cached_tokens	R
OpenAI prompt_tokens_details.cache_write_tokens	max(P - R - tail, 0) on a miss-or-partial, else 0
Anthropic input_tokens	P - R (Anthropic counts uncached input only)
Anthropic cache_read_input_tokens	R
Anthropic cache_creation_input_tokens	same write estimate as OpenAI
Anthropic cache_creation.ephemeral_5m_input_tokens	writes whose breakpoint ttl is 5m / default
Anthropic cache_creation.ephemeral_1h_input_tokens	writes whose breakpoint ttl is 1h

tail is an estimate of tokens after the last cache_control marker (tiktoken-class count is fine for POC; exact xAI tokenization is not public). If the client sent no markers (pure OpenAI automatic cache), treat the entire prompt except the last user turn as the cacheable prefix.

8.3 Streaming usage

• OpenAI: last chunk carries usage when include_usage is set.
• Anthropic: message_delta.usage plus final message_start.usage zeros, matching current Claude behavior.

9. Admin frontend

Served by the same Worker (no separate Pages project). Small server-rendered HTML plus vanilla JS. No client-side framework in the POC.

10. Data model

D1

api_keys(
  id TEXT PK, label TEXT, hash TEXT UNIQUE,
  prefix TEXT, created_at, last_used_at, revoked_at,
  rpm INTEGER, daily_token_cap INTEGER
)
requests(
  id TEXT PK, ts, key_id, protocol, model, status, latency_ms,
  prompt_tokens, completion_tokens,
  cache_read_tokens, cache_write_tokens, error_code
)
usage_daily(
  day TEXT, key_id, model,
  requests, prompt_tokens, completion_tokens,
  cache_read_tokens, cache_write_tokens,
  PRIMARY KEY(day, key_id, model)
)
config(
  key TEXT PK, value TEXT, updated_at
)

KV

grok:access short-TTL copy of the live access token for request-path reads. Source of truth is the DO.

Durable Object TokenLease

Single instance. Methods: getAccessToken(), importSession(blob), status(). Serializes refresh.

Secrets / vars

ADMIN_PASSWORD          secret
GROK_*                  secrets (seed only)
XAI_API_KEY             secret, optional fallback
ALLOWED_ORIGINS         var, optional CORS

11. Security and ToS

• Never log access tokens, refresh tokens, Authorization headers, or prompt bodies.
• Gateway keys stored as SHA-256 (or better, salted). Show the secret once at mint time.
• Admin: timing-safe password compare; HttpOnly Secure cookie; CSRF on mutating admin POST.
• CORS: default deny. Allowlist only if a browser client is required.
• After Cloudflare Access is added, bind Access JWT checks on /admin*.
• Streaming: pipe upstream SSE; do not buffer the full completion in Worker memory.

12. Domain setup on e2e.one

Zone e2e.one is already active on this Cloudflare account (free plan, id e001b235077663f32cf15cca58c0aeec).

Hostname	Status	Action
ai.e2e.one	AAAA 100:: + route → kiro-otai	Leave alone
api.e2e.one	A 84.8.145.139 (existing origin)	Do not steal
llm.e2e.one	unused	Worker Custom Domain

Wrangler:

{
  "name": "llm-ota",
  "main": "src/index.ts",
  "compatibility_date": "2026-08-13",
  "compatibility_flags": ["nodejs_compat"],
  "observability": { "enabled": true, "head_sampling_rate": 1 },
  "routes": [{ "pattern": "llm.e2e.one", "custom_domain": true }]
}

Custom Domain creates the DNS record and certificate. Do not also create a CNAME — that blocks Custom Domain attach.

Public URLs after attach:

• https://llm.e2e.one/ — plan / status
• https://llm.e2e.one/v1/chat/completions
• https://llm.e2e.one/v1/messages
• https://llm.e2e.one/admin

13. Build order

#	Slice	Ships
0	This plan + hostname	PLAN.html, Worker stub, llm.e2e.one Custom Domain
1	Scaffold	wrangler.jsonc, D1 schema, KV, TokenLease DO, health, admin password gate
2	Grok adapter	Importer, DO refresh, fallback API key, GET /v1/models
3	OpenAI Chat Completions	Non-stream + stream, tools, usage + cached_tokens + cache_write_tokens
4	Anthropic Messages	Translate, stream events, cache_control in and cache_* usage out
5	Responses API	Thin map onto the same adapter; prompt_cache_key
6	Admin UI	Config, key mint, usage charts, cache rates
7	Harden	Cloudflare Access on /admin, rate limits, request log retention

14. Verification (when implementing)

1. curl -sS https://llm.e2e.one/health → 200.
2. Import local auth (no token printed). Admin shows email + future expiry.
3. OpenAI:

   curl https://llm.e2e.one/v1/chat/completions \
     -H "Authorization: Bearer $OTA_KEY" \
     -H "Content-Type: application/json" \
     -d '{"model":"grok-4.6","messages":[{"role":"user","content":"ping"}]}'

   Second identical call with the same x-grok-conv-id must show cached_tokens > 0 or a non-zero write then a read.
4. Anthropic SDK pointed at https://llm.e2e.one with a cache_control system block. Response includes cache_creation_input_tokens then cache_read_input_tokens on the follow-up.
5. Admin usage page increments for both protocols; prompt text is absent from D1.
6. Force-expire the access token: next request still succeeds via DO refresh; a parallel burst does not lose the refresh token.

15. Open questions

• Keep llm.e2e.one or also alias ota.e2e.one?
• Should model aliases pretend to be gpt-4o / claude-sonnet-4-6 so existing client configs work, or only advertise real Grok ids?
• Put Cloudflare Access on /admin in the POC or after first working completion?
• After import, is the Worker the sole owner of the refresh token (recommended), or do we accept fighting the local CLI?