Individuals find the right products. Businesses reach the right audience. One platform, free for both.
DEVUP AI introduced ai-api-audit, an MIT licensed Python tool that runs protocol checks to verify how OpenAI compatible LLM gateways behave.
DEVUP AI, an OpenAI and Anthropic compatible inference gateway and GPU cloud in Algiers, has released ai-api-audit, a single file Python tool for auditing OpenAI compatible LLM gateway behavior.
DEVUP AI says pointing an OpenAI SDK to a third party base_url is easy, but verifying what happens behind that endpoint is harder.
In an article published on November 16 by DEVUP AI founder Mohamed Bal, the company introduced ai-api-audit as a way for engineers to run objective, reproducible checks from their own terminal instead of relying on provider marketing.
The post lists several issues an application cannot easily detect from a standard HTTP 200 OK response to POST /v1/chat/completions, including a cheaper model answering under a more expensive model name, hidden wrapper instructions being injected into prompts, padded completion token counts, silent truncation of long context inputs, and missing usage accounting in streaming.
The article explains one approach used by the tool, differential tokenizer fingerprinting. It compares the change in prompt_tokens between two requests to cancel out unknown chat template overhead, then checks whether the resulting token count matches an official tokenizer for the claimed model family.
The post also says that in tests against DEVUP AI’s API, the gateway passed protocol checks and prompt token counts matched in tokenizer verified runs.
For teams using inference gateways, token counts and model identity affect both cost and reliability. A response that looks correct can still hide changes that increase billing, reduce context length, or remove cost visibility.
DEVUP AI is positioning ai-api-audit as a practical way to spot these problems early, using protocol level checks that can be repeated across providers.
Primary Source: DEV Community
Chief Content Officer (Too Long; Didn't Resign)
TL;DR Tara is Liners' AI-assisted editorial agent for African technology news, product explainers, and comparison content. Tara helps turn multiple source materials and signals into clear summaries, while Liners remains responsible for editorial standards, sourcing, and corrections.