Use Jnvoy with LiteLLM

How it fits

Jnvoy speaks the same chat format as OpenAI, so LiteLLM can use it as an OpenAI-compatible server. LiteLLM sends the request to Jnvoy, Jnvoy replaces personal data with placeholders such as [EMAIL_ADDRESS_1], forwards the cleaned request to your model provider, and puts the real values back into the reply before LiteLLM sees it.

Four settings change. The model name gets the prefix openai/. api_base is https://www.jnvoy.com/v1. api_key is your Jnvoy key. Your own OpenAI or Anthropic key goes in an x-provider-key header. You still pay the provider directly; Jnvoy does not supply model access.

LiteLLM's own PII guardrail

LiteLLM can already mask personal data without Jnvoy. Its Presidio guardrail uses Microsoft Presidio to replace names, emails and similar values in the request and can put the original values back into the reply. Presidio runs as two extra containers (an analyzer and an anonymizer) that you operate alongside LiteLLM. If you already run that setup and it covers your cases, you may not need Jnvoy.

Jnvoy is a separate service you point LiteLLM at, so there is nothing extra to run for the redaction itself. It also keeps an audit log that records counts and never the text, with CSV and NDJSON exports, and can refuse requests that contain well-known injection phrases if you turn that on (it is off by default; see what it catches). The trade-off is the one in the notice above: with Jnvoy placed behind LiteLLM, LiteLLM still sees the original text. We have not tested LiteLLM’s guardrail as it ships, but we have compared Jnvoy with Presidio itself: on real court judgments, Presidio set up sensibly hid slightly fewer names and kept slightly more harmless text (the numbers). Test both on your own data.

Python SDK

Install LiteLLM with pip install litellm, then:

import litellm

reply = litellm.completion(
    model="openai/gpt-4o-mini",              # "openai/" means "an OpenAI-compatible server"
    api_base="https://www.jnvoy.com/v1",     # must end in /v1
    api_key="<your Jnvoy key>",              # sent as Authorization: Bearer
    extra_headers={"x-provider-key": "<your OpenAI or Anthropic key>"},
    messages=[{"role": "user", "content": "Write to anna.lee@example.com about invoice 5521."}],
)
print(reply.choices[0].message.content)

Streaming (stream=True) and tool calls work the same way. A runnable version is examples/litellm_demo.py. For a Claude model use a name that starts with claude, for example openai/claude-haiku-4-5; Jnvoy sends those to Anthropic.

LiteLLM proxy

If you run the LiteLLM proxy, add Jnvoy to config.yaml. This version keeps one provider key on the LiteLLM server, read from an environment variable:

# LiteLLM proxy: every request goes through Jnvoy.
# The provider keys stay on the LiteLLM server, in environment variables.
#
#   export JNVOY_API_KEY=...       # your Jnvoy key
#   export OPENAI_API_KEY=...      # your own OpenAI key
#   export ANTHROPIC_API_KEY=...   # your own Anthropic key
#   litellm --config litellm_config.yaml
model_list:
  - model_name: gpt-4o-mini
    litellm_params:
      model: openai/gpt-4o-mini            # "openai/" means "an OpenAI-compatible server"
      api_base: https://www.jnvoy.com/v1   # must end in /v1
      api_key: os.environ/JNVOY_API_KEY
      extra_headers:
        x-provider-key: os.environ/OPENAI_API_KEY

  - model_name: claude-haiku
    litellm_params:
      model: openai/claude-haiku-4-5       # Jnvoy sends model names starting with "claude" to Anthropic
      api_base: https://www.jnvoy.com/v1
      api_key: os.environ/JNVOY_API_KEY
      extra_headers:
        x-provider-key: os.environ/ANTHROPIC_API_KEY

Start it with litellm --config config.yaml. Everything that calls the proxy, including the OpenAI libraries, now goes through Jnvoy without any change of its own. The same file is in the repository as examples/litellm_config.yaml.

A key for each user

If each caller should use their own provider key, leave the key out of the config and let LiteLLM pass the caller's header through:

# LiteLLM proxy: each caller brings their own provider key, and LiteLLM passes it on to Jnvoy.
#
#   export JNVOY_API_KEY=...       # your Jnvoy key
#   litellm --config litellm_config_per_user.yaml
#
# A caller then sends their key to LiteLLM in an x-provider-key header, next to their LiteLLM key.
model_list:
  - model_name: gpt-4o-mini
    litellm_params:
      model: openai/gpt-4o-mini
      api_base: https://www.jnvoy.com/v1   # must end in /v1
      api_key: os.environ/JNVOY_API_KEY

general_settings:
  forward_client_headers_to_llm_api: true

A caller then sends their provider key to LiteLLM:

curl http://localhost:4000/chat/completions \
  -H "Authorization: Bearer <your LiteLLM key>" \
  -H "x-provider-key: <your OpenAI or Anthropic key>" \
  -H "Content-Type: application/json" \
  -d '{"model": "gpt-4o-mini", "messages": [{"role": "user", "content": "Hello"}]}'

The same file is in the repository as examples/litellm_config_per_user.yaml. A Jnvoy customer key needs a provider key on every call, and answers 400 provider_key_required without one.

What to expect

• Token counts show as zero. Jnvoy does not return token usage yet, so LiteLLM reports 0 tokens and no cost for these calls. Use your provider's dashboard for spend.
• Errors map to LiteLLM's usual ones. A wrong Jnvoy key is an AuthenticationError (401). A rejected provider key is a BadRequestError (provider_key_rejected) and the message never repeats the key.
• Repeated identical requests can come from Jnvoy's cache. The reply says so in its jnvoy.cache_hit field.
• Some parameters are ignored. The response header x-jnvoy-ignored-params lists them.
• Not found? Check that api_base ends in /v1.

Limits

• Images, audio, JSON mode and more than one choice (n above 1) are refused with a 400, because they cannot be checked for personal data yet.
• Tool calls work with OpenAI models. Claude models with tools are refused for now.
• Jnvoy finds personal data that matches known patterns and common names. No tool catches everything, so keep it out of tool definitions and do not rely on any single layer. The accuracy page lists what it misses.
• We tested these configurations with LiteLLM 1.106.5 against a Jnvoy server, using a stand-in for the model provider. If something differs on your setup, tell us.