Choosing an AI provider
ReplyRabbit works with cloud providers, local model servers, and a custom OpenAI-compatible endpoint. Choose based on draft quality, cost, and where support content is sent. If you are unsure, start with OpenAI or Anthropic and change it later in your mailbox settings.
Provider choices
Five named providers are cloud services, two run locally, and the custom option connects to a server you configure. Cloud providers bill per token (you pay them directly). Costs for a custom endpoint depend on its operator.
| Provider | Type | Best for | Pricing |
|---|---|---|---|
| OpenAI | Cloud | The usual starting point. Available on the Free tier, so you can try ReplyRabbit with one provider before upgrading. | platform.openai.com |
| Anthropic | Cloud | Claude models. A strong all-round choice for natural, careful support replies. | anthropic.com |
| Google AI Studio | Cloud | Gemini models, including fast, low-cost options. | ai.google.dev |
| OpenRouter | Cloud | One API key, many models from different vendors. Good for experimenting and comparing models without signing up everywhere. | openrouter.ai |
| Z.AI | Cloud | GLM models. A capable, cost-conscious cloud option. | z.ai |
| Ollama (Local) | Local | Runs models on your own hardware (localhost:11434), so nothing leaves your server and there is no per-token cost. | ollama.com |
| LM Studio (Local) | Local | Runs models on your own hardware (localhost:1234), private with no per-token cost. | lmstudio.ai |
| Custom OpenAI-compatible | Your server or gateway | Connect ReplyRabbit to a reachable server that accepts supported OpenAI-style requests. | Depends on your server |
Free tier uses OpenAI
Cloud vs local
Cloud providers (OpenAI, Anthropic, Google AI Studio, OpenRouter, Z.AI) are the easiest way to start and usually offer the top draft quality. You pay per token, and the conversation content used for enabled AI features, such as drafts, next actions, and AI Signals, leaves your server and is sent to that provider's API.
Local providers (Ollama, LM Studio) run the model on your own hardware. Nothing leaves your server, and there is no per-token cost. The trade-off is that you need a reasonably capable machine, and draft quality depends on the local model you run. Local models require a paid tier (Pro or Team).
Custom OpenAI-compatible connects to the server or gateway you choose. If it is remote, support content leaves your FreeScout host. If it is on your private network, FreeScout must still be able to reach it. Enter the exact model IDs and test each capability you need. A chat endpoint does not guarantee embeddings or vision.
Prompt caching and token budgets
ReplyRabbit handles provider prompt caching automatically when it is safe for the selected provider. There is no mailbox setting to turn on.
- OpenAI: ReplyRabbit sends a safe cache routing key for direct OpenAI requests so repeated support prompts can be routed consistently.
- Anthropic: ReplyRabbit uses Claude prompt caching only when it sees likely reuse across matching draft requests. If Anthropic rejects cache control, ReplyRabbit retries the normal prompt.
- Gemini, OpenRouter, and Z.AI: ReplyRabbit records cache token metadata when the provider returns it, but it does not force a generic cache setting through those routes.
- Ollama and LM Studio: local models do not use provider prompt caching. They can still avoid per-token cloud billing because generation runs on your hardware.
- Custom OpenAI-compatible: ReplyRabbit does not assume OpenAI's prompt-cache controls work on a custom server.
Monthly Token Budget remains conservative. ReplyRabbit counts total input and output tokens, even when a provider bills cached input differently, because cached prompt tokens still use context and may count toward provider limits.
Which should I choose?
- Just getting started: use OpenAI or Anthropic. They give good results out of the box, and OpenAI is available on the Free tier so you can try ReplyRabbit first.
- Want to compare models: use OpenRouter. One API key gives you access to many models from different vendors, so you can experiment without separate accounts.
- Privacy or cost control matters most: use Ollama or LM Studio. Your customer data never leaves your server and there is no per-token bill, at the cost of running the model on your own hardware (Pro or Team tier).
- Already run a compatible API server: use Custom OpenAI-compatible on Pro or Team. Verify the chat model first, then set separate embedding and vision models only if your server supports them.
Next