Parallel Works
View as Markdown
Share:

Custom OpenAI-Compatible Provider

A custom provider connects ACTIVATE to any endpoint that implements the OpenAI chat completions API. This gives you flexibility to use self-hosted models, third-party inference services, or any other compatible backend.

Endpoint Requirements

Your endpoint must support the following:

  • POST /v1/chat/completions — The standard OpenAI chat completions endpoint. ACTIVATE sends all chat messages to this route.
  • GET /v1/models (optional) — If your endpoint exposes this route, ACTIVATE can list available models dynamically so users can switch models without reconfiguring the provider.

The endpoint must accept an API key via the Authorization: Bearer <key> header.

Prerequisites

Before creating a custom provider, ensure that:

  • Your inference endpoint is running and reachable from the ACTIVATE platform
  • You have the endpoint URL and a valid API key

Creating a Provider

  1. Navigate to AI > Providers in the ACTIVATE sidebar.
  2. Click Add Provider.
  3. Select Custom as the cloud service provider.
  4. Fill in the required fields:
    • Name — A descriptive name for the provider (for example, "vLLM Llama 3").
    • Endpoint — The full URL to your chat completions endpoint (for example, https://my-server.example.com/v1/chat/completions).
    • API Key — The authentication key for your endpoint.
  5. Optionally add a Description and Tags.
  6. Click Create.

Custom providers are available immediately after creation — no provisioning step is required.

Verifying the Provider

  1. Open Chat.
  2. Select your new custom provider from the provider dropdown.
  3. Send a test message to confirm the endpoint responds correctly.

ACTIVATE automatically discovers available models by querying your endpoint's /v1/models route. If the model list dropdown is populated, your endpoint supports this route and ACTIVATE has successfully queried it.

Updating a Provider

You can update the endpoint and API key for a custom provider at any time:

  1. Navigate to AI > Providers.
  2. Select the provider you want to update.
  3. Modify the fields as needed.
  4. Save your changes.

This is useful for rotating API keys or pointing the provider at a different endpoint without creating a new provider.

Notes for Common Inference Servers

vLLM — Start vLLM with the --api-key flag to enable API key authentication. The default port is 8000. Your endpoint URL will be http://<host>:8000/v1/chat/completions.

Ollama — Ollama exposes an OpenAI-compatible API on port 11434. Set the endpoint to http://<host>:11434/v1/chat/completions. The model name should match the Ollama model tag (for example, "llama3").

Text Generation Inference (TGI) — TGI provides an OpenAI-compatible Messages API. Set the endpoint to http://<host>:8080/v1/chat/completions.

Deleting a Provider

  1. Navigate to AI > Providers.
  2. Find the provider you want to remove.
  3. Click the delete action for that provider.
  4. Confirm the deletion.

Deleting a custom provider only removes it from ACTIVATE. It does not affect the external inference endpoint.

Related Documentation