Cloudflare Docs
AI Gateway
Cloudflare Docs
AI Gateway
GitHub icon
Edit this page on GitHub
Set theme to dark (⇧+D)
  1. Products
  2. AI Gateway
  3. Get started
  4. Connecting your application

Connecting your application

In this guide, you will learn how to connect your application to your AI Gateway. You will need to have an AI Gateway created to continue with this guide.

Once you have configured a Gateway in the AI Gateway dashboard, click on “API Endpoints” to find your AI Gateway endpoint. AI Gateway offers multiple endpoints for each Gateway you create - one endpoint per provider, and one Universal Endpoint.

​​ Universal

https://gateway.ai.cloudflare.com/v1/ACCOUNT_TAG/GATEWAY

AI Gateway offers multiple endpoints for each Gateway you create - one endpoint per provider, and one Universal Endpoint. The Universal Endpoint requires some adjusting to your schema, but supports additional features. Some of these features are, for example, retrying a request if it fails the first time, or configuring a fallback model/provider when a request fails.

You can use the Universal endpoint to contact every provider. The payload is expecting an array of message, and each message is an object with the following parameters:

  • provider : the name of the provider you would like to direct this message to. Can be openai/huggingface/replicate
  • endpoint: the pathname of the provider API you’re trying to reach. For example, on OpenAI it can be chat/completions, and for HuggingFace this might be bigstar/code. See more in the sections that are specific to each provider.
  • authorization: the content of the Authorization HTTP Header that should be used when contacting this provider. This usually starts with “Token” or “Bearer”.
  • query: the payload as the provider expects it in their official API.
Request
curl https://gateway.ai.cloudflare.com/v1/ACCOUNT_TAG/GATEWAY -X POST \
  --header 'Content-Type: application/json' \
  --data '[
    {
   	 "provider": "huggingface",
   	 "endpoint": "bigcode/starcoder",
   	 "headers": {
        "Authorization": "Bearer $TOKEN",
        "Content-Type": "application/json"
       },
   	 "query": {
   		 "input": "console.log"
   	 }
    },
    {
   	 "provider": "openai",
   	 "endpoint": "chat/completions",
       "headers": {
        "Authorization": "Bearer $TOKEN",
        "Content-Type": "application/json"
       },
   	 "query": {
   		 "model": "gpt-3.5-turbo",
   		 "stream": true,
   		 "messages": [
   			 {
   				 "role": "user",
   				 "content": "What is Cloudflare?"
   			 }
   		 ]
   	 }
    },
    {
   	 "provider": "replicate",
   	 "endpoint": "predictions",
   	 "authorization": "Token $TOKEN",
   	 "query": {
   		 "version": "2796ee9483c3fd7aa2e171d38f4ca12251a30609463dcfd4cd76703f22e96cdf",
   		 "input": {
   			 "prompt": "What is Cloudflare?"
   		 }
   	 }
    }
]'

The above will send a request to HuggingFace Inference API, if it fails it will proceed to OpenAI, and then Replicate.

​​ Workers AI

https://gateway.ai.cloudflare.com/v1/ACCOUNT_TAG/GATEWAY/workers-ai/

When making requests to Workers AI, replace https://api.cloudflare.com/client/v4/accounts/ACCOUNT_TAG/ai/run in the URL you’re currently using with https://gateway.ai.cloudflare.com/v1/ACCOUNT_TAG/GATEWAY/workers-ai.

Then add the model you want to run at the end of the URL. You can see the list of Workers AI models and pick the ID.

You’ll need to generate an API token with Workers AI read access and use it in your request.

Request to Workers AI llama model
curl https://gateway.ai.cloudflare.com/v1/ACCOUNT_TAG/GATEWAY/workers-ai/@cf/meta/llama-2-7b-chat-int8  -X POST \
  --header 'Authorization: Bearer $TOKEN' \
  --header 'Content-Type: application/json' \
  --data '{ "prompt": "Where did the phrase Hello World come from" }'

Request to Workers AI text classification model
curl https://gateway.ai.cloudflare.com/v1/ACCOUNT_TAG/GATEWAY/workers-ai/@cf/huggingface/distilbert-sst-2-int8 -X POST \
  --header 'Authorization: Bearer $TOKEN' \
  --header 'Content-Type: application/json' \
  --data '{ "text": "This pizza is amazing!" }'

​​ OpenAI

https://gateway.ai.cloudflare.com/v1/ACCOUNT_TAG/GATEWAY/openai

When making requests to OpenAI, replace https://api.openai.com/v1 in the URL you’re currently using with https://gateway.ai.cloudflare.com/v1/ACCOUNT_TAG/GATEWAY/openai.

Request
curl https://gateway.ai.cloudflare.com/v1/ACCOUNT_TAG/GATEWAY/openai/chat/completions -X POST \
  --header 'Authorization: Bearer $TOKEN' \
  --header 'Content-Type: application/json' \
  --data ' {
   		 "model": "gpt-3.5-turbo",
   		 "messages": [
   			 {
   				 "role": "user",
   				 "content": "how to build a wooden spoon in 3 short steps? give as short as answer as possible"
   			 }
   		 ]
   	 }
'

If you’re using a library like openai-node, set the baseURL to your OpenAI endpoint like this:

index.js
import OpenAI from 'openai';



const openai = new OpenAI({
	apiKey: 'my api key', // defaults to process.env["OPENAI_API_KEY"]
	baseURL: "https://gateway.ai.cloudflare.com/v1/ACCOUNT_TAG/GATEWAY/openai"

});

​​ Azure OpenAI

https://gateway.ai.cloudflare.com/v1/ACCOUNT_TAG/GATEWAY/azure-openai/RESOURCE_NAME/MODEL_NAME

When making requests to Azure OpenAI, you will need:

  • AI Gateway account tag
  • AI Gateway gateway name
  • Azure OpenAI API key
  • Azure OpenAI resource name
  • Azure OpenAI deployment name (aka model name)

Your new base URL will use the data above in this structure: https://gateway.ai.cloudflare.com/v1/ACCOUNT_TAG/GATEWAY/azure-openai/RESOURCE_NAME/MODEL_NAME. Then, you can append your endpoint and api-version at the end of the base URL, like .../chat/completions?api-version=2023-05-15.

Example fetch request
curl --request POST \
  --url 'https://gateway.ai.cloudflare.com/v1/ACCOUNT_TAG/GATEWAY/azure-openai/RESOURCE_NAME/MODEL_NAME/chat/completions?api-version=2023-05-15' \
  --header 'Content-Type: application/json' \
  --header 'api-key: KEY' \
  --data '{
  "messages": [
    {
      "role": "system",
      "content": "You are a helpful assistant."
    },
    {
      "role": "user",
      "content": "What is Cloudflare?"
    }
  ]
}'

If you are using the openai-node library, you can set your endpoint like this:

index.js
import OpenAI from "openai";


  const resource = 'xxx'; //without the .openai.azure.com
  const model = 'xxx';
  const apiVersion = 'xxx';
  const apiKey = env.AZURE_OPENAI_API_KEY;


  const azure_openai = new OpenAI({
    apiKey,
    baseURL: `https://gateway.ai.cloudflare.com/v1/ACCOUNT_TAG/GATEWAY/azure-openai/${resource}/${model}`,
    defaultQuery: { 'api-version': apiVersion },
    defaultHeaders: { 'api-key': apiKey },
  });

​​ HuggingFace

https://gateway.ai.cloudflare.com/v1/ACCOUNT_TAG/GATEWAY/huggingface

When making requests to HuggingFace Inference API, replace https://api-inference.huggingface.co/models/ in the URL you’re currently using with https://gateway.ai.cloudflare.com/v1/ACCOUNT_TAG/GATEWAY/huggingface. Note that the model you’re trying to access should come right after, for example https://gateway.ai.cloudflare.com/v1/ACCOUNT_TAG/GATEWAY/huggingface/bigcode/starcoder.

Request
curl https://gateway.ai.cloudflare.com/v1/ACCOUNT_TAG/GATEWAY/huggingface/bigcode/starcoder -X POST \
  --header 'Authorization: Bearer $TOKEN' \
  --header 'Content-Type: application/json' \
  --data '{
    "inputs": "console.log"
}'

If you are using the HuggingFace.js library, you can set your inference endpoint like this:

index.js
import { HfInferenceEndpoint } from '@huggingface/inference'



const hf = new HfInferenceEndpoint(
	"https://gateway.ai.cloudflare.com/v1/{account_id}/{gateway}/huggingface/gpt2",
	env.HF_API_TOKEN

);

​​ Replicate

https://gateway.ai.cloudflare.com/v1/ACCOUNT_TAG/GATEWAY/replicate

When making requests to Replicate, replace https://api.replicate.com/v1 in the URL you’re currently using with https://gateway.ai.cloudflare.com/v1/ACCOUNT_TAG/GATEWAY/replicate.

Request
curl https://gateway.ai.cloudflare.com/v1/ACCOUNT_TAG/GATEWAY/replicate/predictions -X POST \
  --header 'Authorization: Token $TOKEN' \
  --header 'Content-Type: application/json' \
  --data '{
    "version": "2796ee9483c3fd7aa2e171d38f4ca12251a30609463dcfd4cd76703f22e96cdf",
    "input": {
   	 "prompt": "What is Cloudflare?"
    }
}'

​​ Amazon Bedrock

https://gateway.ai.cloudflare.com/v1/ACCOUNT_TAG/GATEWAY/aws-bedrock

When making requests to Amazon Bedrock, replace https://bedrock-runtime.us-east-1.amazonaws.com/ in the URL you’re currently using with https://gateway.ai.cloudflare.com/v1/ACCOUNT_TAG/GATEWAY/aws-bedrock/bedrock-runtime/us-east-1/.

Then add the model you want to run at the end of the URL.

Request
curl https://gateway.ai.cloudflare.com/v1/ACCOUNT_TAG/GATEWAY/aws-bedrock/bedrock-runtime/us-east-1/model/amazon.titan-embed-text-v1/invoke \
  -u AccessKey:SecretKey \
  -H "Content-Type: application/json" \
  -v --aws-sigv4 aws:amz:us-east-1:bedrock \
  -d '{
    "inputText": "Cloudflare’s AI Gateway allows you to gain visibility and control over your AI apps"
    }'

​​ Next Steps

Observe and control
Observe and control your AI applications