Cloudflare Docs
Workers AI
Cloudflare Docs
Workers AI
GitHub icon
Edit this page on GitHub
Set theme to dark (⇧+D)
  1. Products
  2. Workers AI
  3. Configuration
  4. Workers AI SDK

Workers AI SDK

The Workers AI SDK provides an interface between a Worker or Pages Function and Workers AI.

The Workers AI SDK makes Workers AI APIs available for use in your code. To import the Workers AI SDK, run:

$ npm install --save-dev @cloudflare/ai

$ yarn add --dev @cloudflare/ai

Import the library in your code:

import { Ai } from "@cloudflare/ai";

​​ Constructor

​​ new Ai()

new Ai() creates a new Ai instance:

import { Ai } from "@cloudflare/ai";



export interface Env {
  // If you set another name in wrangler.toml as the value for 'binding',
  // replace "AI" with the variable name you defined.
  AI: any;

}



export default {
  async fetch(request: Request, env: Env) {
    const ai = new Ai(env.AI);


    const response = await ai.run('@cf/meta/llama-2-7b-chat-int8', {
        prompt: "What is the origin of the phrase Hello, World"
      }
    );


    return new Response(JSON.stringify(response));
  },

};

env.AI is the AI binding defined in your wrangler.toml configuration.

​​ Methods

​​ async ai.run()

async ai.run() is a method of the class instance created by new Ai().

async ai.run() runs a model. Takes a model as the first parameter, and an object as the second parameter.

import { Ai } from '@cloudflare/ai'



// sessionOptions are optional

const ai = new Ai(env.AI, { sessionOptions: { ctx }});



const answer = ai.run('@cf/meta/llama-2-7b-chat-int8', {
    prompt: "What is the origin of the phrase 'Hello, World'"

});

Parameters

  • model string required

    • The model to run.

  • options object depends on the model

    • Depends on the model type.

    Supported options

    • stream boolean optional
      • Returns a stream of results as they are available.

Optionally, you can pass a stream property to the options object. This will return a stream of results as they are available.

import { Ai } from '@cloudflare/ai'



// sessionOptions are optional

const ai = new Ai(env.AI, { sessionOptions: { ctx }});



const answer = await ai.run('@cf/meta/llama-2-7b-chat-int8', {
    prompt: "What is the origin of the phrase 'Hello, World'",
    stream: true

});



return new Response(answer, {
    headers: { "content-type": "text/event-stream" }

});