Utilities

Utilities allow you to extract structured information from user inputs or assistant outputs. There are two types of utilities: Classification and Extraction.

Overview

Utilities are defined in the configuration and can be triggered:

• on_input: Process user input before generating a response
• on_output: Process assistant output after generation

By default, utilities only see the last message in the conversation. Use include_messages to control how much conversation history the utility can access (see Conversation History below).

Classification Utilities

Classification utilities assign one of predefined categories to the input/output.

Structure

{
  type: 'classify',
  classification_question: string,
  answers: string[],
  additional_context?: string,  // Domain knowledge for the LLM
  include_messages?: number | 'all',  // How many messages to include (default: 1)
  debug?: boolean  // Include reasoning in response
}

Example: Screen Navigation (on assistant output)

Classify which screen or experience the assistant is directing the user to, so your app can navigate automatically:

{
  "navigation": {
    "type": "classify",
    "classification_question": "Based on the assistant's response, which screen is the user being directed to: {{assistant_output}}",
    "answers": ["treehouse_adventure", "crosswords", "riddles"]
  }
}

When triggered with on_output, this returns:

{
  "kind": "data",
  "uid": "...",
  "data": {
    "navigation": "crosswords"
  }
}

You can then use this in your data handler to navigate the UI:

onDataMessage: (event) => {
  const screen = event.data.navigation
  if (screen && screen !== 'crosswords') {
    navigateTo(screen)
  }
}

Example: User Intent Detection (on user input)

In a game, classify what the user is trying to do so you can respond appropriately:

{
  "user_intent": {
    "type": "classify",
    "classification_question": "What is the user trying to do: {{user_input}}",
    "answers": ["make_a_guess", "ask_for_hint", "skip_question", "else"]
  }
}

Use the result to branch your game logic:

onDataMessage: (event) => {
  const intent = event.data.user_intent
  if (intent === 'ask_for_hint') {
    showHint()
  } else if (intent === 'skip_question') {
    nextQuestion()
  } else if (intent === 'make_a_guess') {
    evaluateGuess()
  }
}

Example: Answer Evaluation (using conversation history)

In a trivia or quiz game, check if the user answered the question correctly. This requires access to the full conversation so the utility knows what question was asked:

{
  "answer_correct": {
    "type": "classify",
    "classification_question": "Based on the conversation, did the user answer the assistant's question correctly?",
    "answers": ["correct", "incorrect"],
    "include_messages": "all"
  }
}

Use the result for scoring:

onDataMessage: (event) => {
  if (event.data.answer_correct === 'correct') {
    incrementScore()
    playSuccessSound()
  } else {
    playTryAgainSound()
  }
}

Template Variables

• {{user_input}}: The user's message
• {{assistant_output}}: The assistant's response (only for on_output)
• Custom context variables defined in your configuration

Extraction Utilities

Extraction utilities extract specific information from the input/output. By default, results are returned as free-form text. You can optionally use return_type to get structured JSON output that conforms to a schema you define.

Structure

{
  type: 'extract',
  extract_prompt: string,
  additional_instructions?: string,  // Extra guidance for the LLM
  include_messages?: number | 'all',  // How many messages to include (default: 1)
  return_type?: {           // Output format
    format: 'string' | 'array' | 'json',
    json_schema?: object    // JSON Schema definition (when format is 'json')
  },
  debug?: boolean  // Include reasoning in response
}

Example: Topic Extraction (Free-Form Text)

{
  "main_topic": {
    "type": "extract",
    "extract_prompt": "Extract the main topic from this text in 2-5 words: {{user_input}}"
  }
}

Returns:

{
  "kind": "data",
  "uid": "...",
  "data": {
    "main_topic": "Machine Learning"
  }
}

Example: Image Prompt Generation (Free-Form Text)

{
  "image_prompt": {
    "type": "extract",
    "extract_prompt": "Based on this story segment, create a detailed image generation prompt: {{assistant_output}}"
  }
}

JSON-Structured Extraction

Use return_type to get structured JSON output instead of free-form text. Define a JSON Schema and the extraction result will conform to it.

Structure

{
  type: 'extract',
  extract_prompt: string,
  return_type: {
    format: 'json',
    json_schema: {
      type: 'object',
      properties: { ... },
      required: [ ... ]
    }
  }
}

Example: Extract User Profile

{
  "user_profile": {
    "type": "extract",
    "extract_prompt": "Extract the person's name and age from the conversation: {{user_input}}",
    "return_type": {
      "format": "json",
      "json_schema": {
        "type": "object",
        "properties": {
          "name": { "type": "string" },
          "age": { "type": "integer" }
        },
        "required": ["name", "age"]
      }
    }
  }
}

Returns:

{
  "kind": "data",
  "uid": "...",
  "data": {
    "user_profile": {
      "name": "Alice",
      "age": 12
    }
  }
}

With return_type, the result is a parsed JSON object rather than a string, so you can access fields directly:

onDataMessage: (event) => {
  const profile = event.data.user_profile
  console.log(profile.name)  // "Alice"
  console.log(profile.age)   // 12
}

Example: Extract Quest Details (Nested Schema)

{
  "quest_details": {
    "type": "extract",
    "extract_prompt": "Extract the quest information from the conversation: {{conversation_history}}",
    "return_type": {
      "format": "json",
      "json_schema": {
        "type": "object",
        "properties": {
          "quest_name": { "type": "string" },
          "difficulty": { "type": "string", "enum": ["easy", "medium", "hard"] },
          "rewards": {
            "type": "array",
            "items": { "type": "string" }
          },
          "completed": { "type": "boolean" }
        },
        "required": ["quest_name", "difficulty"]
      }
    }
  }
}

When to Use JSON-Structured Extraction

Approach	Best For
Free-form text (extract_prompt only)	Simple values, descriptions, prompts for image generation
JSON-structured (return_type)	Multiple fields, typed values, data you'll use programmatically

Triggering Utilities

On Input (Before Response)

Specify utilities to run before generating a response:

await conversation.sendText('I think the answer is kiwi!', {
  on_input: ['user_intent', 'answer_correct']
});

The utilities will process the user input, and their results will be available before the assistant responds.

On Output (After Response)

Specify utilities to run after generating a response:

await conversation.sendText('Tell me a story about a brave cat', {
  on_output: ['scene_description']
});

The utility will process the assistant's output and return structured data.

Conversation History

By default, utilities only see the last message in the conversation. Use include_messages to control how much history the utility receives:

{
  "answer_correct": {
    "type": "classify",
    "classification_question": "Did the user answer the question correctly?",
    "answers": ["correct", "incorrect"],
    "include_messages": "all"
  }
}

Value	Behavior
1 (default)	Only the last message
"all"	Entire conversation history
3	Last 3 messages

When to use "all":

• Answer evaluation (need to see what question was asked)
• Summarization across the conversation
• Tracking topics mentioned earlier

When to use default (last message):

• Intent detection (what is the user doing right now?)
• Sentiment analysis of the current message
• Extracting info from the latest response

Additional Context

Use additional_context (for classify) or additional_instructions (for extract) to provide domain knowledge that helps the LLM make accurate decisions:

{
  "answer_evaluation": {
    "type": "classify",
    "classification_question": "Did the user correctly answer the question?",
    "answers": ["correct", "incorrect", "partially_correct"],
    "additional_context": "Accept common misspellings of the answer. If the question is about naming a fruit, count anything that is botanically a fruit, even if commonly known as a vegetable (e.g., tomatoes count as fruits).",
    "include_messages": "all"
  }
}

Use additional context when:

• The classification requires rules not obvious from the conversation
• You need to define what counts as "correct" for your specific game
• Domain terminology needs clarification

Return Types for Extraction

By default, extract utilities return a string. Use return_type to get structured data:

Array Return Type

{
  "mentioned_animals": {
    "type": "extract",
    "extract_prompt": "List all animals mentioned in the conversation",
    "return_type": { "format": "array" },
    "include_messages": "all"
  }
}

Returns:

{
  "mentioned_animals": ["cat", "dog", "parrot"]
}

JSON Return Type

{
  "story_metadata": {
    "type": "extract",
    "extract_prompt": "Extract story metadata as JSON with keys: title, setting, mood",
    "return_type": { "format": "json" },
    "include_messages": "all"
  }
}

Returns:

{
  "story_metadata": {
    "title": "The Brave Little Fox",
    "setting": "enchanted forest",
    "mood": "adventurous"
  }
}

Debugging Utilities

When utilities return unexpected results, enable debug mode:

{
  "answer_correct": {
    "type": "classify",
    "classification_question": "Did the user answer correctly?",
    "answers": ["correct", "incorrect"],
    "debug": true
  }
}

With debug: true, the response includes a reason field explaining the LLM's reasoning:

{
  "answer_correct": "correct",
  "answer_correct_reason": "The user said 'kiwi' which is a valid fruit starting with K, matching what was asked."
}

Debugging Tips

Classification returning wrong answers?

• Make your answers more distinct from each other
• Add additional_context to clarify edge cases
• Use include_messages: "all" if the utility needs context from earlier messages

Extraction returning unexpected format?

• Be explicit about the output format in your prompt
• Use sentinel values: "Return 'none' if no match found"
• Use return_type for structured data instead of asking for JSON in the prompt

Complete Example: Fruit Guessing Game

A child-friendly game where the avatar asks questions like "Name a fruit starting with K", evaluates the answer, and shows an image of the fruit when correct.

await conversation.setConfiguration({
  prompt: 'You are a friendly game host for children. Ask the child to name fruits starting with different letters. Celebrate when they get it right!',
  utilities: {
    answer_correct: {
      type: 'classify',
      classification_question: 'Based on the conversation, did the user correctly name a fruit that matches what was asked?',
      answers: ['correct', 'incorrect'],
      include_messages: 'all',
      enabled: true
    },
    fruit_name: {
      type: 'extract',
      extract_prompt: 'Extract the fruit name from the user input. Return only the fruit name in lowercase, or "none" if no fruit was mentioned: {{user_input}}',
      enabled: true
    }
  }
});

// Handle utility results
conversation.on('data-event', async (event) => {
  const { answer_correct, fruit_name } = event.data;

  // If the answer is correct and we have a fruit name, generate an image
  if (answer_correct === 'correct' && fruit_name && fruit_name !== 'none') {
    incrementScore();
    playSuccessSound();

    // Generate an image of the fruit as a reward
    await conversation.generateImage({
      provider: 'replicate',
      prompt: `A colorful, child-friendly illustration of a ${fruit_name}`,
      aspect_ratio: '1:1'
    });
  } else if (answer_correct === 'incorrect') {
    playTryAgainSound();
  }
});

// User responds to "Name a fruit starting with K"
await conversation.sendText('Kiwi!', {
  on_input: ['answer_correct', 'fruit_name']
});

Use Cases

Answer Evaluation

Check if the user answered a question correctly in a quiz or trivia game:

{
  "answer_correct": {
    "type": "classify",
    "classification_question": "Did the user answer the question correctly?",
    "answers": ["correct", "incorrect"],
    "include_messages": "all"
  }
}

Game Intent Recognition

Determine what the player wants to do next:

{
  "player_intent": {
    "type": "classify",
    "classification_question": "What does the player want to do: {{user_input}}",
    "answers": ["make_a_guess", "ask_for_hint", "skip_question", "play_again", "quit"]
  }
}

Character & Item Extraction

Extract characters or items the user mentions for game logic or image generation:

{
  "character_name": {
    "type": "extract",
    "extract_prompt": "Extract the character name the user mentioned, or return 'none': {{user_input}}"
  },
  "item_described": {
    "type": "extract",
    "extract_prompt": "Extract the item or object the user described, or return 'none': {{user_input}}"
  }
}

Story Element Extraction

Extract structured data from a storytelling conversation:

{
  "story_title": {
    "type": "extract",
    "extract_prompt": "Extract the story title if mentioned, or return 'none'",
    "include_messages": "all"
  },
  "current_location": {
    "type": "extract",
    "extract_prompt": "What location is the story currently taking place in: {{assistant_output}}"
  }
}

Image Generation Automation

Automatically generate images for storytelling:

{
  "scene_description": {
    "type": "extract",
    "extract_prompt": "Create a detailed visual description of the scene in: {{assistant_output}}"
  }
}

Then use the extracted description to generate images:

conversation.on('data-event', async (event) => {
  if (event.data.scene_description) {
    await conversation.generateImage({
      prompt: event.data.scene_description,
      provider: 'replicate'
    });
  }
});

The enabled Field

Each utility definition supports an enabled field. When set to true, the utility is active and will run when triggered. This allows you to define utilities in your configuration and toggle them on/off without removing them:

{
  answer_correct: {
    type: 'classify',
    classification_question: 'Did the user answer correctly?',
    answers: ['correct', 'incorrect'],
    include_messages: 'all',
    enabled: true
  }
}

When building the list of active utilities to trigger, filter by the enabled field:

const activeUtilities = Object.entries(utilities)
  .filter(([, util]) => util.enabled === true)
  .map(([name]) => name)

Running Utilities on Every Message

The per-message triggering shown above (on_input / on_output in sendText) works well for one-off use. But in most real applications, you want utilities to run automatically on every message. To do this:

1. Define utilities in the conversation configuration when creating or updating the conversation.
2. Pass utility names in on_output (or on_input) with every interaction.

// 1. Define utilities in the config
const config = {
  apiUrl: 'your-api-url',
  apiKey: 'your-api-key',
  prompt: 'You are a friendly quiz host for children.',
  utilities: {
    user_intent: {
      type: 'classify',
      classification_question: 'What is the user trying to do: {{user_input}}',
      answers: ['make_a_guess', 'ask_for_hint', 'skip_question', 'else'],
      enabled: true,
    },
    answer_correct: {
      type: 'classify',
      classification_question: 'Did the user answer correctly?',
      answers: ['correct', 'incorrect'],
      include_messages: 'all',
      enabled: true,
    },
  },
  // Specify which utilities to run on every input
  onInputUtilities: ['user_intent', 'answer_correct'],
  hooks: {
    onDataMessage: (event) => {
      // Handle utility results here
      console.log('Intent:', event.data.user_intent)
      console.log('Correct:', event.data.answer_correct)
    },
  },
}

const manager = new ConversationManager(config)
await manager.initialize()

With this setup, every user message automatically triggers the user_intent and answer_correct utilities, and the results arrive in the onDataMessage callback.

Handling Utility Results

Utility results arrive as a data message in the onDataMessage callback. The event.data object contains one key per utility, with the result as the value.

Type Safety

Utility results can arrive as different types depending on the LLM response. Always validate the type before using the result:

onDataMessage: (event) => {
  const data = event.data

  // Classification results are typically strings
  if (data.answer_correct) {
    const result = typeof data.answer_correct === 'string'
      ? data.answer_correct.trim().toLowerCase()
      : String(data.answer_correct).trim().toLowerCase()

    if (result === 'correct') {
      incrementScore()
    }
  }

  // Extract results may be strings or JSON objects
  if (data.story_metadata) {
    let metadata
    if (typeof data.story_metadata === 'string') {
      try {
        metadata = JSON.parse(data.story_metadata)
      } catch {
        metadata = { raw: data.story_metadata }
      }
    } else if (typeof data.story_metadata === 'object') {
      metadata = data.story_metadata
    }
  }
}

Sentinel Values

A common pattern is to instruct the utility to return a special value when no action is needed. This prevents unnecessary processing:

{
  "scene_description": {
    "type": "extract",
    "extract_prompt": "Describe the scene from the story. If there is no new scene, return 'no_scene'. Otherwise return a 2-sentence visual description.",
    "enabled": true
  }
}

onDataMessage: (event) => {
  const description = event.data.scene_description
  if (description && description !== 'no_scene') {
    // Generate an image for this scene
    generateImage(description)
  }
}

This keeps your handler logic clean and avoids generating images for conversational turns where the scene hasn't changed.

Coordinating Multiple Utilities

When using multiple utilities together, their results arrive in the same data message. You can use one utility's result to gate another's processing:

onDataMessage: (event) => {
  const data = event.data

  // Check if the story has started before processing image descriptions
  if (data.story_ready_to_start) {
    const ready = String(data.story_ready_to_start).trim().toLowerCase()
    if (ready === 'yes') {
      storyStarted = true
    }
  }

  // Only generate images after the story has started
  if (data.image_description) {
    const description = String(data.image_description).trim()
    if (description !== 'no_scene' && storyStarted) {
      generateImage(description)
    }
  }

  // Track if the story has ended
  if (data.story_ended) {
    const ended = String(data.story_ended).trim().toLowerCase()
    if (ended === 'yes') {
      storyEnded = true
      showEndScreen()
    }
  }
}

Combining Utilities with Image Generation

Utilities pair naturally with the SDK's generateImage() method. Use an extract utility to create image prompts from the conversation, then generate images automatically:

const config = {
  prompt: 'You are a storyteller...',
  utilities: {
    scene_description: {
      type: 'extract',
      extract_prompt: `Based on the assistant's latest response, describe the scene
        as a 2-sentence illustration. If there is no new scene, return "no_scene".`,
      enabled: true,
    },
  },
  onOutputUtilities: ['scene_description'],
  hooks: {
    onDataMessage: async (event) => {
      const description = String(event.data.scene_description || '').trim()
      if (description && description !== 'no_scene') {
        try {
          const result = await conversationManager.generateImage({
            provider: 'replicate',
            model: 'black-forest-labs/flux-schnell-lora',
            prompt: description,
            aspect_ratio: '16:9',
          })
          // Display the generated image
          displayImage(result.url)
        } catch (error) {
          console.error('Image generation failed:', error)
        }
      }
    },
  },
}

Real-World Example: Interactive Storyteller

This example shows how the Storyteller template uses four coordinated utilities to drive an entire interactive story experience with automatic illustrations, metadata extraction, and story lifecycle detection.

Utility Definitions

const utilities = {
  // Generates scene descriptions for automatic illustration
  image_description: {
    type: 'extract',
    extract_prompt: `You are a children's book illustrator. Describe a simple,
      colorful illustration matching the current story scene in exactly 2 sentences.
      If the story hasn't progressed, return "no_scene".`,
    enabled: true,
  },

  // Extracts story title and topic as JSON
  story_metadata: {
    type: 'extract',
    extract_prompt: `Extract the story title and topic from the conversation.
      Return as JSON: {"title": "string or null", "topic": "string or null"}`,
    enabled: true,
  },

  // Detects when the story has concluded
  story_ended: {
    type: 'extract',
    extract_prompt: `Has the story concluded? Look for "THE END!" or similar
      conclusion phrases in the assistant's messages. Return only "yes" or "no".`,
    enabled: true,
  },

  // Detects when the user is ready to start
  story_ready_to_start: {
    type: 'extract',
    extract_prompt: `Did the assistant ask if the user is ready to start the
      story, and did the user agree? Return only "yes" or "no".`,
    enabled: true,
  },
}

Handling Results

let storyStarted = false
let storyEnded = false

onDataMessage: (event) => {
  const data = event.data

  // 1. Check if user is ready to start the story
  if (data.story_ready_to_start === 'yes' && !storyStarted) {
    storyStarted = true
  }

  // 2. Extract metadata for the UI (title bar, cover image)
  if (data.story_metadata) {
    const metadata = JSON.parse(String(data.story_metadata))
    if (metadata.title) updateTitle(metadata.title)
    if (metadata.topic) updateTopic(metadata.topic)
  }

  // 3. Generate illustrations when the scene changes
  if (data.image_description) {
    const desc = String(data.image_description).trim()
    if (desc !== 'no_scene' && storyStarted && !storyEnded) {
      conversationManager.generateImage({
        provider: 'replicate',
        prompt: desc,
        aspect_ratio: '16:9',
      })
    }
  }

  // 4. Detect story end
  if (data.story_ended === 'yes') {
    storyEnded = true
    showNewStoryButton()
  }
}

This pattern — defining extract utilities with clear output rules, sentinel values, and a centralized data handler — scales well to any application that needs structured data from conversations.

Best Practices

1. Keep classification answers mutually exclusive - Each answer should represent a distinct category
2. Be specific in extract prompts - Clearly describe what you want to extract and the exact output format
3. Limit the number of categories - 3-10 answers work best for classification
4. Use descriptive variable names - Name utilities based on what they extract
5. Test with various inputs - Ensure utilities work across different user messages
6. Combine utilities - Use multiple utilities together for rich structured data
7. Use sentinel values - Return values like "no_scene" or "none" when no action is needed, rather than leaving the LLM to decide the format of a "no result"
8. Always validate result types - Utility results can arrive as strings, objects, or JSON-wrapped strings. Always check typeof before parsing
9. Use enabled: true/false - Toggle utilities without removing them from the configuration
10. Gate processing on state - When coordinating multiple utilities, use one utility's result to control whether another's result is acted upon

Examples in the API Tester

Try these example scenarios in the API Tester:

• Quiz Game - Evaluate user answers and track scores
• Storytelling - Extract scene descriptions for image generation
• Game Navigation - Detect which activity the user wants to play
• Character Creation - Extract character traits from user descriptions

Next Steps

• Try the Interactive API Tester
• See Example Use Cases
• WebSocket Protocol