LLM Integration (OpenRouter)

The platform provides a gateway to OpenRouter, a unified API that gives access to hundreds of language models. Through this gateway, your backend can send prompts and receive AI-generated text — without managing API keys or dealing with third-party authentication directly.

The LLM integration is not provided as a pre-built library class. You are responsible for implementing your own service that communicates with the platform gateway. A ready-to-use reference implementation is available at the end of this page to help you get started.

When to use:

• Generating dynamic text content based on user input
• Summarizing, translating, or analyzing text
• Building AI-powered features like chatbots or content generators
• Any scenario requiring natural language understanding or generation

How It Works

Your service communicates with OpenRouter through the platform gateway. The gateway endpoint expects standard OpenAI-compatible chat completions requests with model and messages fields, and authenticates using the APP_BACKEND_APP_TOKEN environment variable passed as a Bearer token. This architecture means no API keys need to be stored in your code, and all requests benefit from centralized monitoring, rate limiting, and access control managed by the platform.

Error Handling

Your service doesn't need to detect or catch request failures. fetchAsPrincipal() / fetchAsApp() throw UpstreamHTTPError on non-2xx responses. If you let that error propagate, the backend's error middleware catches it and returns an appropriate response to the client. The one case worth handling yourself is an empty response (a successful call that returns no content), as shown in the reference implementation below. For how the platform handles errors, including automatic forwarding of 403 consent errors, see Error Handling.

Example Implementation

Express

import { env } from "@lib/env";
import { fetchAsPrincipal } from "@lib/utils";

export class LLMService {
  //#region modelConfig
  /**
   * Model used for LLM generation.
   * Can be changed to any compatible model available from https://openrouter.ai/models
   */
  static readonly MODEL_NAME = "~google/gemini-flash-latest";
  //#endregion modelConfig

  /**
   * Generates text from the configured LLM endpoint.
   *
   * @param systemInstruction - System prompt guiding model behavior
   * @param userPrompt - User prompt passed to the model
   * @returns A Promise that resolves to the generated response text
   * @throws {Error} If the LLM request fails or returns no content
   */
  static async generate(
    systemInstruction: string,
    userPrompt: string,
  ): Promise<string> {
    const url = `/api/gw/platform/llm/openrouter/chat/completions`;

    console.log("Sending request to LLM...");

    const response = await fetchAsPrincipal(url, {
      method: "POST",
      headers: {
        "Content-Type": "application/json",
        Authorization: `Bearer ${env.APP_BACKEND_APP_TOKEN}`,
      },
      body: JSON.stringify({
        model: LLMService.MODEL_NAME,
        messages: [
          { role: "system", content: systemInstruction },
          { role: "user", content: userPrompt },
        ],
      }),
    });

    if (!response.ok) {
      throw new Error(
        `LLM request failed: ${response.status} ${response.statusText}`,
      );
    }

    const data = (await response.json()) as {
      choices?: Array<{
        message?: {
          content?: string;
        };
      }>;
    };

    const answer = data.choices?.[0]?.message?.content;

    if (!answer) {
      console.error("The LLM did not return any content.");
      throw new Error("The LLM did not return any content.");
    }

    console.log("LLM response received successfully.");
    return answer;
  }
}

Python

import os
from openrouter import OpenRouter
from ..lib.utils import platform_url, raise_upstream_error_if_failure

class LLMService:
    """
    Handles interactions with the OpenRouter API for LLM generation.
    """
    
    #region modelConfig
    # model used for LLM generation.
    # Can be changed to any compatible model available from https://openrouter.ai/models
    MODEL_NAME = "~google/gemini-flash-latest"
    #endregion modelConfig

    @staticmethod
    def generate(system_instruction: str, user_prompt: str) -> str:
        """Generate text from the configured LLM endpoint.

        Args:
            system_instruction (str): System prompt guiding model behavior.
            user_prompt (str): User prompt passed to the model.

        Returns:
            str: Generated response text.
        """

        # Construct the client using the utils helper for the URL
        # and the environment variable for the token.
        with OpenRouter(
            server_url = platform_url("/api/gw/platform/llm/openrouter"),
            api_key=os.getenv("APP_BACKEND_APP_TOKEN")
        ) as client:
            print("Sending request to LLM...")
            response = client.chat.send(
                model=LLMService.MODEL_NAME,
                messages=[
                    {"role": "system", "content": system_instruction},
                    {"role": "user", "content": user_prompt}
                ]
            )

            # Extract the content from the response
            answer = response.choices[0].message.content

            if answer is None:
                print("The LLM did not return any content.")
                raise Exception("The LLM did not return any content.")
            
            print("LLM response received successfully.")
            return str(answer)