Skip to main content
Version: Next

LLMInferenceService with AgentGateway

This guide walks through integrating a KServe LLMInferenceService with AgentGateway to enable LLM-aware routing with token tracking, GenAI telemetry (OpenTelemetry semantic conventions), and token-based rate limiting. The key mechanism is overriding the HTTPRoute backendRef via LLMInferenceServiceConfig so that KServe's auto-generated routes point to an AgentgatewayBackend instead of a plain InferencePool or Service.

AgentGateway Overview​

AgentGateway is a Rust-based proxy under the AI Agent Infrastructure Foundation (AAIF) at the Linux Foundation. It implements the Kubernetes Gateway API but is LLM-aware: it parses OpenAI chat completion requests and responses, extracts token usage, emits OpenTelemetry GenAI semantic conventions, and enforces token-based rate limits and policies. Key custom resources:

  • AgentgatewayBackend: Declares a backend as an LLM provider so the gateway activates its LLM pipeline (token parsing, model tracking, GenAI telemetry).
  • AgentgatewayPolicy: Attaches governance policies such as token-based rate limiting.
  • HTTPRoute: Standard Gateway API routing — AgentGateway supports AgentgatewayBackend as a backendRef kind.

For more information, see the AgentGateway documentation.

Why This Integration Matters​

When KServe generates HTTPRoutes for an LLMInferenceService, the backendRef defaults to InferencePool (or Service for catch-all routes). AgentGateway treats these as generic HTTP traffic — it routes requests correctly but cannot activate LLM-aware features because it doesn't know the backend serves LLM traffic.

By overriding the backendRef to use AgentgatewayBackend, the gateway recognizes the backend as an LLM provider and activates:

CapabilityPlain Service / InferencePoolAgentgatewayBackend
RoutingYesYes
Protocol detectionhttp (generic)llm (LLM-aware)
Token parsingNoYes (extracts usage.input_tokens, usage.output_tokens from response body)
Model trackingNoYes (gen_ai.request.model, gen_ai.response.model)
GenAI semantic conventions (OTel)NoYes (gen_ai.* attributes per OTel GenAI spec)
Token-based rate limitingNoYes
Cost trackingNoYes
note

KServe supports distributed tracing natively via spec.tracing, which provides request-level spans and traces. The GenAI semantic conventions listed above are complementary — they add LLM-specific attributes (token counts, model name, operation type) at the gateway level.

Prerequisites​

Before you begin, ensure you have the following components installed and configured:

Deploy LLMInferenceService​

Create Namespace​

kubectl create namespace kserve-test

Create Gateway​

Create an AgentGateway Gateway resource:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: agentgateway
  namespace: kserve-test
spec:
  gatewayClassName: agentgateway
  listeners:
    - name: http
      protocol: HTTP
      port: 80
      allowedRoutes:
        namespaces:
          from: Same

Deploy Your Model​

Deploy an LLMInferenceService. This example uses a small model for demonstration; replace with your model of choice:

apiVersion: serving.kserve.io/v1alpha1
kind: LLMInferenceService
metadata:
  name: my-model
  namespace: kserve-test
spec:
  model:
    uri: "hf://Qwen/Qwen2.5-0.5B-Instruct"
    name: Qwen/Qwen2.5-0.5B-Instruct
  replicas: 1
  router:
    route: {}
  template:
    containers:
      - name: main
        resources:
          limits:
            nvidia.com/gpu: 1
          requests:
            nvidia.com/gpu: 1

Wait for the LLMInferenceService to be ready:

kubectl wait --for=condition=Ready llminferenceservice/my-model \
  -n kserve-test --timeout=300s

Configure LLM-Aware Routing​

Step 1: Create AgentgatewayBackend​

Create an AgentgatewayBackend that declares the KServe predictor Service as an LLM provider. This tells AgentGateway to activate its LLM pipeline for traffic to this backend:

apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayBackend
metadata:
  name: my-model-backend
  namespace: kserve-test
spec:
  ai:
    provider:
      openai:
        model: Qwen/Qwen2.5-0.5B-Instruct
      host: my-model-kserve-workload-svc.kserve-test.svc.cluster.local
      port: 8000
tip

The host should point to the KServe workload Service. The naming convention is {llminferenceservice-name}-kserve-workload-svc. Verify with:

kubectl get svc -n kserve-test | grep workload

Step 2: Override HTTPRoute backendRef​

You have two options to override the backendRef in KServe's auto-generated HTTPRoutes.

Override the route configuration on an individual LLMInferenceService using spec.router.route.http:

apiVersion: serving.kserve.io/v1alpha1
kind: LLMInferenceService
metadata:
  name: my-model
  namespace: kserve-test
spec:
  model:
    uri: "hf://Qwen/Qwen2.5-0.5B-Instruct"
    name: Qwen/Qwen2.5-0.5B-Instruct
  replicas: 1
  router:
    route:
      http:
        spec:
          parentRefs:
            - group: gateway.networking.k8s.io
              kind: Gateway
              name: agentgateway
              namespace: kserve-test
          rules:
            - backendRefs:
                - group: agentgateway.dev
                  kind: AgentgatewayBackend
                  name: my-model-backend
              matches:
                - path:
                    type: PathPrefix
                    value: /v1/chat/completions
              timeouts:
                backendRequest: 0s
                request: 0s
            - backendRefs:
                - group: agentgateway.dev
                  kind: AgentgatewayBackend
                  name: my-model-backend
              matches:
                - path:
                    type: PathPrefix
                    value: /v1/completions
              timeouts:
                backendRequest: 0s
                request: 0s
  template:
    containers:
      - name: main
        resources:
          limits:
            nvidia.com/gpu: 1
          requests:
            nvidia.com/gpu: 1

Step 3: Attach Token-Based Rate Limiting (Optional)​

Apply an AgentgatewayPolicy to enforce token-based rate limits on the route:

apiVersion: agentgateway.dev/v1alpha1
kind: AgentgatewayPolicy
metadata:
  name: token-ratelimit
  namespace: kserve-test
spec:
  targetRefs:
    - group: gateway.networking.k8s.io
      kind: HTTPRoute
      name: my-model-llminferenceservice-route
  traffic:
    rateLimit:
      local:
        - tokens: 10000
          unit: Hours

Configure $GATEWAY_URL​

Check if your Gateway has an external IP address assigned:

kubectl get svc -n kserve-test -l gateway.networking.k8s.io/gateway-name=agentgateway

If the EXTERNAL-IP shows an actual IP address (not <pending>):

export GATEWAY_URL="http://$(kubectl get gateway -n kserve-test agentgateway \
  -o jsonpath='{.status.addresses[0].value}')"

Testing the Integration​

Send a test request:

curl -s "$GATEWAY_URL/v1/chat/completions" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen2.5-0.5B-Instruct",
    "messages": [{"role": "user", "content": "Hello"}]
  }' | jq .

Verify LLM-Aware Processing​

Check the AgentGateway logs to confirm the LLM pipeline is active:

kubectl logs -n kserve-test deploy/agentgateway --tail=10

With AgentgatewayBackend, you should see GenAI fields in the log:

route=my-model-llminferenceservice-route
http.status=200
protocol=llm
gen_ai.operation.name=chat
gen_ai.request.model=Qwen/Qwen2.5-0.5B-Instruct
gen_ai.response.model=Qwen/Qwen2.5-0.5B-Instruct
gen_ai.usage.input_tokens=12
gen_ai.usage.output_tokens=15

Without AgentgatewayBackend (plain Service or InferencePool backendRef), the same request would only show:

protocol=http
http.status=200

How It Works​

The default LLMInferenceServiceConfig route template generates HTTPRoutes with backendRef pointing to InferencePool (kind: InferencePool, group: inference.networking.x-k8s.io).

By overriding spec.router.route.http — either directly on the LLMInferenceService or via a reusable LLMInferenceServiceConfig — you change the backendRef to AgentgatewayBackend (kind: AgentgatewayBackend, group: agentgateway.dev). This leverages Gateway API's support for arbitrary backendRef kinds (GEP-1742).

AgentGateway recognizes AgentgatewayBackend references and activates its LLM pipeline: parsing OpenAI request/response payloads, extracting token usage, emitting GenAI telemetry, and enforcing token-based policies.

note

This approach is generic. Any gateway that supports custom backendRef kinds via Gateway API can use the same LLMInferenceServiceConfig override mechanism. Replace AgentgatewayBackend with your gateway's backend CRD.

Next Steps​