Building a Secure & Governed LLM Application for Fanbase using Semantic Kernel & Azure

Semantic Kernel filters are the first line of defense, acting as middleware to intercept and act upon LLM operations. They provide a structured way for us to enforce policies and inject custom logic directly into the AI execution pipeline.

Core Concepts

Filters allow us to wrap logic around function invocations and prompt rendering. This modular approach keeps our core business logic clean while centralizing security, policy, and enrichment tasks.

Official Documentation: Concepts: Filters (Microsoft Learn): The definitive source for understanding how filters work in both .NET and Python.

Types of Filters & Key Use Cases

// Pseudocode for a prompt rendering filter with PII redaction

public static string PromptRenderingFilter(string prompt)
{
    // Detect and redact PII before sending to LLM
    string redactedPrompt = PiiDetectionService.Redact(prompt);
    return redactedPrompt;
}

While SK filters provide the mechanism for control, Azure AI Content Safety provides a powerful, pre-built service for detecting and filtering harmful content. This is a critical layer for protecting our brand and our users.

Key Capabilities

• Multi-Modal Harm Detection: Analyzes both text and images to detect content related to hate, violence, self-harm, and sexual content, assigning a severity score for nuanced filtering.
• Advanced Threat Protection: Includes specific features to detect prompt injection attacks (jailbreaking) and to identify if the model is generating content that is ungrounded or hallucinated (a major risk in RAG scenarios).

Architectural Pattern: The Safety Sandwich

We cannot secure what we cannot see. Semantic Kernel's built-in support for telemetry provides the deep visibility needed for auditing, debugging, and anomaly detection.

Core Concepts

Semantic Kernel uses the industry-standard OpenTelemetry framework to export rich diagnostic data. This data can be sent directly to Azure Application Insights for powerful analysis and visualization.

Official Documentation & Guides:

• Inspect Telemetry with Application Insights (Microsoft Learn): A step-by-step tutorial on configuring Semantic Kernel to send telemetry to Azure.
• Blog: Unlock the Power of Telemetry in Semantic Kernel SDK (Microsoft Dev Blogs): An excellent post explaining the "why" and "how" of telemetry, covering traces, metrics, and logs.

What to Monitor

• Traces: Visualize the entire lifecycle of a request as it flows through different filters, plugins, and AI model calls. This is essential for debugging complex chains and identifying performance bottlenecks.
• Metrics: Track key performance indicators (KPIs) like LLM response times, token consumption (for cost management), function error rates, and the number of times content safety filters are triggered.
• Logs: Capture structured logs that include the rendered prompts, raw model responses, and any exceptions. This creates an invaluable audit trail for compliance and security investigations.

To ensure full control over data and maintain compliance, we should host our LLMs within our own Azure tenancy. Semantic Kernel seamlessly integrates with Azure's secure LLM offerings, meaning we do not need to send data to external providers.

Azure Hosting Options

• Azure OpenAI Service: Provides access to powerful models like GPT-4 with the enterprise-grade security and privacy guarantees of Azure. Our data is not used for training external models.
• Azure AI Inference: Deploy and manage a curated catalog of powerful open-source models (e.g., from Mistral, Meta) as fully managed serverless API endpoints within our environment.

Security Benefits

• Network Isolation: Use Azure Virtual Networks (VNets) and Private Endpoints to ensure our models are not accessible from the public internet.
• Unified Governance: Manage access and permissions using Microsoft Entra ID (formerly Azure AD) and Role-Based Access Control (RBAC), providing a single control plane for our entire application.

While Azure provides a secure and tightly integrated ecosystem, Semantic Kernel's greatest strength is its vendor-agnostic architecture. This allows us to avoid vendor lock-in and strategically use the best model for any given task, ensuring maximum performance and cost-effectiveness.

Connecting to Alternative Providers

Semantic Kernel's extensibility makes it simple to connect to a wide range of model providers beyond Azure. This is primarily achieved through its support for OpenAI-compatible API endpoints and dedicated connectors.

Self-Hosting Models on a Cloud VM

For ultimate control over security, cost, and customization, we can host open-source models on our own cloud infrastructure (e.g., an Azure VM with GPU support).

• Maximum Control & Privacy: The model and data never leave our virtual network, providing the highest level of data sovereignty.
• Cost Management: For high-throughput scenarios, self-hosting can be more cost-effective than pay-per-token APIs, as we pay for the underlying compute infrastructure.
• Customization: Enables the use of highly specialized or fine-tuned open-source models tailored to specific Fanbase domains.

Use Cases for Local Models in a Cloud Platform

• Cost-Free Development & Testing: The primary use case. Developers can iterate on new features, build complex agentic workflows, and write unit tests without incurring any API costs or dealing with network latency.
• Offline Tooling: Enables the creation of powerful developer tools or CLI applications that can function without an internet connection.
• Specialized, Low-Latency Tasks: For small, highly-specialized models (e.g., a model fine-tuned for code formatting or data classification), running them locally can sometimes provide lower latency than a round-trip to a cloud endpoint.

Technical Guides & Tutorials

• Self-Hosting: Guides on using tools like vLLM or Ollama to serve models via an API are widely available in their respective documentation.
• OpenRouter: Community discussion on connecting Semantic Kernel to OpenRouter.

What is a Token?

Think of a token as the basic currency for LLMs. A token isn't exactly one word; it's a piece of a word. On average, 100 tokens represent about 75 words.

All interactions with an LLM—both the questions we ask (input) and the answers we receive (output)—are measured in tokens. The total cost of an operation is the sum of the input token cost and the output token cost.

1. Azure OpenAI: The Main Pricing Models

2. Azure AI Content Safety Costs

This is an optional but highly recommended service for protecting our brand. It checks both user input and AI output for harmful content.

• Free Tier: Includes 5,000 text checks per month at no cost.
• Paid Tier: After the free tier, it costs approximately $0.75 for every 1,000 text checks. A "text check" covers up to 1,000 characters (roughly a paragraph).

For most applications, this cost is minimal. For example, processing 50,000 user prompts would cost around $37.50 per month.

3. Third-Party and Open-Source Model Costs

Semantic Kernel's flexibility allows us to use models from other providers. Costs here are determined by the provider itself.

• Router Services (e.g., OpenRouter): We pay the router's rate for each model, which is often very close to the direct API cost. This offers great flexibility.
• Direct APIs (e.g., Anthropic, OpenAI): Models like Anthropic's Claude 3 family can be accessed via platforms like Amazon Bedrock. They often have competitive pricing and can sometimes be cheaper than Azure equivalents for specific tasks.
• Self-Hosting: Hosting an open-source model on a cloud VM means we pay for the underlying infrastructure (the virtual machine, GPU, storage, and data transfer). This can be cost-effective for extremely high-volume, specialized tasks but carries a higher operational overhead.

4. Summary & Example Scenarios

The following table compares different models based on a "2,000-token exchange" (e.g., a 750-word input and a 750-word output). This provides a standardized way to compare real-world costs.

🔧 Strategic Recommendations

• Tier Our Models: Use the most cost-effective model that can accomplish the task. Use GPT-4o mini for simple tasks and reserve more powerful models like GPT-4o for complex reasoning.
• Monitor Everything: Use the observability tools in Semantic Kernel and Azure to track token consumption in real-time. This helps identify costly operations and opportunities for optimization.
• Start with PAYG, Scale with PTUs: Begin with the flexible pay-as-you-go model and collect data. If usage becomes high and predictable, perform a cost analysis to see if switching to provisioned throughput will save money.
• Leverage the Ecosystem: Don't hesitate to connect to third-party models if they offer better performance or pricing for a specific need. Semantic Kernel makes this easy.

Our development strategy is designed for a low-risk, high-value rollout. We will introduce AI capabilities in carefully managed phases, building directly on top of our existing platform. This iterative approach ensures stability, allows us to gather feedback, and provides immediate value to both our internal teams and end-users without disruption.

A New Layer, Not a New Platform

🧩 A key advantage of this approach is that the AI acts as an intelligent interface over our current systems. We are exposing our existing, robust codebase—handling reservations, orders, club details, and reporting—as secure "plugins" for the AI. Clubs don't need to learn a new platform or migrate data. The AI simply provides a new way to access live data and trigger familiar workflows.

How We Create AI Plugins from Our Existing API

The core of our technical approach involves "wrapping" our existing, secure API endpoints into plugins that Semantic Kernel can understand and use. This process is deliberate and requires careful planning.

1. 📄 OpenAPI Specification: The foundation of a plugin is a well-defined OpenAPI (formerly Swagger) specification file. We will generate or author these files for each API endpoint we want to expose. The descriptions within this file are critical—they are what the LLM reads to understand what a function does, what parameters it needs, and when it should be used.
2. ⚡ Automatic Plugin Creation: Semantic Kernel can ingest an OpenAPI specification and automatically generate a native plugin from it. This allows us to rapidly convert our documented API endpoints into callable functions for the AI.
3. 🛡️ Planning & Curation: We will not expose our entire API at once. A significant planning effort is required to decide which endpoints offer the most value and can be exposed safely. We will start with simple, read-only endpoints (e.g., getClubDetails, listRecentOrders) before moving to more complex or write-based operations. Each endpoint will be reviewed for its suitability as a discrete "tool" for the AI.

Proposed Phased Rollout & Future Considerations

Each phase progressively expands the AI's capabilities. Estimated Effort for each phase is additive—the time for each phase includes the work of all previous phases. For example, Phase 2's estimate includes the time for Phase 1, and so on. (E.g., "6 weeks" for Phase 2 means 2 weeks for Phase 1 plus 4 more weeks for Phase 2.)

Project Timeline Overview