That's not hypothetical. Supabase runs 1.5 million databases this way. Most of them power full production applications with auth, APIs, background jobs, and access control. The application server? There isn't one. It's PostgreSQL all the way down. And the extension ecosystem has quietly matured to the point where this approach isn't a prototype or a hack. It's a real architecture, and it happens to be the one AI agents are best equipped to work with.

The Extension Stack

The reference point here is supabase/postgres, which packages a curated set of extensions into a single Postgres distribution. You don't have to use Supabase's hosted platform to benefit from this. The image is open source, and the extensions work on any Postgres instance. But looking at what they've bundled tells you exactly what a "Postgres as backend" stack looks like in practice.

pg_graphql turns your database schema into a GraphQL API automatically. You define your tables, your foreign keys, your constraints, and pg_graphql reflects them as a fully queryable GraphQL endpoint. No resolvers, no schema stitching, no code generation step. Add a column, and the API updates. It handles filtering, pagination, ordering, and relationships out of the box. If you've ever spent three days writing boilerplate CRUD resolvers in a Node.js backend, this will feel like cheating. And because the entire schema is defined in SQL, an agent can create tables, add relationships, and immediately test the resulting API through the same database connection.

For REST, there's PostgREST, which does the same thing over HTTP. Define a table, get a REST API. It respects your Row Level Security policies, so authorization is handled at the database layer. The API is surprisingly complete: you get bulk inserts, upserts, resource embedding for joins, and content negotiation. It's been in production at serious scale for years.

pg_cron gives you cron scheduling inside PostgreSQL. Schedule a function to run every night at 2 AM, every five minutes, every Monday. The syntax is standard cron. The jobs execute as SQL, which means they can do anything your database can do: aggregate data, clean up old records, refresh materialized views, call external services through pg_net. No separate scheduler service, no Lambda functions, no Kubernetes CronJobs. An agent can set up a scheduled job with a single SELECT cron.schedule() call, verify it's registered, and monitor its execution, all without leaving SQL.

pgmq is a message queue built as a Postgres extension, originally created by Tembo. It gives you exactly what you'd use Redis or RabbitMQ for: publish messages, consume them, handle visibility timeouts, dead letter queues. The difference is that your queue lives in the same database as your data, participates in the same transactions, and doesn't require a separate infrastructure component. For most applications that aren't processing millions of messages per second, this is more than enough.

pg_net lets Postgres make HTTP requests. This is the piece that closes the loop. Your database can now call external APIs, send webhooks, trigger serverless functions. Combine it with pg_cron and you have a fully autonomous system that can poll APIs, process data, and push results without any application code running anywhere.

Auth Without an Auth Service

The most underappreciated part of the Postgres-as-backend approach is how naturally it handles authorization. Row Level Security (RLS) is a built-in Postgres feature, not an extension, that lets you define access policies directly on tables. When you write CREATE POLICY user_data ON profiles USING (auth.uid() = user_id), you're telling Postgres: no matter how someone queries this table, they only see their own rows. This isn't application-level filtering that someone might forget to add. It's enforced at the storage engine level.

Pair RLS with pgjwt, which can generate and verify JSON Web Tokens inside PostgreSQL, and you have a complete auth flow. The JWT contains the user's role and ID, Postgres verifies it on every request, and RLS policies use that identity to filter data. PostgREST and pg_graphql both pass the JWT through, so your API layer is stateless and your authorization logic lives exactly where your data lives.

This is the kind of architecture that would normally require an Express server, a Passport.js configuration, a middleware chain, and a few hundred lines of authorization code scattered across your route handlers. Here it's a few SQL statements. Statements that an agent can write, test, and iterate on with immediate feedback about whether the policy does what it should.

Why This Is an Agent-Native Architecture

The deeper you look at this stack, the more you realize it's not just convenient for AI agents. It's structurally ideal.

SQL has an exceptionally tight feedback loop. An agent writes a query, runs it, and immediately knows if it worked. The error messages are specific and actionable: "column 'user_naem' does not exist" tells the agent exactly what to fix. Compare this to debugging a REST API integration where the failure mode might be a silent 200 response with unexpected data. An agent working with SQL can self-correct in a single retry. It reads the error, fixes the typo or adjusts the join, and tries again. Most queries converge on the correct result within one or two iterations.

Then there's EXPLAIN ANALYZE. An agent can not only write a query but immediately evaluate its performance, see the query plan, identify sequential scans, and rewrite for efficiency. This kind of introspection is rare in programming interfaces. You don't get an equivalent of "here's exactly how your code will execute and how long each step takes" in most languages. With Postgres, it's one command away.

The extension ecosystem multiplies this advantage. An agent that can write SQL can also create RLS policies, schedule cron jobs, enqueue messages, define GraphQL schemas, and set up webhooks. All through the same interface. All with the same feedback loop. All correctable in the same way. You're not asking the agent to context-switch between a Terraform config, a YAML deployment manifest, a JavaScript middleware function, and a SQL migration. It's SQL the whole way through.

This matters because the bottleneck with AI agents isn't intelligence, it's interface friction. Every time an agent has to switch between tools, formats, or paradigms, error rates go up. A Postgres-native backend compresses the entire surface area into one language that agents already handle well.

What This Actually Looks Like

Picture a SaaS application. Users sign up, manage projects, get daily email digests, and can query their data through a dashboard. In a traditional stack, you'd need an API server (Express, FastAPI, Rails), a background job processor (Sidekiq, Celery, Bull), a message queue (Redis, SQS), an auth service (Auth0, Clerk, or hand-rolled), and a cron scheduler (CloudWatch Events, Kubernetes CronJobs).

With the Postgres extension stack, your migration files are your backend. You write CREATE TABLE for your schema. CREATE POLICY for your auth rules. SELECT cron.schedule() for your daily digest job. A pgmq queue for async processing. pg_graphql for your dashboard API. pg_net to call your email provider. The "application code" is SQL, and it lives in version-controlled migration files.

Now point an AI agent at that database. It can scaffold the schema, write the security policies, set up the background jobs, and test everything, all in one session, all in one language. Try doing that across Express, Redis, Auth0, and CloudWatch.

Is this right for every application? No. If you're building a real-time multiplayer game or processing video streams, you need application servers. But for the vast majority of web applications, the ones that are fundamentally CRUD with some business logic and background processing, the Postgres extension ecosystem covers more ground than most developers realize.

Here is a startup idea that kills the integration treadmill and builds for the agentic era at the same time.

The Integration Tax

Every workflow platform eventually becomes an integration company. n8n, Zapier, Make, Pipedream, Tray.io, they all spend enormous engineering effort building and maintaining connectors. Each connector is a custom adapter: someone reads the API docs for Salesforce or HubSpot or Stripe, writes the mapping code, handles auth, pagination, error states, rate limiting, and then maintains it forever. When Stripe changes a webhook payload or HubSpot deprecates an endpoint, that connector needs an update. Multiply that by thousands and you have a permanent tax on the business.

This made sense in a world where the consumer of these workflows was a human dragging boxes on a canvas. Humans need pre-built nodes with nice dropdown menus and labeled fields. But agents don't need any of that. An agent can read a spec.

Specs Are the New Connectors

MCP (Model Context Protocol) gives you a standardized way to describe tools that AI agents can call. OpenAPI specs already describe REST APIs in machine-readable format. GraphQL has introspection built into the protocol. Every serious API already publishes at least one of these.

Instead of building a Stripe connector, you point the engine at Stripe's OpenAPI spec. Instead of maintaining a Slack integration, you register Slack's MCP server. The engine reads the spec, understands the available operations, their parameters, their auth requirements, and generates step configurations automatically. You get instant support for any service that publishes a spec, which is effectively every service worth integrating with.

The step configuration layer works like this: given a spec (MCP, OpenAPI, or GraphQL schema), the engine extracts available operations, input/output schemas, and authentication flows. It builds typed step definitions that can be used in flows, both by agents constructing flows dynamically and by humans editing them in a UI. When the upstream API changes and publishes an updated spec, the step definitions update automatically. No connector code to maintain. No integration team racing to keep up.

Built for Enterprise Scale

n8n runs as a single Node.js process. You can add workers, bolt on Redis, configure queue mode, but at its core you're scaling an application server. For a team running a few hundred workflows, that's fine. For an enterprise running tens of thousands of concurrent flows across dozens of teams, it's a ceiling you'll hit hard.

The engine should be Kubernetes-native from day one. Each step in a flow runs as its own container. The orchestrator schedules steps across the cluster, scales horizontally without configuration, retries failed steps independently, and handles massive parallelism because that's just what Kubernetes does. There's no single bottleneck, no shared process, no "upgrade to our enterprise tier for more workers." You scale by adding nodes to your cluster, the same way you scale everything else in your infrastructure.

This matters for enterprises specifically because their workflows aren't simple "if this then that" chains. They're complex graphs with fan-out, fan-in, conditional branching, human approval gates, and SLA requirements. Running these on a single process with a queue is like running your CI/CD pipeline on a cron job. It technically works until it doesn't.

Agentic First, Not Agentic Bolted On

Here's the part that makes this more than just "n8n with better infrastructure." The engine is designed from the ground up for flows where agents are participants, not just triggers.

A traditional workflow is fully deterministic. Step A calls an API, passes the result to Step B, which transforms it, passes it to Step C. Every input and output is known at design time. These workflows are valuable and this engine supports them through the spec-driven step system described above. An MCP client or HTTP client executes the call, the output schema is known, validation is straightforward.

But the interesting workflows combine deterministic and non-deterministic steps. An agent step takes context from previous steps, reasons about it, decides what to do, and produces output that might vary every time. Maybe an agent reads a customer support ticket, decides which internal team should handle it, drafts a response, and flags whether the issue needs escalation. The reasoning is non-deterministic. The actions it takes (filing a Jira ticket, sending a Slack message, updating a CRM record) are deterministic steps driven by specs.

The engine lets you compose these freely. You can follow an agent step with a validation step that checks the output against a schema or a set of business rules before the flow continues. You can have an agent step that dynamically constructs a sub-flow by selecting which spec-driven steps to execute based on its reasoning. You can insert human review gates after agent steps for high-stakes decisions. The flow graph becomes a mix of reliable, typed, spec-driven operations and flexible, reasoning-capable agent nodes, with explicit boundaries between them.

This is what enterprises actually need. Not a chatbot that can call tools, but a governed system where agent behavior is observable, auditable, and constrained by the flow structure around it. The deterministic steps give you predictability and compliance. The agent steps give you flexibility and intelligence. The validation steps give you guardrails.

Why Agents Should Build the Flows

The last piece is that agents aren't just participants in flows, they're also builders. Because every integration is spec-driven, an agent can browse available specs, understand what operations are possible, and assemble a flow from them. A product manager describes what they need in natural language: "When a deal closes in HubSpot over $50k, create a Jira onboarding epic, notify the CS team in Slack, and schedule a kickoff in Google Calendar." The agent reads the relevant specs, constructs the flow graph with the right steps, parameters, and connections, and presents it for review.

This works because the specs are self-describing. The agent doesn't need to know how your custom n8n node for HubSpot works. It reads HubSpot's OpenAPI spec, finds the deal webhook and the relevant endpoints, and wires them up. The same agent can build flows against any service you've registered, even internal APIs, as long as they publish a spec.

For enterprises with hundreds of internal services, this is transformative. Instead of waiting for an integration team to build a connector for your internal billing API, you publish an OpenAPI spec for it and every agent in the system can immediately build flows against it.

The Opportunity

The workflow automation market is large and growing, but every incumbent is anchored to the connector model. They can't easily abandon it because their entire value proposition is "we integrate with everything, pre-built." Switching to a spec-driven approach would mean admitting that the thing they spent years building is now a liability.

A new entrant doesn't carry that baggage. You start with zero connectors and infinite integrations. You ship a platform that scales like infrastructure, governs agent behavior like an enterprise demands, and lets teams build workflows by describing what they need rather than dragging boxes around.

The wedge is any enterprise already deep in Kubernetes that has outgrown n8n or hit the limits of Zapier's enterprise tier. They already think in containers and specs. They just need a workflow engine that thinks the same way.

Build it.

Auth0 sold for \(6.5B. Okta is worth \)15B+. CyberArk, Delinea, BeyondTrust, all printing money from managing who gets access to what. And yet none of them were built for what's actually happening right now.

Your Slack bot spins up a Claude agent that calls your billing API, which triggers a background job that queries production, which fires a webhook to Stripe, all because a user pressed a button twenty minutes ago and has since gone to lunch. Who authorized that database query? With what permissions? For how long? Can you prove it to an auditor?

Be honest. The answer at most companies is that there's probably a long-lived API key in an environment variable somewhere. It has way more permissions than it needs. Nobody remembers who created it. It never expires.

That's the startup.

Human Auth is Cooked (in a Good Way)

Here's something that was controversial two years ago but is obvious now. Traditional human authentication is a commodity. Login with email, OAuth, TOTP, passkeys, magic links. Solved problems with a dozen good implementations. In the age of AI-assisted development, spinning up solid human auth takes hours not months. Claude Code can scaffold a complete auth integration with social login, MFA, and session management faster than you can read Auth0's docs.

The moat in auth is no longer "we make login work." The moat is everything that happens after login, and everything that happens without a login at all.

Look at how identity actually works at a modern company. Humans get proper auth. SSO, MFA, RBAC, audit trails. Well solved. Services get environment variables. An API key minted six months ago by someone who may have left the company. Permissions? Whatever the key has. Expiry? Never. AI agents get the worst deal of all. They inherit the human's complete credentials. Your AI assistant that only needs to read today's calendar can delete every event you've ever created. No scoping, no time-bounding, no containment.

The idea here is a single primitive that covers all three. Every access event, whether it's a human logging in, a service calling an API, or an agent performing a task, goes through the same thing: an Access Grant. Time-bound, scope-limited, auditable. Who wants access, what do they want to touch, for how long, who approved it, and what happened during the grant. That's the whole abstraction. HashiCorp Vault does credential minting. OPA does policy evaluation. Teleport does infrastructure access. Nobody unifies these into a single grant that works the same way for humans, agents, and services.

Auth is the Best Guardrail You Have

Here's the conceptual leap that makes this more than just another IAM startup. Auth and agent guardrails are the same problem viewed from different angles.

Everyone is building guardrails at the prompt layer. Don't say bad things, don't hallucinate, stay in character. But the most effective guardrail is the simplest one: if the agent doesn't have permission to do something, it can't do it. No prompt injection, no jailbreak, no cleverness overrides the fact that the credential simply doesn't exist yet. You constrain agent behavior at the infrastructure level, which is deterministic, instead of the language model level, which is fundamentally unreliable. The agent can say whatever it wants about what it intends to do. It can only do what the access layer allows.

When an agent hits a permission boundary, the system doesn't just return a 403 and kill the flow. It initiates an access request. This is what I'd call Human-in-the-Loop for Auth. Same pattern as "human-in-the-loop" for AI decisions, but for permissions. The agent needs to access customer PII? Instead of either blocking it or giving it a permanent key, the system pings a human approver in Slack. "Agent billing-reconciler wants read access to prod.customers for 5 minutes, triggered by monthly-reconcile, approve?" Engineer approves, a scoped credential is minted, the agent does its work, the credential dies, everything is logged.

The agent is contained, not killed. The work continues, just with appropriate oversight. This is radically different from today where the agent either has permanent access or no access at all.

MCP, Tool-Level Policies, and the Credential Gateway

Think about MCP for a second. Model Context Protocol is essentially an agent tool use standard. An agent connects to an MCP server and gets access to a set of tools: read a database, search files, create a ticket, send an email. Today when you connect an agent to an MCP server it gets access to all the tools on that server. There's no per-tool policy. There's no "this agent can use read-ticket but not delete-ticket." There's no "this agent can use send-email only during business hours and only to internal addresses."

Per-tool-call policy enforcement is a natural extension of this platform. Every MCP tool invocation goes through the access grant system. The policy engine evaluates whether this agent has permission to call this specific tool, with these specific parameters, right now. The answer might be auto-approve, might require human approval, might be deny. This turns the platform into the control plane for agent capabilities. Not just "can this agent access this resource" but "can this agent perform this action through this tool with these parameters at this time." That's a complete permission boundary for agentic systems.

Now layer the Credential Gateway on top. When you give an AI agent an API key, even a short-lived one, that key exists in the agent's context. It can leak in a response, get extracted via prompt injection, get logged or cached. The moment a credential touches an agent's context window your security boundary is the LLM. And LLMs are not a security boundary.

The gateway ensures agents never hold real credentials at all. They get an opaque internal token that means nothing outside your system. When the agent needs to call Stripe or GitHub or your internal API, it goes through the gateway. The gateway validates the token, checks the policy engine, swaps it for the real external credential, forwards the request, and returns the response. The agent calls gateway.yourplatform.com/stripe/charges with its internal token. The gateway translates that to api.stripe.com/charges with the actual Stripe key. The agent never knows the key exists.

This gives you credential isolation, request-level enforcement, response filtering so you can strip sensitive fields before they reach the LLM, instant revocation without rotating external keys, full observability of what every agent does across all external services, and even sandbox mode where the gateway returns mock responses instead of hitting real services. Same agent code, same auth flow, zero risk in development. Think of it as a reverse proxy, but for auth. The same way a load balancer sits between the internet and your services, the Credential Gateway sits between your agents and the outside world.

Delegated Mode, Background Mode, and Trust Escalation

There's a subtle but important distinction in agent permissions. When a user triggers an agent directly ("check my calendar and find a meeting time") the agent is operating as a delegate of that user. It makes sense for it to have the user's calendar permissions, scoped to read-only, time-bound to that interaction.

But when that same agent runs a background job at 3 AM processing data or running reports, it's not acting on behalf of anyone. It shouldn't have any user's permissions. It should have its own identity with its own policy-governed grants. Today this distinction doesn't exist. Agents either run as the user, which means too much access in background mode, or as a service account, which means permanent unscoped access in interactive mode. Delegated mode versus background mode should be a first-class concept in the platform.

And then there's trust escalation. A new agent starts with zero trust. Every access request needs human approval. After 10 approved requests of the same type the policy auto-upgrades to auto-approve. The agent earns trust, like a new employee does. If something anomalous happens, trust resets.

Where This Goes

What excites me about this idea is how many directions it can expand. The starting point is narrow: give your AI agents secure time-bound access to your database with Slack approval. One integration, immediate value, clear before and after. But it naturally grows into MCP auth, per-tool policy enforcement, agent guardrails, credential gateways, sandbox environments, compliance automation, trust scoring. Each of these could be its own feature or its own product.

The architecture can lean on strong existing infrastructure. SPIFFE and SPIRE for workload identity, where agents get cryptographic identity based on what they are, not what secret they hold. Cedar or OPA for policy evaluation. Vault or OpenBao for credential minting. Temporal for durable approval workflows. Envoy for the gateway proxy. PostgreSQL for audit. What the startup builds is the orchestration layer, the UX, and the product intelligence on top.

The market straddles IAM (Okta, Auth0, Clerk at \(20B+, focused on humans) and PAM (CyberArk, BeyondTrust at \)5B+, built for enterprise IT). Agent Access Management as a category doesn't formally exist yet. The wedge is developer teams building AI agent products who need to answer one question: how do I give my agent access to production without giving it the keys to the kingdom?

Auth0 gave every app a login page. This startup gives every agent a permission boundary. Somebody should build it.

Ideas I think are worth building. Some I'll build, some I hope you will.

These two paradigms served us well. But something happened in the last eighteen months that neither was designed for: a third kind of consumer showed up. Not a human navigating your dashboard. Not a script calling your REST endpoint with a hardcoded bearer token. Something in between, an agent, that reasons, improvises, discovers capabilities on the fly, and decides what to do next based on context it gathered three tool calls ago.

And we are, collectively, building for it all wrong.

The Two Doors (and Why They're Not Enough)

UI-first solved friction for humans. You obsess over information hierarchy, button placement, loading states. The assumption: your user has eyes, a cursor, and the ability to look at something confusing and figure it out through visual context clues.

API-first enabled programmatic access. You define schemas, version endpoints, write docs, publish SDKs. The assumption: the consumer is a developer who reads your documentation, writes integration code once, and maintains it.

Both share a hidden assumption: someone already knows what your system can do before they use it. The human read the onboarding flow. The developer read the API docs.

Agents don't do either of those things. An agent lands in your system like a new employee on day one, except this employee can read every document in the company in 200 milliseconds but can't find the bathroom unless someone labeled the door clearly.

The Third Door Is Already Wide Open

The agent interface isn't one technology. It's an entire surface area that emerged in parallel across protocols, runtimes, CLIs, and sandboxes, all converging on the same idea: software needs a machine-native entry point that isn't just "the API with a prompt wrapper."

Protocols. MCP went from an Anthropic side project in late 2024 to an industry standard governed by the Linux Foundation, adopted by OpenAI, Google, Microsoft, and AWS, pulling nearly 100 million monthly SDK downloads. Meanwhile, researchers are already pushing past it. The ANX protocol out of Hangzhou combines CLI, skills, and MCP into a unified agent-native framework, reporting 47-55% token reduction over MCP-based skill approaches alone. The protocol layer is maturing fast.

Skills and tool registries. Agents don't just call endpoints. They discover capabilities. OpenAI's updated Agents SDK introduced progressive disclosure via skills, custom instructions via AGENTS.md files, and standardized tool primitives. The pattern is the same everywhere: give agents a structured way to understand what your system can do, what parameters it expects, and when to use each capability. Skills are the new API docs, except the reader isn't human.

CLIs. The command line became the agent's native habitat almost by accident. Claude Code, Codex CLI, Gemini CLI, Aider, Goose: these aren't IDE plugins. They're agent runtimes that happen to let humans watch. The CLI is structured, scriptable, parseable, and composable. It turns out that everything we built for power users over the last 30 years is exactly what agents need. Consistent --help output, JSON response formats, composable subcommands: this is agent UX.

Sandboxes. When agents write and execute code, they need somewhere safe to do it. This went from a niche concern to a full market in under a year. E2B, Daytona, Blaxel, Cloudflare Dynamic Workers, Together Code Sandbox, OpenAI's sandbox agents: all shipping microVM or isolate-based infrastructure where agents get filesystems, shell access, and mounted data rooms. Cloudflare's approach is particularly telling. Their MCP server exposes the entire Cloudflare API through just two tools, search and execute, because agents write code against a typed API inside a sandbox instead of navigating hundreds of individual tool definitions.

The infrastructure exists. The question is whether your product is legible to the things using it.

What Agents Actually Need

Most teams think "building for agents" means exposing their existing API with a good system prompt. That's like thinking "building for mobile" meant making the desktop site smaller.

Discoverability over documentation. Agents don't browse your developer portal. They receive a list of tools with names, descriptions, and parameter schemas, then decide what to call based on how well those descriptions match the user's intent. Your tool descriptions aren't metadata. They're your entire UX. If your tool is called batchProcessResourceModification with a description that says "Processes resources," you've built the agent equivalent of a button labeled "Click Here" that navigates to a 404.

Composability over completeness. APIs mirror your internal domain model: Users, Invoices, clean RESTful nouns. Agents don't care about your domain model. They care about tasks. Don't expose your object model. Expose your capability surface. Think verbs, not nouns.

Structured errors over status codes. When your API returns 422, a developer reads the message, checks the docs, fixes the request. An agent needs to understand exactly what went wrong and what to try differently, in a structured format, not prose. A suggestion field in your error response isn't for humans. It's a hint that saves a retry and a reasoning cycle.

Guardrails as a feature. Agents will try things humans never would, not out of malice, but out of optimization. An agent asked to "clean up staging" might decide the most efficient path is to delete everything and re-provision. Technically correct. Operationally catastrophic. OpenAI's sandbox architecture makes this explicit: the harness (approvals, audit logs, recovery) never trusts the sandbox (file ops, code execution). Your agent interface needs the same separation. Define what the agent can do. Define what it must ask about first. Make the boundary machine-readable.

Context windows are the new rate limits. Every tool description, every schema, every response payload competes for space in the agent's context window. If your MCP server exposes 200 tools with verbose descriptions, you've burned thousands of tokens before the agent does anything useful. Fewer, smarter tools beat a sprawling catalog every time.

SELECT * FROM orders WHERE user_id = $1

And this one:

DELETE FROM invoices WHERE id = \(2 AND user_id = \)1

And somewhere, deep in the codebase, someone forgot the AND user_id = $1 part. Maybe it was a new hire. Maybe it was you at 11pm before a deadline. The query works fine in testing because your test user happens to own the right data. It ships. And three months later you find out that any logged-in user could read every invoice in the system.

This is not a hypothetical. This is the #1 cause of data leaks in multi-user applications. You're relying on every single query, in every single endpoint, written by every single developer, to always remember to filter by the right user. That's not a security model. That's a prayer.

PostgreSQL has a built-in feature that makes this entire class of bug impossible. It's called Row-Level Security (RLS), and after reading this post, you'll wonder why you ever did it any other way.

Now Think About Who's Actually Writing Your Queries

AI is generating a growing share of our backend code. Copilot autocompletes queries. Claude writes data access layers. Cursor refactors endpoints. The code works, tests pass, it ships.

But here's the reality: AI-generated queries are often complex. Joins across five tables, nested CTEs, multi-branch subqueries. The kind of SQL where a missing WHERE clause doesn't jump out at you during review. And the volume keeps growing. Reviewing every generated query for authorization correctness is a losing game.

And it's not just code generation. More and more teams are connecting AI agents directly to their databases, letting them build and run queries at runtime based on user input. That trend is only going to accelerate. These queries don't exist in your codebase. You can't review them in a PR. They're constructed on the fly.

Authorization is the one domain where "almost always correct" isn't good enough. One missed filter in one query is a data leak.

RLS takes this off your plate entirely. The database enforces access rules regardless of what the query looks like or where it came from. Define the rules once, and every query gets scoped automatically. Whether it was written by a developer, generated by Copilot, or constructed by an agent at runtime. It's one of those rare things you can genuinely delegate and stop thinking about.

-- At the start of every agent turn
SELECT set_config('app.current_user_id', '<uuid-from-jwt>', true);
SET ROLE authenticated;

-- Now let the agent run whatever it generates
-- RLS makes it impossible to access unauthorized data

The agent operates inside a sandbox it can't escape, not because you told it nicely, but because the database won't let it. That's a fundamentally different security posture than hoping your prompt engineering holds up.

How It Actually Works

RLS has three building blocks: Roles, Grants, and Policies. They work as layers, each one narrowing what's possible.

Roles: Who Are You?

PostgreSQL doesn't separate "users" and "groups." It has one concept: the role. A role can log in (acts like a user), or it can be a container that other roles inherit from (acts like a group).

In practice, every RLS setup lands on three roles:

anon is the unauthenticated visitor. Almost no access. Exists so that requests without a valid token fail loudly instead of silently returning empty results.

authenticated is any logged-in user (or any agent acting on behalf of a logged-in user). Can read and write tables, but every operation passes through RLS policies first.

service_role bypasses RLS entirely. Reserved for migrations, background jobs, and admin scripts. Never exposed to end users. Never given to an agent.

CREATE ROLE anon NOLOGIN;
CREATE ROLE authenticated NOLOGIN;
CREATE ROLE service_role NOLOGIN BYPASSRLS;

CREATE ROLE app_api LOGIN PASSWORD '...' NOINHERIT;
GRANT anon, authenticated, service_role TO app_api;

That NOINHERIT is critical. Without it, app_api automatically inherits service_role's bypass power the moment it connects. With NOINHERIT, your app must explicitly SET ROLE authenticated per request. This is the gate that makes the whole model work.

Grants: What Can You Do?

Grants control which operations a role can perform on which objects. Can this role SELECT from this table at all? Can it INSERT?

GRANT USAGE ON SCHEMA app TO authenticated;
GRANT SELECT, INSERT, UPDATE, DELETE ON app.documents TO authenticated;
-- No grants to anon = anon can't touch the table, period

If a role doesn't have SELECT on a table, RLS never even comes into play. Grants are the outer wall. RLS is the room-by-room access control inside the building.

Policies: Which Rows Can You See?

This is the main event. A policy is a rule attached to a table that Postgres injects into every query as an invisible WHERE clause.

ALTER TABLE documents ENABLE ROW LEVEL SECURITY;
ALTER TABLE documents FORCE ROW LEVEL SECURITY;

CREATE POLICY users_own_documents ON documents
  FOR SELECT TO authenticated
  USING (owner_id = app.current_user_id());

Now when any authenticated role runs SELECT * FROM documents, Postgres silently rewrites it to SELECT * FROM documents WHERE owner_id = '<current-user>'. Every time. Whether the query came from your application, a raw SQL console, or an AI agent that decided to get creative.

The FORCE part matters. Without it, the table owner bypasses RLS by default. Add it, or your migration user can see everything.

USING vs WITH CHECK: The Two Guards

Every policy has up to two expressions that do different jobs.

USING filters which existing rows you can see or target. It's your read filter. Applied to SELECT, and to the "before" state of UPDATE and DELETE.

WITH CHECK validates new or modified rows after a write. It catches sneaky mutations.

Why do you need both? Without WITH CHECK, someone (or some agent) could do:

INSERT INTO documents (owner_id, title)
VALUES ('someone-elses-id', 'Gotcha');

USING wouldn't catch it because there's no existing row to filter. WITH CHECK rejects the write because the new row doesn't belong to the current user. For UPDATE, both work together: USING controls which rows you can touch, WITH CHECK ensures you can't mutate a row into something you shouldn't own.

Permissive vs Restrictive: Building Walls and Doors

Postgres has two flavors of policy, and they combine with different logic.

Permissive (the default) policies are OR together. If any one of them says yes, the row is visible. These are doors. Each one is a way in.

Restrictive policies are AND on top of the permissive result. They can only narrow access, never widen it. These are walls. They hold no matter how many doors you add.

This is where multi-tenancy gets elegant:

-- WALL: tenant isolation. Always enforced.
CREATE POLICY tenant_wall ON documents AS RESTRICTIVE
  FOR ALL TO authenticated
  USING (tenant_id = app.current_tenant_id())
  WITH CHECK (tenant_id = app.current_tenant_id());

-- DOOR: see your own docs within your tenant
CREATE POLICY own_docs ON documents
  FOR SELECT USING (owner_id = app.current_user_id());

-- DOOR: see public docs within your tenant
CREATE POLICY public_docs ON documents
  FOR SELECT USING (is_public = true);

The effective filter: (own_docs OR public_docs) AND tenant_wall

You can add ten more permissive doors and the tenant wall still holds. A developer (or an agent) can never accidentally create a path that leaks data across tenants. The restrictive layer is a structural guarantee, not a convention.

One thing to watch: if you only have restrictive policies and zero permissive ones, nothing is visible. Restrictive narrows the permissive set. An empty set AND with anything is still empty. You always need at least one door.

The Plumbing

Policies reference app.current_user_id(), but how does Postgres know who the current user is? Through session variables that your application sets at the start of each request.

CREATE FUNCTION app.current_user_id() RETURNS uuid
LANGUAGE sql STABLE AS $$
  SELECT nullif(current_setting('app.current_user_id', true), '')::uuid;
$$;

Your backend (or your agent runtime) sets context before running any queries:

BEGIN;
  SELECT set_config('app.current_user_id', '<uuid-from-jwt>', true);
  SET ROLE authenticated;

  -- queries go here, RLS is active

COMMIT;  -- settings revert, safe for connection pooling

The true in set_config means "transaction-local." When the transaction ends, the setting vanishes. No state leaks between requests, even with connection pooling. This is what makes it safe to share a single Postgres connection pool across thousands of users (or thousands of agent sessions).

Teams and Shared Access

Real apps aren't just "users own rows." You need teams, organizations, shared workspaces. The pattern is clean:

CREATE FUNCTION app.my_team_ids() RETURNS SETOF uuid
LANGUAGE sql STABLE SECURITY DEFINER SET search_path = public AS $$
  SELECT team_id FROM team_members
  WHERE user_id = app.current_user_id();
$$;

CREATE POLICY team_access ON projects
  FOR SELECT TO authenticated
  USING (team_id IN (SELECT app.my_team_ids()));

SECURITY DEFINER means the function runs as the function owner, bypassing RLS on the team_members table itself. This avoids a circular dependency. Always set search_path explicitly on SECURITY DEFINER functions to block injection.

Want role-based permissions? Layer it. Viewers can read. Members can write. Admins can delete. Same table, different policies, each checking the user's team role.

Why RLS Matters More Now Than Ever

We're entering an era where the code accessing your database isn't fully written by humans anymore. Agents generate queries. Copilots autocomplete SQL. Low-code platforms abstract away the data layer. The surface area for "someone forgot the WHERE clause" is exploding.

The traditional approach of scattering authorization checks across application code was already fragile. In an agentic world, it's untenable. You can't review dynamically generated queries. You can't unit test every possible natural language input mapped to SQL. You can't guarantee that an LLM will always respect a system prompt that says "only access the current user's data."

But you can guarantee that the database won't return unauthorized rows. That's not a prompt. That's not a convention. That's a constraint enforced by the database engine on every query, from every source, every time.

RLS turns your database from a dumb store that trusts its callers into an active participant in your security model. It doesn't replace application logic. It catches everything your application logic misses. And when an AI agent is the one writing the application logic at runtime, that safety net isn't optional anymore.

Managing a handful of Kubernetes clusters is difficult but manageable. Managing hundreds or thousands of clusters is a fundamentally different problem. At that scale, Kubernetes stops being "infrastructure you run" and becomes a system you must design.

This article is a companion to my ContainerDays talk. It walks through why Kubernetes fleets are inevitable, where most teams go wrong, how fleet controllers emerged, and how to evaluate and choose the right tools using a practical framework. We'll do a fast-paced review of five open-source tools - Clusternet, Karmada, Crossplane, Cluster API, and Rancher - and show how each addresses different multi-cluster management challenges.

From Clusters to Fleets: How We Got Here

Nobody sets out to build a Kubernetes fleet. Teams usually start with one cluster, one environment, and a small number of services. Then reality arrives: more regions, more environments, more teams, sometimes per-tenant clusters. Infrastructure costs drop. Automation improves. Kubernetes makes scale accessible.

Nothing breaks. And yet everything changes.

At some point, adding "just one more cluster" doesn't feel free anymore. Coordination overhead grows. Consistency erodes. Visibility fragments. Operational toil creeps in. This isn't a Kubernetes problem - it's a systems problem.

Why Fleet Problems Are Predictable

The key insight is simple: infrastructure scales linearly, but complexity does not. Every new cluster adds policies, permissions, upgrades, observability pipelines, and human coordination. This is why fleets appear suddenly and feel overwhelming. Teams didn't fail - they reached the next stage of maturity.

Why This Feels Familiar to Developers

If you've ever worked on a large codebase, this story should sound familiar. Kubernetes fleets fail for the same reasons software systems fail.

Think of it this way:

• Clusters behave like services
• YAML becomes an untyped API
• Helm charts act like shared libraries
• Configuration drift looks exactly like forked code
• Snowflake clusters are just technical debt by another name

The core failure mode is always the same: too much reuse, too late, without abstraction. Teams copy YAML. They templatize configurations. They standardize Helm charts. And then every cluster needs "just one exception." Standardization slows divergence - it doesn't prevent it.

The Shift: From DevOps to Platform Engineering

At fleet scale, infrastructure is no longer a task - it's a product. This is where many organizations shift from DevOps as operators to platform engineering as enablers. The goal changes from "manage clusters" to "enable teams through self-service, automation, and clear abstractions."

When you're running clusters at a 100:1 ratio, you can't afford to have engineers manually configuring each one. Standardization, observability, security, and access control become pressing issues that demand automation. Fleet controllers emerge as a direct response to this shift.

Understanding Controllers in Kubernetes

Before diving into fleet controllers, let's understand what a controller is in Kubernetes. A controller is a control loop that watches the state of your cluster through the API server and makes changes to move the current state toward the desired state.

Controllers work with Custom Resource Definitions (CRDs) to extend Kubernetes capabilities. For example, imagine you want to manage a fleet of speakers at a conference. You could define a CRD for Speaker resources:

apiVersion: conference.example.com/v1
kind: Speaker
metadata:
  name: john-doe
spec:
  topic: "Kubernetes Fleet Management"
  duration: 45
  room: "main-hall"
  requiredEquipment:
    - microphone
    - projector
status:
  assigned: false
  equipmentReady: false

A speaker controller would continuously watch for Speaker resources and ensure the actual state matches the desired state. When a new Speaker is created, the controller might:

1. Check room availability and assign the speaker

2. Verify required equipment is available

3. Send calendar invites to attendees

4. Update the status to reflect current state

5. Handle conflicts or resource constraints

If the speaker's room changes, the controller detects the difference between desired state (spec) and current state (status), then takes action to reconcile them. This same pattern applies at fleet scale - fleet controllers watch cluster resources and continuously reconcile state across hundreds or thousands of clusters.

What Is a Fleet Controller (Really)?

A fleet controller follows the exact same control-loop model as a standard Kubernetes controller - it just operates across clusters instead of inside a single cluster.

If a regular controller watches resources within one cluster and reconciles their state, a fleet controller watches many clusters and reconciles their collective state against a desired configuration.

Think of it as moving up one level:

Controller: “Given this desired state, make this cluster match it.”

Fleet controller: “Given this desired state, make all these clusters match it.”

Instead of reconciling Pods, Services, or custom resources, a fleet controller reconciles things like:

• Which clusters exist and are healthy

• Which workloads should run on which clusters

• Which policies apply globally vs regionally

• How upgrades, failures, and drift are handled across the fleet

From an implementation perspective, a fleet controller is not magic and not a single product. It is a higher-level control plane built from familiar Kubernetes primitives: APIs, CRDs, controllers, and reconciliation loops - just applied at fleet scale.

Common responsibilities include:

• Cluster registration and inventory

• Declarative propagation of workloads and policies

• Lifecycle automation (create, upgrade, decommission clusters)

• Observability, governance, and drift detection

• High availability and failover across clusters

These are not “advanced” features. Once you operate more than a handful of clusters, they become survival features.

The Big Mistake: Treating Tools as Competitors

One of the most common mistakes teams make is asking "which fleet management tool should we choose?" This is the wrong question. Fleet management is layered, and different tools solve different layers of the problem.

However, this doesn't mean you should adopt every tool that exists. There's a balance between composition and consolidation. Adding a new tool to your stack should be justified by real pain points and clear value. More importantly, your tools must work in synergy with each other - not in conflict.

Before adopting a new tool, ask: does this solve a problem we actually have? Does it integrate well with our existing stack? Are we introducing unnecessary complexity? The goal is a composed platform where each tool has a clear purpose and they work together cohesively, not a fragmented collection of competing solutions.

The Three Layers of Fleet Management

Layer 1: Infrastructure - Cluster Lifecycle

This layer answers how clusters are created, upgraded, and how consistency is enforced.

Cluster API gives you declarative, repeatable cluster lifecycle management. Instead of scripts and manual processes, clusters become versioned resources. Features like ClusterClass let organizations define reusable cluster templates - the same way you'd define a base class in software. Other teams rely on managed Kubernetes offerings combined with automation and policy tooling. The key shift is that clusters stop being snowflakes and start being managed resources with lifecycle, ownership, and consistency built in.

Layer 2: Platform - Abstraction and Reuse

This is where most teams struggle - and where the biggest wins exist. This layer answers how teams consume infrastructure, how complexity is hidden, and how intent is expressed.

Crossplane lets you expose infrastructure and services as Kubernetes APIs, using compositions to define reusable intent. kro helps teams define Kubernetes-native abstractions for applications and environments. Many organizations also build lighter-weight internal platforms using CRDs, controllers, and policy engines like Kyverno or OPA. What matters is that teams consume intent - not raw YAML.

Layer 3: Application - Delivery and Distribution

This layer answers how workloads are packaged, deployed across clusters, and how changes are rolled out safely. Tools like Helm, Kustomize, Argo CD, and Flux are commonly used here. GitOps works extremely well at this layer, but it's important to be honest: GitOps delivers workloads - it does not define infrastructure abstractions or solve fleet architecture.

Five Open-Source Fleet Tools: A Practical Review

Let's look at five tools that address multi-cluster management from different angles. Each has unique strengths in provisioning, management, and application support.

Clusternet

Clusternet is a lightweight, Kubernetes-native multi-cluster management platform. It focuses on managing clusters as a fleet by providing a hub-agent architecture where child clusters register with a parent hub. Its strength is workload distribution - you can define scheduling policies to deploy applications across clusters based on labels, regions, or custom rules. Clusternet treats multi-cluster workload orchestration as a first-class concern and is a good fit for teams that need to distribute applications across many clusters without heavy infrastructure investment.

Karmada

Karmada (Kubernetes Armada) is built specifically for multi-cloud and multi-cluster orchestration. It extends Kubernetes APIs to work across clusters, so you can use familiar resources like Deployments and Services while Karmada handles the propagation and scheduling across your fleet. Its PropagationPolicy and OverridePolicy resources give fine-grained control over where and how workloads land. Karmada shines when you need cross-cluster failover, replica scheduling, and policy-based distribution at scale. It's one of the most feature-complete open-source options for fleet-wide workload management.

Crossplane

Crossplane takes a fundamentally different approach. Rather than managing clusters directly, it turns infrastructure into Kubernetes APIs through Compositions and Claims. Teams define what they need using custom resources, and Crossplane provisions the underlying infrastructure - cloud resources, databases, clusters, anything with a provider. At the fleet level, Crossplane is invaluable as a platform layer tool: it enables self-service infrastructure consumption and enforces organizational standards through compositions. It doesn't orchestrate workloads across clusters, but it's the best open-source option for building platform abstractions.

Cluster API

Cluster API (CAPI) focuses squarely on the lifecycle of Kubernetes clusters themselves. It lets you declaratively create, configure, upgrade, and destroy clusters using the Kubernetes API. With ClusterClass, you can define reusable cluster templates - think of it as inheritance for your infrastructure. This means new clusters are consistent by default, upgrades are version-controlled, and provisioning is repeatable across clouds. Cluster API is the go-to tool at the infrastructure layer when you need to manage cluster lifecycle at scale.

Rancher

Rancher by SUSE is the most complete platform in this list - it provides a full management plane for Kubernetes clusters across any infrastructure. It handles cluster provisioning, centralized authentication, monitoring, policy enforcement, and application catalog management through a single UI and API. Rancher is often the first fleet management tool organizations adopt because it offers immediate visibility and control. Its strength is breadth: it covers lifecycle, observability, security, and app delivery in one package, making it a pragmatic starting point for teams that need results quickly.

How These Tools Map to the Three Layers

No single tool covers all three layers perfectly. Here's where each tool provides the most value:

Infrastructure Layer (Lifecycle): Cluster API and Rancher are strongest here. CAPI for declarative lifecycle-as-code, Rancher for centralized management with a UI.

Platform Layer (Abstraction): Crossplane dominates this space. It's purpose-built for turning infrastructure into consumable APIs.

Application Layer (Distribution): Karmada and Clusternet focus on workload propagation and scheduling across clusters. Rancher also provides app catalog and deployment capabilities.

The practical takeaway: compose these tools based on your organizational needs rather than picking one and forcing it to do everything.

Compose, Don't Consolidate

There is no single tool that solves fleet management end-to-end. Successful platforms are composed, layered, and intentional. Cluster lifecycle tools don't replace GitOps. Platform abstractions don't replace delivery pipelines. Some teams lean more heavily on managed services. Some invest deeply in platform tooling. Some do both. What successful teams have in common is that they align tools to layers instead of forcing one tool to do everything.

Reusable Components Are the Unit of Scale

This is the most important idea in fleet management: clusters are not the unit of scale - reusable components are. Cluster templates, platform APIs, application abstractions, and policy bundles are what allow organizations to grow without growing operational load. Clusters are execution environments. Components encode intent.

The Interface for Successful Fleet Management

There's a concept from the development world that applies directly to fleet management but is often overlooked: the developer portal. While fleet controllers manage the technical orchestration, teams still need a unified interface to discover, understand, and interact with their distributed infrastructure.

A developer portal should orchestrate all the operational context: documentation, service inventory, metrics dashboards, deployment URLs, regional endpoints, team ownership, dependencies, and SLOs. It's the human interface to your fleet - a single place where engineers can answer questions like "where is this service running?", "who owns this cluster?", "what's the health of services in us-west?", or "how do I deploy to production?"

These concepts are not new. Tools like Backstage, Port, and others have made developer portals mainstream in software engineering. However, they're not always treated as native components in Kubernetes fleet management - and they should be. A well-designed platform includes not just the control plane (fleet controllers, GitOps, policy engines) but also the developer plane (portals, service catalogs, observability interfaces).

Without this layer, you end up with infrastructure that works but nobody knows how to use it. Engineers waste time searching for information, duplicating work, or making changes blindly. A developer portal bridges the gap between powerful infrastructure and productive teams by making the fleet discoverable, understandable, and accessible.

A Framework for Evaluating Fleet Tools

When choosing tools for your fleet, evaluate against these dimensions:

Provisioning: Does the tool help you create and destroy clusters declaratively? Can you template and version cluster configurations?

Management: Does it provide centralized inventory, health monitoring, and policy enforcement? Can you manage upgrades across the fleet?

Application Support: Does it handle workload distribution, scheduling policies, and multi-cluster deployment? Does it integrate with your existing CI/CD and GitOps workflows?

Abstraction Quality: Does it let you hide complexity without hiding capability? Can teams consume infrastructure through clean APIs rather than raw YAML?

Composability: Does it play well with other tools in your stack, or does it demand to be the single pane of glass?

Use these questions as a replicable framework to evaluate tools based on your specific organizational needs rather than feature comparisons.

Principles for Success

Across organizations, the same principles apply:

• Start with Standards. Define clear conventions for clusters, networking, security, and observability before adopting tools.
• Layer Properly. Separate lifecycle, abstraction, and delivery concerns. Don't conflate them.
• Abstract Progressively. Hide complexity, not capability. Bad abstractions are worse than none.
• Compose Solutions. Align tools to problems instead of chasing a silver bullet.
• Measure Impact. Track reduced toil, faster onboarding, and fewer incidents - not just tool adoption.

Where to Start (Practically)

There is no universal starting point. Ask instead: where is the pain? Where is the toil? What breaks most often?

If provisioning and upgrades are painful, focus on lifecycle automation with Cluster API or Rancher. If teams struggle with consistency, invest in platform abstractions with Crossplane. If workload distribution is complex, evaluate Karmada or Clusternet. If delivery is slow or risky, improve your GitOps workflows.

The reality is that most large companies end up building custom solutions - custom CRDs and controllers tailored to their specific workflows and constraints. This is reasonable and often necessary at scale when off-the-shelf tools can't fully capture your domain logic.

For smaller organizations or those just starting their fleet journey, my recommendation is different: start by composing a solution from existing tools. Extend and adapt them to fit your needs. Only move to fully custom solutions when it becomes a must - when the complexity or constraints of existing tools outweigh the cost of maintaining your own controllers.

Custom solutions give you ultimate flexibility, but they also come with maintenance burden, technical debt, and the need for deep Kubernetes expertise. Compose first, customize progressively, and build fully custom only when justified by clear requirements that cannot be met otherwise.

Pick one problem, solve it well, and build reusable components around it. Start small. Document it well. Expand intentionally.

The Next Step: Autonomous Operations

Today, my daily work focuses on building AI-driven SRE systems. The goal is simple: engineers become the bosses, not the on-call responders.

We're building autonomous AI SRE agents that monitor systems in real time, understand incidents as they happen, and actively heal issues before you even wake up. Imagine starting your morning with an incident report waiting for you - including a pull request that fixes the root cause, clear explanations of what happened, and concrete recommendations to prevent similar issues in the future.

That future only works if infrastructure is well-designed. If fleets are observable. If abstractions are clear. And if systems are built to be reasoned about - by humans and machines.

Fleet management, platform engineering, and reusable components are not just about scale - they are the foundation for autonomous operations.

Final Thoughts

Fleets are inevitable. Chaos is not.

By applying software architecture principles to infrastructure - abstraction, reuse, composition - and by choosing the right tools for the right layers, teams can scale Kubernetes without losing control. The five tools we covered each solve a genuine piece of the puzzle, and understanding where they fit is more valuable than any feature comparison.

If any of this resonates with you, feel free to reach out. I'm always happy to talk.

What is TinyML?

TinyML refers to the application of machine learning techniques on extremely resource-constrained devices, such as microcontrollers and other small embedded systems with limited memory, processing power, and energy resources. Unlike traditional machine learning that runs on powerful servers or cloud infrastructure, TinyML brings inference capabilities directly to devices that might have as little as 256KB of RAM and a few hundred MHz of processing power.

The appeal is compelling: TinyML helps tiny devices make decisions based on huge amounts of data without wasting time and energy transmitting it elsewhere, and with inference on-device, user privacy is protected since no audio or data ever needs to be sent to the cloud.

ESP-Skainet: Voice Intelligence for ESP32

What is ESP-Skainet?

ESP-Skainet enables convenient development of wake word detection and speech command recognition applications based on Espressif's ESP32 series chips, allowing developers to easily build wake word and command recognition solutions. It's Espressif's answer to bringing voice assistant capabilities to low-power microcontrollers.

Core Components

ESP-Skainet consists of several key engines:

WakeNet (Wake Word Detection): WakeNet is designed for low-power embedded MCUs with a low memory usage of approximately 20KB and achieves a 97% wake-up performance within a one-meter distance in a quiet environment, and 95% within a three-meter distance. Espressif provides wake words such as "Hi, Lexin" and "Hi, ESP" for free, and also supports custom wake words.

MultiNet (Speech Command Recognition): MultiNet is a lightweight model that allows ESP32 to perform offline speech-recognition of multiple commands, using Convolutional Recurrent Neural Networks (CRNN) and Connectionist Temporal Classification (CTC), processing an audio clip's Mel-Frequency Cepstral Coefficients (MFCC) as input and outputting phonemes. Currently, MultiNet supports up to 200 Chinese or English speech commands such as "Turn on the air conditioner" and "Turn on the bedroom light", and users can easily add their own commands without retraining the model.

Audio Front-End (AFE): The Audio Front-End integrates AEC (Acoustic Echo Cancellation), VAD (Voice Activity Detection), BSS (Blind Source Separation), and NS (Noise Suppression), with Espressif's two-mic AFE qualified as a "Software Audio Front-End Solution" for Amazon Alexa Built-in devices.

What You Can Do with ESP-Skainet

ESP-Skainet is ideal for AIoT and smart home applications, enabling local voice control of devices, with typical applications including smart-home devices like voice-controlled switches, outlets, lamps, thermostats, and security systems, smart-office equipment like voice-controlled displays and phones, and interactive products such as educational toys or assistants for the elderly.

Hardware Requirements

To run ESP-Skainet, you need an ESP32 or ESP32-S3 development board with an integrated audio input module. Popular boards include:

• ESP32-Korvo

• ESP32-S3-Korvo-1 and Korvo-2

• ESP-BOX

• ESP32-S3-EYE

The ESP32-Korvo includes an ESP32-WROVER-B module with 16MB SPI flash (comfortably above the 4MB minimum for MultiNet support) and 64Mb pseudo-static RAM (PSRAM).

Limitations

• Language support is primarily Chinese and English

• Requires specific hardware with adequate RAM (ESP32 or ESP32-S3)

• Requires ESP-IDF v4.4 or ESP-IDF v5.0

• The wake word detection must be active before command recognition begins

TensorFlow Lite for Microcontrollers: The Foundation of TinyML

Overview

TensorFlow Lite for Microcontrollers is designed for the specific constraints of microcontroller development, allowing deployment of machine learning models on tiny microcontrollers to boost the intelligence of billions of devices in our lives, including household appliances and Internet of Things devices, without relying on expensive hardware or reliable internet connections.

How It Works

Because machine learning is computationally expensive, TensorFlow Lite for Microcontrollers requires a 32-bit processor, such as an ARM Cortex-M or ESP32, and the library is mostly written in C++, requiring a C++ compiler.

The workflow follows these steps:

1. Train the model on a computer or server

2. Convert to FlatBuffer format (.tflite file)

3. Convert to C array for embedding in firmware

4. Deploy using the TensorFlow Lite for Microcontrollers library

5. Run inference on the microcontroller

On the microcontroller, the TensorFlow Lite for Microcontrollers library uses the model to perform inference, such as feeding unseen photos to determine if there is a cat in the photo.

What You Can Build

Example applications include a Hello World demonstration of the absolute basics, and person detection that captures camera data with an image sensor to detect the presence or absence of a person. TinyML applications include visual and audio wake words that trigger an action when a person is detected in an image or a keyword is spoken, predictive maintenance on industrial machines using sensors to continuously monitor for anomalous behavior, and gesture and activity detection for medical, consumer, and agricultural devices, such as gait analysis, fall detection or animal health monitoring.

Edge Impulse: The TinyML Development Platform

What is Edge Impulse?

Edge Impulse is a development platform that simplifies building, training, and deploying machine learning models on embedded systems and edge devices such as microcontrollers, sensors, and single-board computers like the Raspberry Pi or Arduino. Edge Impulse is a cloud-based machine learning operations (MLOps) platform for developing embedded and edge ML (TinyML) systems that can be deployed to a wide range of hardware targets, addressing challenges of fragmented software stacks and heterogeneous deployment hardware by streamlining the TinyML design cycle with various software and hardware optimizations.

Key Features

End-to-End Workflow: Edge Impulse makes it easy to collect a dataset, choose the right machine learning algorithm, train a production-grade model, and run tests to prove that it works, with the whole process quick enough to run through in a few minutes.

Data Collection: Edge Impulse can easily collect data from any sensor and development board using the Data forwarder, a small application that reads data over serial and sends it to Edge Impulse.

Model Optimization: The platform provides estimates of how the model will perform on the target device, including memory usage (RAM and flash) and latency, helping ensure the model fits within hardware constraints.

Wide Hardware Support: Edge Impulse launched with the Arduino Nano 33 BLE Sense, but models can be exported as an Arduino library to run on any Arm-based Arduino platform including the Arduino MKR family or Arduino Nano 33 IoT, providing the board has enough RAM.

Typical Workflow

1. Data acquisition - Connect your device and collect labeled sensor data

2. Impulse design - Configure processing blocks (like MFCC for audio) and learning blocks (neural network)

3. Feature generation - Extract features from raw data

4. Training - Train the model with configurable parameters

5. Testing - Validate accuracy and performance

6. Deployment - Export as optimized library for your target hardware

The model trained using Edge Impulse can be around 18kb in size, which is mind-blowingly small for something so sophisticated and leaves a lot of space for application code.

Other Major Tools and Frameworks

STM32Cube.AI / X-CUBE-AI

X-CUBE-AI is a package that extends the capabilities of STM32CubeMX, adding the possibility to convert a pre-trained neural network into an ANSI C library that is performance optimized for STM32 microcontrollers based on ARM Cortex-M4 and M7 processor cores. The tool has advantages for developers such as a graphical user interface, support for different deep learning frameworks such as Keras and TensorFlow Lite, 8-bit-quantization, and compatibility with different STM32 microcontroller series.

ARM CMSIS-NN

Common Microcontroller Software Interface Standard (CMSIS) has a version to deploy Neural Networks (CMSIS-NN) that was developed hand by hand with TensorFlow Lite engineers, meaning the operations supported by ARM microcontrollers are the same that TensorFlow Lite supports. CMSIS-NN kernels are used at a low level by tools like STM32Cube.AI.

Common Practices and Optimization Techniques

Model Optimization

The key to running ML on microcontrollers is aggressive optimization:

Quantization: Converting 32-bit floating-point models to 8-bit integer representations, dramatically reducing memory footprint and computation requirements with minimal accuracy loss.

Pruning: Removing unnecessary connections in neural networks to reduce model size.

Knowledge Distillation: Training smaller "student" models to mimic larger "teacher" models.

Hardware Selection

A 32-bit processor such as an ARM Cortex-M or ESP32 is required for TensorFlow Lite for Microcontrollers. Popular platforms include:

• ARM Cortex-M4/M7: STM32 series, nRF52 series

• ESP32/ESP32-S3: For Wi-Fi/Bluetooth connectivity

• Arduino Nano 33 BLE Sense: Features an Arm Cortex-M4 microcontroller running at 64 MHz with 1MB Flash memory and 256 KB of RAM, with onboard sensors including a 9-axis IMU

Development Workflow

Currently, STM32Cube.AI is more commonly used because the X-CUBE-AI expansion package provides end-to-end solutions for automatic neural network model conversion, validation, and system performance measurements, making ARM Cortex-M 32-bit STMicroelectronics microcontrollers the most common platform.

What You Can and Can't Do

What Works Well

Audio Classification: Models can recognize household sounds like running water from a faucet, and be trained in just a few minutes with only a small amount of audio data

Simple Image Recognition: Person detection, basic object recognition Gesture Detection: Using accelerometer/IMU data Keyword Spotting: Wake word detection and simple voice commands Anomaly Detection: Identifying unusual patterns in sensor data

Limitations

Memory Constraints: TinyML systems often have very flat memory hierarchies, due to small or non-existent caches and often no off-chip memory

Model Complexity: Complex vision models like modern CNNs, large language models, and multi-modal systems typically won't fit

Computation Speed: Real-time video processing remains challenging

Training: All training must happen off-device; microcontrollers can only run inference

Accuracy Trade-offs: Smaller models generally mean lower accuracy compared to cloud-based alternatives

How TinyML Works: A Technical Overview

The Inference Engine

The main application performs real-time predictions by defining the model to be loaded from a header file, adding necessary operation resolvers (like Fully Connected and ReLU layers), allocating a tensor arena which provides memory for intermediate computations and tensor data during inference, and initializing the interpreter using the model, the operation resolver, the tensor arena, and its size.

Memory Management

Models are typically stored in flash memory as constant C arrays. During inference, a "tensor arena" in RAM holds intermediate calculations. The challenge is balancing model complexity with available resources.

Hardware Acceleration

Many modern microcontrollers include DSP instructions or dedicated ML accelerators. ARM Cortex-M processors have advanced development of hardware architectures and DSP capabilities while maintaining low cost and power consumption.

Getting Started: A Practical Guide

Prerequisites

• Basic programming skills (C/C++ and Python)

• Understanding of machine learning fundamentals

• Familiarity with embedded systems (helpful but not required)

Recommended Hardware

For Beginners:

• ESP32 DevKit ($10-20) - great value, Wi-Fi/Bluetooth

For Voice Projects:

• ESP32-S3-BOX or ESP32-Korvo boards

• Boards with built-in microphones

Software Setup

1. Choose Your Path:

   • Beginner-friendly: Edge Impulse (web-based, no local setup)

   • ESP32 voice: Clone ESP-Skainet and install ESP-IDF v4.4 or v5.0

   • General TinyML: Install TensorFlow, TensorFlow Lite, and your preferred IDE

2. First Project: Follow a tutorial to train a model that can recognize household sounds like running water from a faucet, using Edge Impulse to collect audio data, train a simple model, and export it as a C++ library

Learning Resources

Books:

• TinyML: Machine Learning with TensorFlow Lite on Arduino and Ultra-Low-Power Microcontrollers by Pete Warden and Daniel Situnayake, the standard textbook on embedded machine learning

Courses:

• Harvard's Deploying TinyML course provides hands-on experience with deploying TinyML to physical devices, teaching programming in TensorFlow Lite for microcontrollers and featuring projects based on a TinyML Program Kit that includes an Arduino board with onboard sensors and an ARM Cortex-M4 microcontroller

Online Platforms:

• Edge Impulse documentation and tutorials

• TensorFlow Lite for Microcontrollers examples

• ESP-Skainet GitHub repository

Sample Projects to Start With

1. Audio wake word detection - Using ESP-Skainet or Edge Impulse

2. Gesture recognition - Using accelerometer data

3. Simple image classification - Person detection with a camera module

4. Anomaly detection - Monitor sensor patterns for unusual behavior

Conclusion

AI on microcontrollers represents a paradigm shift in how we think about embedded intelligence. Machine learning at the very edge enables valuable use of the 99% of sensor data that is discarded today due to cost, bandwidth or power constraints, with applications spanning health, white goods, mobility, industry, retail and agriculture.

While the technology is still maturing, the tools available today—from ESP-Skainet's voice capabilities to TensorFlow Lite's flexibility and Edge Impulse's user-friendly platform—make it possible for developers to add AI capabilities to their embedded projects without needing deep machine learning expertise.

The key is understanding the constraints: work within memory limits, optimize aggressively, and choose problems suited to edge inference. Most papers on TinyML have shown quite promising results in terms of accuracy, execution time, power consumption, and memory footprint, though edge computing is a relatively new research topic facing many new challenges.

Start small, experiment with existing tools and examples, and gradually build your understanding. The future of embedded intelligence is being written right now, and with the democratization of TinyML tools, anyone can contribute to it.

Sources:

• ESP-Skainet GitHub Repository

• Espressif ESP-Skainet Overview

• TensorFlow Lite for Microcontrollers Documentation

• Edge Impulse Platform

• STM32Cube.AI and CMSIS-NN Research

• Harvard TinyML Course

What is USB Army Knife?

USB Army Knife is an ESP32-based multitool designed for security research, penetration testing, and hardware hacking. Built on the versatile ESP32 platform, it combines multiple capabilities into a single portable device, including BadUSB functionality, WiFi attacks, file management, and custom script execution.

Why a New Modern Interface?

While the original USB Army Knife has a functional interface, I wanted to create something that embraced modern web development practices:

• Modern Client Code: Built with current web standards and frameworks for better performance and maintainability

• Easier Maintenance: Clean, modular codebase that's straightforward to update and extend

• Better UX: Intuitive design with responsive layouts and smooth interactions

• Developer-Friendly: Well-structured code that other developers can easily contribute to

This isn't meant to replace the official interface - rather, it's an alternative for users who prefer a more modern web application experience and developers who want cleaner code to work with.

The New Web Interface

The new web interface provides a comprehensive dashboard accessible through your browser, offering real-time device monitoring and full control over all device capabilities. The interface is designed with usability in mind, featuring a clean navigation structure and intuitive controls.

Key Features

1. Real-Time Device Dashboard

The dashboard provides an at-a-glance view of your device status:

• Device uptime and current status

• USB mode indicator (Serial + HID)

• Memory and heap usage monitoring

• SD card storage status

• Error tracking

• Agent connection status

• Hardware capabilities (SD, WiFi, TFT, Button, LED, Marauder)

• System information including chip type and firmware version

2. File Management

The file management system allows you to:

• Browse all files stored on the device's SD card

• Upload new files directly through the browser

• Download files from the device

• Edit text-based configuration files

• Execute scripts with a single click

• View and display images stored on the device

• Delete unwanted files

The interface supports multiple file types including scripts (.ds), images (.png), text files (.txt), and configuration files (.json).

3. Script Execution

One of the most powerful features is the script execution system:

• Run DuckyScript files and custom commands

• Browse available scripts with easy-to-read listings

• Access a comprehensive command reference

• Execute raw commands directly

• View execution results in real-time

The built-in command reference includes support for delays, keyboard inputs, LED control, and display operations.

4. Display & LED Control

Control the device's display and LED indicators:

• Display custom text at specific coordinates

• Show images from the SD card

• Control RGB LEDs (Red, Green, Blue, Off)

• Clear the display

• Upload and display custom graphics

5. ESP32 Marauder Integration

For WiFi security testing, the interface includes full ESP32 Marauder support:

• Execute Marauder attack commands

• Common commands readily available (attack, scan, sniff, beacon, deauth, probe, list, select, clear, help)

• View command results in the device logs

• Easy command input with helpful placeholders

6. On-Screen Keyboard

Send keystrokes to target devices using the virtual keyboard:

• Full QWERTY layout with all standard keys

• Support for modifier keys (Ctrl, Shift, Alt, Win)

• Function keys (F1-F12)

• Special keys (Tab, Caps, Enter, Backspace, etc.)

• Navigation keys (Home, End, PgUp, PgDn, Insert, Delete)

• Text input area for typing longer strings

7. Device Logs

Monitor all device activity through the comprehensive logging system:

• Real-time log viewing

• Track script execution

• Monitor command execution

• View display operations

• Debug device behavior

• Refresh logs on demand

• Clear logs when needed

8. Complete API Documentation

For developers, the interface includes full API documentation:

• REST API endpoints for all device functions

• WebSocket connections for real-time updates

• Device status and information endpoints

• File management operations

• Script and command execution

• Agent operations

• Display control

• Marauder command execution

• Audio capture and streaming

• Settings configuration

Getting It Running

The CORS Thing

Okay, so there's this annoying browser security thing called CORS. Basically, browsers don't like it when web pages talk to random devices on your network (for good reason). The fix? Use Corsy, a lightweight CORS proxy I made. It's super simple and solves the problem.

If you're running the interface locally (like, actually cloning the repo and running it on your machine), you don't need Corsy. Otherwise, yeah, you'll want it.

Quick Start

1. Make sure your USB Army Knife is powered up and on your network
2. Set up Corsy if you need it (using the GitHub Pages version or accessing remotely)
3. Open the interface - either the live demo, or clone it and run locally
4. Start playing around!

What Can You Do With This?

Honestly? Tons of stuff:

• Security Research: Test WiFi networks, try out different attacks, see what works
• Penetration Testing: BadUSB payloads, automated keystroke injection, all that good stuff
• Hardware Hacking: Build ESP32 projects, mess with sensors, debug weird behavior
• Learning: Figure out how this hardware/software combo works, break things, fix things
• IoT Experiments: Test device security, poke at protocols, find vulnerabilities

But really, the best use case is "I wonder what happens if I do this..." and then spending your weekend finding out.

Links

• This Modern Interface: USBArmyKnife-web
• CORS Proxy: Corsy
• Live Demo: https://niradler.github.io/USBArmyKnife-web/
• Original USB Army Knife: USBArmyKnife

Wrap Up

This project is what happens when you combine curiosity, modern web dev, and a love for hardware that does cool stuff. It's not perfect, it's not trying to be corporate, and it's definitely not the "official" anything. It's just a fun alternative that works well and is easy to tinker with.

The whole point is learning and experimentation - mixing software and hardware, seeing what you can build, and maybe learning something along the way. If you want to add features, change things, or just see how it works, go for it. The code is there, the hardware is fun, and there's always something new to try.

Whether you're into security research, hardware hacking, or just like building things that do stuff, give it a shot. And if you break something, hey, that's half the fun.

Contributing

Pull requests welcome! Found a bug? Cool. Want to add a feature? Even better. The codebase is clean enough that you won't want to cry when you look at it, which is always a plus.

Disclaimer: This is for learning, research, and authorized testing only. Don't be that person who uses this stuff without permission. Seriously, don't.

The Flipper Zero Sweet Spot

What makes Flipper Zero special is its polish. Everything works out of the box. The UI is intuitive, the form factor is perfect, and the community around it is massive. You can sniff NFC cards, replay Sub-GHz signals, control IR devices, and even write BadUSB payloads without breaking a sweat.

But here's where it gets interesting: Flipper Zero wasn't designed to be a WiFi powerhouse. Yes, you can add WiFi dev boards as modules, but the native capabilities are limited. If you want to dive into WPA3 attacks, 5GHz testing, or custom ESP-IDF exploits, you'll quickly hit a wall.

Enter the ESP32 Ecosystem

The DIY hardware community has been quietly building something remarkable around ESP32, CC1101, and M5Stack boards. We're talking about "Flipper-class" setups sometimes even better for a fraction of the cost.

Here's what caught my attention:

ESP32Marauder - A WiFi swiss army knife that handles deauth attacks, PMKID/EAPOL capture, and beacon spam. It runs on inexpensive ESP32 boards and gives you capabilities that would require expensive WiFi Pineapple hardware.

Ghost_ESP - Takes things further with 5GHz support and WPA3 flood testing on ESP32-C5 hardware. This is territory Flipper Zero doesn't touch without significant modifications.

Bruce Firmware - This one's ambitious. Running on M5Stack Cardputer or T-Embed devices, it combines WiFi, BLE, Sub-GHz, NFC, IR, and BadUSB plus a JavaScript engine for automation. Add a CC1101 module and PN532 NFC board, and you've essentially built your own Flipper Zero with better WiFi capabilities.

EvilCrow RF - Focused purely on RF, the V2 version uses dual CC1101 transceivers for more sophisticated Sub-GHz experiments like rolljam attacks. More flexible than Flipper's stock firmware for RF research.

The Economics Are Compelling

Let's talk numbers. A Flipper Zero costs $199-299. A basic ESP32 dev board? $15-30. An M5Stack Cardputer with all the modules to match Flipper's capabilities? Around $60-90 total. Even if you build multiple specialized devices one for WiFi, one for RF, one for NFC you're still spending less than a single Flipper Zero.

This isn't about cheap knockoffs. These are purpose-built tools that often exceed Flipper's capabilities in their respective domains.

Why This Matters

I'm not saying Flipper Zero is obsolete. Far from it. If you want an all-in-one device with excellent build quality, great documentation, and a thriving community, Flipper Zero is still the best choice. It's plug-and-play security research.

But if you're comfortable with DIY electronics, if you want to deeply understand how these systems work, or if you need specific capabilities like advanced WiFi pentesting, the ESP32 ecosystem is incredibly powerful.

What I'm Building

Over the next few posts, I'll be documenting my journey building a DIY security research lab using ESP32 hardware. I want to understand:

• How these tools compare in real-world scenarios

• What the trade-offs are between convenience and capability

• How to build a modular testing setup that's both powerful and portable

• Whether the DIY approach actually delivers on its promise

The goal is to explore what's possible when you combine modern microcontrollers, open-source firmware, and a willingness to get your hands dirty.

Next Steps

In the next post, I'll break down how to choose the right hardware stack for your needs.

I'll compare specific use cases: WiFi pentesting, RF research, NFC/RFID work, and multi-protocol scenarios. We'll look at what works, what doesn't, and what you should actually build.

Important Note: All the tools and techniques discussed in this series are for educational purposes and authorized security testing only. Always get explicit permission before testing any systems or networks you don't own.

Resources

• Flipper Zero Official Site

• ESP32Marauder GitHub

• Ghost_ESP Releases

• Bruce Firmware

• EvilCrow RF Information

Does it solve a problem? Kind of. I can glance at my notifications and contribution stats without opening a browser. But that's the excuse I gave myself. The real reason was wanting to build something cool with an e-paper display and ESP32.

Screens

Press Button (GPIO 39) to cycle through:

1. Notifications - Reviews, mentions, assignments

2. Profile - Repos, stars, open PRs, followers

3. PR Overview - Open, Waiting to review, Ready to merge, Request changes

Each screen fetches fresh data when you switch to it.

Hardware

LILYGO T5 V2.3.1 (~$15-20) - ESP32 with 2.13" e-paper and two built-in buttons:

• GPIO 39: Cycle screens

• GPIO 0: Force refresh

That's it. Flash and go.

How It Works

Data fetching:

• REST API for notifications

• GraphQL for profile/activity (smaller payloads)

• Only fetches active screen data

• Only refreshes display when data changes

Power:

• Deep sleep between updates (default 10 min)

• Wakes on timer or button press

• Web server runs 30 seconds after boot for config access

Web Interface

First boot creates AP: "NotificationHub" (password: configure)
Connect and go to 192.168.4.1

Four tabs:

• Dashboard - Status, refresh button

• WiFi - Network config, admin password

• Providers - GitHub token and username

• Settings - Update interval

GitHub Token

Generate a Classic token with:

• notifications

• read:user

Add it in Providers tab.

Setup

1. Flash firmware

2. Connect to "NotificationHub" (password: configure)

3. Go to 192.168.4.1 (or whatever your network assigns)

4. Configure WiFi and admin password

5. Add GitHub token and username

6. Reboot

Updates every 10 minutes. Press buttons for manual control.

Built with Arduino MCP

Used arduino-mcp - connects Claude/Cursor to Arduino CLI:

"Compile for ESP32"
"Upload to board"
"Convert this PNG to C array"

Way faster than copy-pasting commands.

Claude Desktop config:

{
  "mcpServers": {
    "arduino-uvx": {
      "command": "uvx",
      "args": ["arduino-mcp"],
      "env": {}
    }
  }
}

API Endpoints

GET /api/status          # Current status
POST /api/refresh        # Force refresh
POST /api/reset          # Factory reset

Memory Handling

ESP32 RAM is tight:

• Paged API requests

• Dynamic JSON buffers

• Independent screen state

• Display hibernation during sleep

What's Next

Could add:

• GitLab/Bitbucket support

• Larger displays (4.2" or 7.5")

• Historical graphs

• Battery indicator

• Webhooks for instant updates

• 3D printed case

Stack

Hardware: LILYGO T5 V2.3.1

Software: ESP32 Arduino, GxEPD2, ArduinoJson

APIs: GitHub REST + GraphQL, NTP

Links

Code: github.com/niradler/github-dashboard-esp32-epaper

Arduino MCP: github.com/niradler/arduino-mcp

License: MIT | Cost: ~$15-20 | Time: 1-2 hours

This flexibility lets you orchestrate for example GitHub via MCP server, GitHub CLI, AND the Octokit SDK in a single execution, choosing the right tool for each subtask while keeping the model's context clean.

The Problem: When Direct Tool Calls Hit Their Limits

The Model Context Protocol excels at giving AI models structured access to external systems. Want to query GitHub? Use the GitHub MCP server. Need to process files? Use the filesystem MCP server. This works beautifully until it doesn't.

Token waste from intermediate data. Fetch a 50KB document to extract three numbers? The entire document flows through the model's context. Query an API that returns 200 results when you need 5? All 200 results consume tokens.

Sequential execution bottlenecks. Need data from three APIs? Make three separate MCP tool calls, waiting for each to complete before starting the next even when they're independent operations.

Limited data transformation. MCP servers return raw data. If you need to filter, aggregate, or transform it, the model must either do it in-context (expensive) or make additional tool calls (slow).

Rigid interfaces. Sometimes the MCP server doesn't expose exactly what you need. Maybe you need a complex Git operation that requires multiple commands, or you want to use a specific npm package that's perfect for the job.

Consider automating a GitHub workflow:

1. Check open PRs (MCP tool call → 10,000 tokens of PR data)

2. For each PR, check CI status (5 more MCP calls → 15,000 tokens)

3. Filter to PRs with passing CI (model does this in context)

4. Generate a summary report (more context processing)

Total: 30,000+ tokens, 3-4 seconds of sequential execution, and the model is drowning in raw API responses.

Solution: Execute Code, Not Just Tools

MCP Conductor provides a single MCP tool run_deno_code that executes TypeScript/JavaScript in a secure Deno sandbox. But here's what makes it powerful: it gives you three ways to integrate with external systems, and you can mix them freely:

1. Call MCP Servers via mcpFactory (Reuse the Ecosystem)

Want to use existing MCP servers? The global mcpFactory object is auto-injected into your code:

// Connect to GitHub MCP server
const github = await mcpFactory.load('github');

// Call its tools directly from code
const prs = await github.callTool('list_pull_requests', {
  repo: 'myorg/myrepo',
  state: 'open'
});

// Process in execution environment - no tokens wasted
const passingPRs = prs.filter(pr => pr.ci_status === 'passing');

// Only final result returns to model
return { count: passingPRs.length, titles: passingPRs.map(pr => pr.title) };

Benefit: Leverage the entire MCP ecosystem (100+ community servers) while processing data efficiently.

2. Shell Out to CLI Tools (Maximum Flexibility)

Sometimes CLI tools are the best interface. With --allow-run permission, spawn subprocesses directly:

// Use GitHub CLI for complex operations
const ghProcess = new Deno.Command('gh', {
  args: ['pr', 'list', '--json', 'number,title,state', '--limit', '100'],
  stdout: 'piped'
});

const { stdout } = await ghProcess.output();
const prs = JSON.parse(new TextDecoder().decode(stdout));

// Use git CLI for repository operations
const gitLog = new Deno.Command('git', {
  args: ['log', '--oneline', '-10'],
  stdout: 'piped'
});

const commits = new TextDecoder().decode((await gitLog.output()).stdout);

return { prs: prs.length, recentCommits: commits };

Benefit: Full access to mature CLI tools and their rich feature sets.

3. Import npm/JSR Packages (Direct API Access)

Need fine-grained control or want to use specialized packages? Import them directly:

import { Octokit } from 'npm:@octokit/rest@^20';

const octokit = new Octokit({ auth: Deno.env.get('GITHUB_TOKEN') });

// Use full Octokit SDK
const { data: prs } = await octokit.pulls.list({
  owner: 'myorg',
  repo: 'myrepo',
  state: 'open',
  per_page: 100
});

// Complex filtering and transformation
const analysis = prs
  .filter(pr => pr.draft === false)
  .reduce((acc, pr) => {
    acc[pr.user.login] = (acc[pr.user.login] || 0) + 1;
    return acc;
  }, {});

return { totalPRs: prs.length, prsByAuthor: analysis };

Benefit: Direct SDK access with full TypeScript support and comprehensive APIs.

The Power of Three Options: A Real Example

Here's what makes MCP Conductor powerful you can mix all three approaches in a single execution:

// Use MCP server for discovery (good for structured MCP operations)
const github = await mcpFactory.load('github');
const repos = await github.callTool('list_repositories', { 
  org: 'myorg' 
});

// Use CLI for complex git operations (best tool for the job)
for (const repo of repos.slice(0, 5)) {
  const clone = new Deno.Command('git', {
    args: ['clone', '--depth', '1', repo.clone_url, `/tmp/${repo.name}`],
  });
  await clone.output();
}

// Use npm package for analysis (specialized functionality)
import { analyze } from 'npm:code-complexity@^2';

const results = [];
for (const repo of repos.slice(0, 5)) {
  const stats = await analyze(`/tmp/${repo.name}`);
  results.push({ repo: repo.name, complexity: stats.average });
}

// Return only the summary
return results.sort((a, b) => b.complexity - a.complexity);

Why this matters: You choose the best tool for each subtask:

• MCP server for structured operations

• CLI for mature tooling

• npm packages for specialized logic

• Process everything in the sandbox

• Return only what the model needs

How It Actually Works

1. Configuration

Configure MCP Conductor in your MCP client (Cursor, Claude Desktop, etc.):

{
  "mcpServers": {
    "mcp-conductor": {
      "command": "deno",
      "args": ["run", "--allow-read", "--allow-write", "--allow-net", 
               "--allow-env", "--allow-run=deno", "jsr:@conductor/mcp", "stdio"],
      "env": {
        "MCP_CONDUCTOR_WORKSPACE": "${userHome}/.mcp-conductor/workspace",
        "MCP_CONDUCTOR_RUN_ARGS": "allow-read=/workspace,allow-write=/workspace,allow-net,allow-run=gh,git",
        "MCP_CONDUCTOR_MCP_CONFIG": "${userHome}/.mcp-conductor/mcp.json"
      }
    }
  }
}

2. Optional: Configure MCP Servers for Proxy

Create ~/.mcp-conductor/mcp.json to expose MCP servers to executed code:

{
  "mcpServers": {
    "github": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-github"],
      "env": {
        "GITHUB_PERSONAL_ACCESS_TOKEN": "ghp_..."
      }
    },
    "filesystem": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-filesystem", "/allowed/path"]
    }
  }
}

3. Model Writes Code

The model uses the run_deno_code tool:

{
  "tool": "run_deno_code",
  "arguments": {
    "deno_code": "const github = await mcpFactory.load('github');\nconst prs = await github.callTool('list_pull_requests', { repo: 'myrepo' });\nreturn prs.filter(pr => pr.state === 'open').length;",
    "timeout": 30000
  }
}

4. Secure Execution

Conductor spawns a fresh Deno subprocess for each execution:

Key security features:

• Zero permissions by default

• Admin-controlled via MCP_CONDUCTOR_RUN_ARGS

• Fresh process per execution (no state leakage)

• --no-prompt flag prevents permission escalation

• --cached-only by default (no dynamic package fetching)

5. Return Results

Only the final expression/return value goes back to the model:

// Execution processes 50KB of data internally
// Model receives 200 bytes of summary
{
  "content": [
    {
      "type": "text", 
      "text": "Execution successful\nReturn value: {\"openPRs\": 12, \"passingCI\": 8}"
    }
  ]
}

Why This Approach Works

1. Maximum Flexibility

You're not locked into one integration method:

Scenario: GitHub Automation

• Use GitHub MCP server for straightforward operations

• Use GitHub CLI for complex git workflows

• Use Octokit SDK when you need fine-grained API control

• Mix them in the same execution as needed

2. Token Efficiency

Process data in the execution environment, not in the model's context:

Before (Direct MCP calls):

• Fetch 50KB document → 50KB in context

• Extract 3 metrics → Model does this (more tokens)

• Total: ~15,000 tokens

After (Code execution):

• Fetch + process in sandbox → 3 metrics returned

• Total: ~500 tokens

3. Parallel Execution (Your Code, Your Control)

// Model writes parallel execution logic
const [repos, issues, prs] = await Promise.all([
  github.callTool('list_repositories', { org: 'myorg' }),
  github.callTool('search_issues', { query: 'is:open' }),
  github.callTool('list_pull_requests', { state: 'open' })
]);

// 3x faster than sequential MCP calls
return { repoCount: repos.length, openIssues: issues.length, openPRs: prs.length };

4. Rich npm/JSR Ecosystem

Any package is available (if you grant --allow-net or pre-cache it):

// Data processing
import { parse } from 'npm:csv-parse@^5';
import { z } from 'npm:zod@^3';

// API clients
import { Octokit } from 'npm:@octokit/rest';
import Stripe from 'npm:stripe@^14';

// Utilities
import { format } from 'npm:date-fns@^3';
import * as path from 'jsr:@std/path';

Security Model: Admin Control, Not Model Control

Critical design choice: The model writes code, but admins control all permissions via environment variables.

Admin-Configured Permissions

{
  "env": {
    "MCP_CONDUCTOR_RUN_ARGS": "allow-read=/workspace,allow-write=/workspace,allow-net=api.github.com,allow-run=gh,git"
  }
}

Two-Process Isolation

Server Process (Trusted):

• Full permissions to manage workspace

• Install dependencies

• Spawn sandbox subprocesses

User Code Subprocess (Untrusted):

• Only admin-granted permissions

• Fresh process (no state leakage)

• Crashes don't affect server

• Auto-killed after timeout

Security by Default

Every execution runs with:

• --no-prompt - Fails fast, no permission dialogs

• --cached-only - No dynamic package fetching (unless admin allows)

• --no-remote - Blocks remote imports

• Zero permissions + only admin-granted ones

Real-World Example: The GitHub Automation Workflow

Let's see all three integration methods working together:

/**
 * Task: Generate a weekly team report from GitHub
 * - List all PRs merged this week (MCP server - simple structured call)
 * - Clone repos to analyze code quality (CLI - best tool for git operations)
 * - Calculate complexity metrics (npm package - specialized analysis)
 */

// 1. Use MCP server for structured GitHub operations
const github = await mcpFactory.load('github');

const oneWeekAgo = new Date(Date.now() - 7 * 24 * 60 * 60 * 1000).toISOString();

const mergedPRs = await github.callTool('search_pull_requests', {
  query: `is:pr is:merged merged:>${oneWeekAgo}`,
  repo: 'myorg/myrepo'
});

// 2. Use CLI for git operations (better than MCP for complex git workflows)
const cloneDir = '/workspace/analysis';
const gitClone = new Deno.Command('git', {
  args: ['clone', '--depth', '1', 'https://github.com/myorg/myrepo.git', cloneDir],
  stdout: 'piped',
  stderr: 'piped'
});
await gitClone.output();

// 3. Use npm package for specialized analysis
import { analyze } from 'npm:code-complexity-analyzer@^2';
import { format } from 'npm:date-fns@^3';

const complexity = await analyze(cloneDir);

// 4. Generate report (all processing happens in sandbox)
const report = {
  week: format(new Date(), 'MMM dd, yyyy'),
  prsAnalyzed: mergedPRs.length,
  averageComplexity: complexity.average,
  highComplexityFiles: complexity.files
    .filter(f => f.score > 10)
    .map(f => f.path),
  topContributors: Object.entries(
    mergedPRs.reduce((acc, pr) => {
      acc[pr.author] = (acc[pr.author] || 0) + 1;
      return acc;
    }, {})
  )
  .sort(([,a], [,b]) => b - a)
  .slice(0, 5)
};

// 5. Only the summary returns to the model (not raw PR data, git output, or analysis details)
return report;

Token savings:

• Without Conductor: ~50,000 tokens (all PR data, git output, analysis results in context)

• With Conductor: ~800 tokens (just the summary report)

Execution time:

• Without Conductor: ~8 seconds (sequential MCP calls)

• With Conductor: ~3 seconds (parallel execution in sandbox)

Security Considerations

• Admin must carefully configure permissions

• Code execution always carries risk (even in sandboxes)

• Monitor executions and set reasonable timeouts

• Use --cached-only in production (no dynamic package fetching)

• Never use --allow-all

Getting Started

1. Install (No Installation Needed!)

MCP Conductor is available on JSR:

{
  "mcpServers": {
    "mcp-conductor": {
      "command": "deno",
      "args": [
        "run", "--allow-read", "--allow-write", "--allow-net", 
        "--allow-env", "--allow-run=deno",
        "jsr:@conductor/mcp@^0.1", "stdio"
      ]
    }
  }
}

2. Configure Permissions

Set admin-controlled permissions:

{
  "env": {
    "MCP_CONDUCTOR_WORKSPACE": "${userHome}/.mcp-conductor/workspace",
    "MCP_CONDUCTOR_RUN_ARGS": "allow-read=/workspace,allow-write=/workspace,allow-net,allow-run=gh,git"
  }
}

3. Optional: Configure MCP Servers

Create ~/.mcp-conductor/mcp-config.json:

{
  "mcpServers": {
    "github": {
      "command": "npx",
      "args": ["-y", "@modelcontextprotocol/server-github"],
      "env": { "GITHUB_PERSONAL_ACCESS_TOKEN": "..." }
    }
  }
}

4. Start Using

The model can now execute code with all three integration options available.

Conclusion

MCP Conductor provides a secure Deno sandbox accessible via MCP's run_deno_code tool. What makes it powerful isn't complex orchestration or content-addressed storage it's flexibility:

Three ways to integrate:

1. Call MCP servers via mcpFactory (reuse ecosystem)

2. Shell out to CLI tools (use mature tooling)

3. Import npm/JSR packages (specialized functionality)

Mix them freely:

• Use GitHub MCP server for simple queries

• Use GitHub CLI for complex git operations

• Use Octokit SDK for fine-grained API control

• All in one execution, choosing the best tool for each subtask

Security Considerations

• Admin must carefully configure permissions

• Code execution always carries risk (even in sandboxes)

• Monitor executions and set reasonable timeouts

• Use --cached-only in production (no dynamic package fetching)

• Never use --allow-all

Sources & Further Reading

1. Anthropic Engineering: "Code execution with MCP: building more efficient AI agents"
   anthropic.com/engineering/code-execution-with-mcp
   Introduces the pattern of code-based MCP orchestration for token efficiency.

2. Pydantic: mcp-run-python
   github.com/pydantic/mcp-run-python
   Two-process security model for sandboxed code execution.

3. MCP Conductor Repository
   github.com/niradler/mcp-conductor
   Full documentation, examples, and security guidelines.

4. Deno Security Model
   docs.deno.com/runtime/fundamentals/security
   Capability-based permission system documentation.

That’s where AG UI Protocol and MCP UI come in. They share a similar mission—making agent-user interaction smoother—but they take very different paths to get there.

Two philosophies of agent–UI interaction

AG UI Protocol is built around event streams. Agents don’t send UI elements directly—they send events: lifecycle updates, tool calls, token-by-token messages, or state diffs. The frontend consumes those events with a client SDK (React, Vue, etc.) and re-renders the UI in real time. Think “chatting with an agent that’s typing in front of you” or “watching tool calls unfold as they happen.”

MCP UI, on the other hand, is all about embedding UI resources. Instead of events, the agent returns a UIResource that might be raw HTML, an external app URL, or a Shopify Remote DOM component tree. The client renders this in a sandbox—usually an iframe—or directly through Remote DOM. Think “the agent gives you a form, a product gallery, or a media player right inside the chat window.”

What it feels like for the user

• With AG UI Protocol, the user experiences something like Google Docs with an AI collaborator: the UI updates continuously, users can intervene mid-process, and agents can hand off tasks between each other seamlessly.

• With MCP UI, the user feels like they’re interacting with mini-apps spawned by the agent: a shopping widget, a chart viewer, a quiz form, all sandboxed for security.

Both experiences are valuable—but very different.

Where AG UI Protocol shines

If your application depends on real-time collaboration, AG UI Protocol is hard to beat. It’s already running in production at enterprises and supports multi-agent orchestration, live state synchronization, and human-in-the-loop workflows.

Use it for things like:

• AI coding assistants that stream results as you type

• Financial/trading dashboards where latency matters

• Real-time customer support where agents and humans co-pilot a session

• Any system that requires streaming state across multiple agents or clients

The ecosystem is strong, with SDKs in TypeScript and Python, and integrations with LangGraph, CrewAI, Mastra, and others. If you’re aiming for production stability and scalability, AG UI Protocol is the pragmatic choice.

Where MCP UI makes sense

MCP UI is younger, but don’t underestimate it—especially with Shopify’s Remote DOM backing. It’s purpose-built for teams already inside the MCP ecosystem who want to enrich agent responses with actual UI components, not just text.

It’s ideal for:

• E-commerce experiences (product carousels, checkout forms)

• Interactive data visualizations embedded directly in chat

• Content management or education tools where interactivity is more important than raw speed

• Teams experimenting with new UI paradigms inside MCP

If your priority is visual richness and UI extensibility, and you’re comfortable with evolving APIs, MCP UI is the better fit.

Quick Comparison Table

Thinking in trade-offs

So how do you decide? It comes down to what matters more for your product:

• Do you need streaming, low-latency collaboration with enterprise reliability? → Go with AG UI Protocol.

• Do you want to give users embedded, sandboxed components inside responses—forms, widgets, visual apps—especially if you’re already using MCP? → Try MCP UI.

Some teams may even combine them: AG UI Protocol for the real-time backbone, MCP UI for rich component embedding where visual interactivity adds value.

I built a Cursor rule file that turns an AI into a collaborative SRE copilot, able to run kubectl, helm, argocd, istioctl, and more, all under your control.

Why Give AI Your SRE Toolbox?

As human SREs, we already have the skills and instincts. But incidents eat time on repetitive steps:

• Checking service discovery & endpoints

• Inspecting Istio routes

• Pulling Helm diffs

• Reading logs & events

• Hunting down recent config changes

AI can do those steps fast, and with our approval, so we can focus on judgment calls, not keystrokes.

What is a .cursorrules File?

.cursorrules files are configuration files that provide Cursor AI with specific, always-on instructions.
Think of them as the blueprint for how the AI should behave, what tools it should use, and how it should structure its responses.

Inside the Cursor Rule File

We designed the rule file to make the AI:

1. Context-Aware

   • First confirm cluster/context, namespace, and affected service/workload.

   • Detect which tools are installed in the cluster.

2. Structured in Reasoning

   • Always produce a Diagnostic Plan before running anything.

   • Steps cover service health, routing, workload status, logs/events, and recent changes.

3. Safe by Default

   • Read-only diagnostics first.

   • Approval gates before any change.

   • Rollback plans for every mitigation.

4. Multi-Tool Fluent

   • If Istio exists, check routing.

   • If ArgoCD is present, diff last known good state.

   • If Helm is installed, compare release histories.

5. Human in the Loop

   • You approve every action.

   • The AI explains what it’s doing, no black boxes.

Why It’s Structured This Way

• Consistency: Every incident starts with the same high-quality checklist, so nothing gets missed.

• Transparency: The “Diagnostic Plan” makes the AI’s thought process visible, not hidden.

• Tool-Agnostic: The AI can dynamically use whatever’s in your cluster, from plain kubectl to full GitOps pipelines.

• Safety: Two approval gates (diagnostics, then changes) mean no accidental breakage.

• Auditability: Each command is logged with the reason it’s being run.

Here’s the AI in Action

Below is an actual run from Cursor using our rule file, in this case, debugging a CrashLoopBackOff issue.

User: help me debug the issue
kubectl get pods -n scenario-1753115617359
(Output shows pod in CrashLoopBackOff)

Cursor’s Structured Diagnosis

• Found pod restarting 424 times over 35 hours.

• Pulled logs & inspected related secret.

• Discovered APP_MODE was set to broken instead of safe.

• Confirmed the app exits immediately if APP_MODE is not safe.

Proposed Mitigation Options

1. Patch Secret (recommended):

    kubectl patch secret availability-issue-bad-secret-change-a37a6159-secret \
      -n scenario-1753115617359 \
      -p '{"data":{"APP_MODE":"c2FmZQ=="}}'
   

2. Create a new secret and update the deployment.

3. Update deployment to hardcode APP_MODE temporarily.

Rollback plan included. Awaited approval before doing anything.

The full chat is at chat.md.

The Takeaway

By letting AI use the same battle-tested tools you already rely on, but with a structured, safe, and transparent process, you turn it into a real teammate.

You still own the decisions.
You just get there faster.

The full .cursorrules file.

Feel free to visit komodor.com for more info on AI SRE.

That’s why we built Portable MCP Manager - an open-source, cross-platform CLI tool that makes managing, sharing, and syncing MCP configurations simple and reliable.

✨ Why Portable MCP Manager?

Portable MCP Manager solves a common problem for developers and power users who rely on MCP-based tools. Instead of juggling files and paths across environments, you now have a single, streamlined way to:

• 🔄 Replace or merge configs from multiple sources

• 🔗 Fetch configs directly from URLs or GitHub Gists (even private ones)

• 📱 Manage configs for Cursor and Claude Desktop

• 🖥️ Run seamlessly on Windows and macOS

• 📤 Upload configs back to GitHub Gists with ease

• 🎯 Use an interactive mode if you prefer guided setup

• 🔀 Handle multi-file gists without hassle

• 🛠️ Take advantage of GitHub CLI if installed

⚡ Getting Started

Install it globally via npm:

npm install -g portable-mcp

Or just run it on-demand with npx:

npx portable-mcp --help

🚦 Quick Start

1. Interactive Mode (Beginner-Friendly)

portable-mcp

2. Replace a Configuration from Gist

portable-mcp replace --type cursor --gist 50007c6cd60db13cf8477b3b5caa96f0

3. Merge Configurations

portable-mcp merge --type claude --gist abc123def456

4. Upload Config to Gist

portable-mcp store --type cursor --private

🔍 Example Workflows

Complete Sync Cycle

# Check default config path
portable-mcp path --type cursor  

# Replace with a config from Gist
portable-mcp replace --type cursor --gist abc123def456  

# Upload updated config back
portable-mcp store --type cursor --private

Multi-file Gist Example

portable-mcp replace --type claude --gist abc123def456/claude.json

🔗 GitHub Integration

• If you have GitHub CLI installed, the tool uses it automatically.

• Otherwise, set a GITHUB_TOKEN for direct API uploads.

export GITHUB_TOKEN=ghp_your_personal_access_token_here

📁 Supported Config Paths

• Cursor

  • Windows: C:\Users\<username>\.cursor\mcp.json

  • macOS: ~/Library/Application Support/Cursor/User/mcp.json

• Claude Desktop

  • Windows: C:\Users\<username>\AppData\Roaming\Claude\claude_desktop_config.json

  • macOS: ~/Library/Application Support/Claude/claude_desktop_config.json

Source code: https://github.com/niradler/portable-mcp

• You ask your LLM to add a library.

• It writes npm install some-package@1.0.0… but the latest version is actually 3.2.4.

• Or worse, it invents a version that doesn’t even exist.

Suddenly, instead of building features, you’re wrestling with dependency mismatches.

I built the Dependency MCP Server to solve exactly this problem.

What It Does

The Dependency MCP is an MCP (Model Context Protocol) server that lets your AI development tools check dependencies across multiple registries in real time.

That means whenever the AI suggests a package, it can instantly verify:

• ✅ What the latest version is

• ✅ Whether a specific version exists

• ✅ Full package metadata including all available versions

• ✅ Run bulk checks across your entire dependency list

It supports all the major registries: npm, PyPI, Maven, NuGet, RubyGems, Crates.io, and Go modules.

So whether you’re in Node.js, Python, Java, .NET, Ruby, Rust, or Go, the AI never needs to guess again.

Why It Matters

This isn’t just about convenience. Correct dependencies make a huge difference in:

• Reducing errors – no more wasted time debugging phantom versions.

• Faster development – the AI can give you working install commands immediately.

• CI/CD reliability – bulk validation tools let you enforce correct versions across pipelines.

• Security audits – you can fetch full package metadata for reviews.

Instead of trusting the model’s memory (which is always a little stale), you give it a direct way to ask the source of truth: the package registries themselves.

Fun Fact: How I Use It with Cursor

One of the coolest things about building MCP servers is that you can actually use Cursor’s own MCP integration to create an automatic development feedback loop.

The flow looks like this: you start by writing or updating your MCP server, then register it in Cursor’s configuration. Once it’s wired up, Cursor can immediately call your server and test each of the tools you’ve exposed. If something doesn’t work, Cursor surfaces the error right away - and even suggests fixes or improvements directly in your editor. That means you’re not just coding; you’re effectively building software that validates itself, with an AI co-pilot constantly reviewing and stress-testing your MCP server in real time.

Try It Out

Wire it into your AI tooling (like Claude Desktop or Cursor), just drop this into your config:

{
  "mcpServers": {
    "dependency-checker": {
      "command": "npx",
      "args": ["dependency-mcp"]
    }
  }
}

Closing Thoughts

Building with AI is amazing - but only if the code it writes actually runs. With the Dependency MCP server, you never have to worry about outdated or invalid dependencies again.

For me, this has completely changed the dev cycle: instead of correcting the AI, the AI corrects itself.

The reality of modern development has fundamentally shifted. Studies show that over 80% of new code is now generated or co-written by AI assistants like Claude, GPT-4, and Copilot. But here's the problem: LLMs generate code without knowing if it actually compiles, passes tests, or meets quality standards.

This creates a dangerous gap between code generation and code validation that traditional development workflows weren't designed to handle.

Code Feedback MCP Server bridges this gap by providing LLMs with the real-time feedback they need to generate better code, catch their own mistakes, and iteratively improve until the code meets production standards.

The LLM Code Generation Revolution (And Its Problem)

The shift to AI-generated code has been dramatic:

• Volume: Developers report 40-60% of their code is now AI-generated

• Speed: What took hours now takes minutes with AI assistance

• Scope: LLMs can generate entire modules, APIs, and applications

• Languages: AI excels across TypeScript, Python, Go, and more

But this revolution comes with a critical flaw:

LLMs Generate Code Blind

When an LLM writes code, it has no way to know:

• ❌ Does the code actually compile?

• ❌ Are there syntax or type errors?

• ❌ Do the tests pass?

• ❌ Does it follow project conventions?

• ❌ Are there security vulnerabilities?

• ❌ Is the performance acceptable?

The result? Developers spend significant time debugging and fixing AI-generated code, often losing the productivity gains that AI promised to deliver.

The Solution: Real-Time AI Code Validation & Auto-Correction

Code Feedback MCP Server creates the essential feedback loop for AI-generated code by providing:

🤖 LLM-First Architecture

• Instant feedback: LLMs get immediate validation results after code generation

• Structured responses: JSON format that LLMs can parse and act upon

• Error descriptions: Detailed explanations that help LLMs understand and fix issues

• Iterative improvement: Enable LLMs to generate → validate → fix → repeat until perfect

🔄 The AI Quality Loop

• Generate: LLM creates code based on requirements

• Validate: Code Feedback MCP tests compilation, syntax, and quality

• Analyze: Advanced prompts provide detailed feedback and suggestions

• Iterate: LLM uses feedback to automatically improve the code

• Verify: Final validation ensures production readiness

🧠 Multi-Language AI Validation

• TypeScript/JavaScript: Catch type errors that confuse LLMs

• Python: Detect linting issues LLMs commonly miss

• Go: Ensure compilation and formatting standards

• Extensible for any language your LLMs work with

🔄 Auto-Correction Capabilities

• Smart error reporting: LLMs understand exactly what went wrong

• Fix suggestions: Prompts provide specific guidance for improvements

• Iterative refinement: LLMs can automatically apply fixes and re-validate

• Quality enforcement: Ensure AI-generated code meets your standards

⚡ Developer Experience First

• Cross-platform support (Windows, macOS, Linux)

• Simple configuration with mcp-config.json

• Comprehensive error reporting with actionable feedback

• Integration with popular editors and CI systems

Game-Changing: AI That Fixes Its Own Code

The Old Way (Broken):

Human: "Create a TypeScript API handler"
LLM: *Generates code with type errors*
Human: *Discovers errors during manual testing*
Human: "Fix these 5 compilation errors"  
LLM: *Attempts fixes, introduces new issues*
Human: *Repeats cycle multiple times*

The New Way (Code Feedback MCP):

Human: "Create a TypeScript API handler"
LLM: *Generates code*
LLM: *Automatically validates with Code Feedback MCP*
Code Feedback MCP: *Returns structured error feedback*
LLM: *Automatically fixes issues based on feedback*
LLM: *Re-validates until compilation succeeds*  
Human: *Receives working, tested, quality code*

Here's what this looks like in practice:

Step 1: Initial Generation

// LLM generates this code
export function handleRequest(req: Request): Response {
  const data = req.body.parse(); // Error: parse() doesn't exist
  return { status: 200, data }; // Error: wrong return type
}

Step 2: Automatic Validation The AI immediately checks if the code actually works by running it through the validation system.

Step 3: Smart Feedback Instead of cryptic error messages, the AI gets clear, actionable feedback:

• "Hey, parse() doesn't exist on request bodies - try json() instead"

• "This return type won't work - you need to return a proper Response object"

Step 4: LLM Auto-Correction

// LLM automatically fixes based on feedback
export async function handleRequest(req: Request): Promise<Response> {
  const data = await req.body.json();
  return new Response(JSON.stringify({ data }), { 
    status: 200,
    headers: { 'Content-Type': 'application/json' }
  });
}

Step 5: Success! The AI validates again and confirms everything works perfectly. No more broken code!

Advanced AI Code Intelligence: Beyond Basic Validation

The real breakthrough is the AI-powered prompt system that enables LLMs to perform sophisticated code analysis and self-improvement:

🔍 Intelligent Code Review

Think of this as having a senior developer review your AI's code instantly. The LLM can ask for detailed feedback on any code it generates, focusing on specific areas like performance, security, or maintainability.

🛡️ Automated Security & Bug Detection

Your AI can now audit its own code for vulnerabilities and common mistakes - catching issues that even experienced developers sometimes miss.

🚀 Performance Optimization

The LLM can analyze its own code for performance bottlenecks and automatically implement optimizations. It's like having a performance expert built right into your coding workflow.

Transformative Use Cases for AI Development

1. Autonomous Code Generation & Validation

LLMs can now generate complete, working features without human intervention:

Human: "Build a REST API for user management with TypeScript"

AI Process:
1. Generate initial code structure
2. Validate with Code Feedback MCP → Find compilation errors
3. Auto-fix type issues and re-validate
4. Run security audit → Detect missing input validation
5. Add validation and re-audit
6. Performance analysis → Optimize database queries
7. Final validation → All checks pass

Result: Production-ready code delivered in minutes, not hours.

2. Smart Code Improvement

Instead of just accepting the first code an AI generates, the LLM can continuously improve existing code by asking for refactoring suggestions, then automatically applying and testing improvements.

3. Intelligent Problem Solving

When the AI hits an error (like a missing dependency), it can automatically diagnose and fix the issue - installing packages, updating configurations, or correcting code - then continue with the original task seamlessly.

4. Full-Stack Project Management

The AI can work across different programming languages in the same project, ensuring everything works together. Generate a Python backend, TypeScript frontend, and Go microservice - all validated and tested as a complete system.

The Future is Here: AI That Actually Works

Here's what's really exciting - LLMs can now handle the complete development cycle:

✅ Generate code from your ideas and requirements
✅ Test compilation and fix syntax errors instantly
✅ Run and validate tests to ensure functionality
✅ Check for security issues and patch vulnerabilities
✅ Optimize performance based on real analysis
✅ Maintain quality standards consistently
✅ Handle project setup and dependencies automatically

This isn't some far-off future - it's working right now.

Why This Changes Everything

The biggest pain point in AI coding has always been the back-and-forth debugging dance:

1. Ask AI to write code

2. Copy code and try to run it

3. Hit errors and spend time figuring out what's wrong

4. Go back to AI with error messages

5. Repeat until something works (maybe)

Code Feedback MCP cuts through all of that. The AI can now test, debug, and perfect its code automatically, giving you working solutions on the first attempt.

Ready to Supercharge Your AI Development?

Code Feedback MCP Server is the missing infrastructure for reliable AI-generated code. Whether you're building with Claude, GPT-4, or any other LLM, this tool ensures your AI can generate production-ready code autonomously.

Perfect for:

• 🤖 AI-First Development Teams seeking autonomous code generation

• 🚀 Startups moving fast with AI-generated features

• 🏢 Enterprise Teams needing quality assurance for AI code

• 👨‍💻 Individual Developers maximizing AI productivity

• 🔧 DevTools Builders creating intelligent development experiences

Get started today:

• 📚 GitHub Repository

Contributing is welcome! Add support for new languages, improve existing tools, or enhance the prompt system. Every contribution makes the tool better for the entire community.

The era of unreliable AI-generated code is over. With Code Feedback MCP, your LLMs can generate, validate, and fix code autonomously — delivering production-ready solutions that just work. Join the autonomous development revolution today.

Tags: #LLM #AICode #CodeGeneration #MCP #DevTools #TypeScript #Python #Go #Automation #CodeQuality #OpenSource

Remember the last time you tried to find something in your company's project management tool? You probably clicked through three different menus, filtered a table, exported a CSV, and then realized you were looking at last month's data. Sound familiar?

We've all been there. Modern business software has become a maze of tabs, dashboards, and dropdown menus that would make even a GPS jealous. But what if I told you there's a better way—one that feels as natural as asking a colleague for help?

The Problem with Today's Software

Think about your typical workday. You probably juggle between 5-10 different software tools: your CRM, project manager, analytics dashboard, HR system, and who knows what else. Each one has its own logic, its own way of organizing information, and its own special quirks that you have to remember.

It's like having to learn a new language for every room in your house. Want to check the thermostat? Better remember the HVAC control system. Need to turn on the TV? Time to decipher that remote with 47 buttons.

The result? Most people use maybe 20% of their software's capabilities. The other 80% remains hidden behind menus they've never clicked or features they don't know exist.

Enter the Chat-First Revolution

Now imagine this: instead of clicking through endless menus, you simply type or say what you need, just like you would ask a knowledgeable coworker.

Instead of:

1. Navigate to Reports → Customer Analytics → Filter by Date Range → Select Metrics → Export to Excel → Open Excel → Create Charts

You simply say: "Show me this month's customer growth by region"

And boom—there's your answer, complete with charts, insights, and follow-up suggestions.

Why Conversation Changes Everything

1. No More Hide-and-Seek with Features

Ever spent 10 minutes looking for a feature you used last week? With chat-first software, you don't need to remember where things are—you just ask for what you need. It's like having a super-smart assistant who knows every corner of your business tools.

2. Context That Actually Matters

Traditional software treats every click as a fresh start. But conversations have memory. When you ask about "those customers from yesterday's discussion," the system remembers. When you say "create a similar report," it knows what you're referring to.

3. Learning as You Go

The best part? Chat-first tools get smarter the more you use them. They learn your patterns, suggest shortcuts, and even anticipate what you might need next. It's like having software that actually pays attention.

Real-World Magic: How It Actually Works

Let's look at a practical example. Imagine you're a customer success manager dealing with a tricky situation:

Traditional Way:

• Log into CRM → Search customer → Open profile → Check ticket history → Switch to billing system → Verify payment status → Open communication tool → Draft email → Switch back to CRM → Update customer notes

Chat-First Way: Simply type: "Customer TechCorp seems unhappy, help me understand what's going on"

The system responds with everything you need: recent tickets, payment history, communication timeline, and even suggests next steps—all in one conversation.

But What About Power Users?

"This sounds nice for beginners," you might think, "but I'm fast with the current system."

Here's the thing: chat-first doesn't mean dumbed-down. Power users can create custom shortcuts, chain multiple commands together, and even build their own workflows through conversation. Think of it as upgrading from clicking buttons to having a conversation with an expert who never gets tired or forgets details.

You can still get your data fast—actually faster—but now your colleagues can too, without a 3-hour training session.

The Ripple Effect on Teams

When software becomes conversational, something interesting happens to teams:

Knowledge Sharing Gets Effortless: Instead of "How do I run that report again?" conversations, team members can just ask the system directly.

Onboarding Becomes Natural: New hires don't need to memorize complex workflows—they learn by asking questions, just like they would with a mentor.

Collaboration Improves: When everyone can access information through natural conversation, meetings become about decisions, not data-gathering.

Beyond the Hype: Real Benefits

This isn't just about being trendy or following the latest tech fad. Chat-first software addresses real business problems:

• Reduced Training Costs: Less time teaching people where buttons are, more time on actual work
• Fewer Mistakes: When software guides you through processes conversationally, you're less likely to miss steps
• Better Decision Making: Faster access to information means decisions based on current data, not last week's report
• Happier Employees: Less time fighting with software, more time doing meaningful work

What This Means for Your Business

We're not talking about replacing every tool overnight. This is about reimagining how people interact with business software. Instead of adapting to rigid interfaces, software adapts to how people naturally communicate.

Think about it: when you need information from a colleague, you don't hand them a form to fill out—you have a conversation. Why should software be any different?

The Future Is Conversational

This shift is already happening. The companies that recognize it early will have teams that are more efficient, more collaborative, and frankly, happier with their tools.

The question isn't whether chat-first software will become mainstream—it's whether your organization will be an early adopter or playing catch-up.

Coming up next in our series: "From Dashboards to Dialogues: The Technical Magic Behind Chat-First Software" where we'll explore how this actually works behind the scenes.

What do you think? Have you ever wished you could just ask your software what you need instead of hunting for it? Share your biggest software frustration in the comments—you might be surprised how many others share it.

This is Part 1 of our Chat-First SaaS series. Follow along as we explore how conversation is transforming business software, one chat at a time.

This comprehensive analysis examines the leading GPU inference servers: NVIDIA Triton Inference Server, Text Generation Inference (TGI), vLLM, and Ollama.

What Are Inference Servers? A Primer for AI Practitioners

If you're working with AI models but haven't yet deployed them in production, you might wonder: "Why do I need an inference server when I can just run my model directly?" The answer lies in the gap between research/development and production deployment.

Core Features Every Inference Server Provides

1. Concurrent Request Handling 🔄

What it does: Serves multiple users simultaneously instead of processing one request at a time.

Why you need it: Your Jupyter notebook can't handle 1,000 users hitting your model at once. Inference servers use queuing, batching, and resource management to serve multiple requests efficiently.

Real impact: Transform from serving 1 user to serving 1,000+ concurrent users.

2. Dynamic Batching 📦

What it does: Automatically groups individual requests into batches for more efficient GPU utilization.

Why you need it: GPUs are designed for parallel processing. Processing requests one-by-one wastes 90%+ of your expensive GPU resources.

Example: Instead of processing 10 text requests individually, the server batches them together, reducing inference time from 10 seconds to 2 seconds total.

3. Model Optimization ⚡

What it does: Automatically optimizes your model for faster inference through quantization, kernel fusion, and memory layout optimization.

Why you need it: Your research model might run fine on your laptop but be too slow/expensive for production. Inference servers can make models 2-10x faster without code changes.

Techniques include:

• Quantization: Converting FP32 models to FP16 or INT8 (2-4x memory reduction)
• Kernel Fusion: Combining operations to reduce GPU memory transfers
• Memory Layout Optimization: Reorganizing data for faster access

4. Auto-Scaling 📈

What it does: Automatically spins up/down server instances based on demand.

Why you need it: Your AI app might have 10 users at 3 AM but 10,000 users at peak hours. Manual scaling is impossible.

Cost impact: Pay for resources only when needed, potentially reducing infrastructure costs by 60-80%.

5. Health Monitoring & Observability 📊

What it does: Tracks model performance, latency, throughput, error rates, and resource usage.

Why you need it: When your model starts giving wrong answers or becomes slow, you need to know immediately, not when users complain.

Metrics tracked:

• Requests per second
• Average latency (P50, P95, P99)
• Error rates
• GPU/CPU utilization
• Memory usage

6. A/B Testing & Model Versioning 🧪

What it does: Allows you to test new model versions against existing ones with real traffic.

Why you need it: You've trained a new model version that performs better in testing, but will it perform better with real users? Inference servers let you route 10% of traffic to the new model to compare performance.

7. Caching & Request Deduplication 💾

What it does: Stores results of common requests and detects duplicate requests to avoid redundant computation.

Why you need it: If 100 users ask "What's the weather like?", why run inference 100 times? Caching can reduce compute costs by 30-70% for many applications.

Inference Server Comparison

1. Text Generation Inference (TGI) 🚀

Developer: Hugging Face
Specialty: Production-ready LLM serving with enterprise focus

Key Strengths:

• Hugging Face Ecosystem Integration: Seamless compatibility with HF model hub and datasets
• Production-Ready Architecture: Built for high-throughput, low-latency production environments
• Advanced Quantization: Supports FP16 and INT8 quantization for memory optimization
• Kubernetes-Native: Designed for cloud-scale deployments with auto-scaling capabilities
• Asynchronous Processing: Handles high-volume concurrent requests efficiently

Performance Profile:

• Best For: Text generation, chatbots, customer support systems
• Memory Management: Excellent with FP16/INT8 quantization
• Batch Processing: Full support with dynamic batching
• Scaling: Enterprise-grade Kubernetes integration

Real-World Applications:

TGI excels in customer support chatbots where consistent response times and automatic scaling based on demand fluctuations are crucial. Its tight integration with Hugging Face makes it ideal for teams already invested in the HF ecosystem.

Benchmarking Results:

• MPT-30B achieved 35.43 tokens/second with a remarkable 36.23% performance increase over TensorRT-LLM in specific configurations

2. vLLM (Very Large Language Models) ⚡

Developer: UC Berkeley
Specialty: Memory-efficient inference with innovative architecture

Revolutionary Features:

• PagedAttention: Breakthrough memory management technique that optimizes GPU memory usage
• Token Parallelism: Reduces memory requirements by breaking inference into manageable tokens
• Dynamic Batching: Automatic batch size optimization based on available resources
• Multi-GPU Distribution: Efficient model distribution across multiple GPUs

Performance Profile:

• Best For: Large-scale LLM inference in resource-constrained environments
• Memory Efficiency: Industry-leading memory optimization
• Cost Optimization: Ideal for educational and enterprise applications focused on cost efficiency
• GPU Utilization: Maximizes throughput while minimizing memory waste

Benchmarking Highlights:

• SOLAR-10.7B: Peak performance of 57.86 tokens/second
• Qwen1.5-14B: 46.84 tokens/second, consistently outperforming Triton configurations
• Strong performance across multiple model sizes, often matching or exceeding TensorRT-LLM

Why It's Gaining Traction:

The Reddit community notes that "vLLM is catching up with TensorRT-LLM" in performance while maintaining superior user-friendliness and memory efficiency.

3. NVIDIA Triton Inference Server 🏢

Developer: NVIDIA
Specialty: Enterprise-grade multi-model inference platform

Enterprise-Grade Features:

• Framework Agnostic: Supports PyTorch, TensorFlow, ONNX, and custom backends
• Multi-Model Serving: Deploy multiple models simultaneously on a single server
• Model Ensembles: Chain models together for complex AI pipelines (e.g., text-to-vision workflows)
• NVIDIA Hardware Optimization: Maximum performance on NVIDIA GPU stack
• Dynamic Batching: Efficient GPU utilization through intelligent batching

Performance Profile:

• Best For: Enterprise environments requiring diverse model deployments
• Versatility: Handles everything from recommendation engines to image classification
• Integration: Deep NVIDIA ecosystem integration
• Scalability: Multi-GPU scaling with model parallelism

Real-World Applications:

Triton dominates enterprise settings where multiple AI models need deployment across diverse workloads. It's particularly strong in recommendation engines, image classification pipelines, and NLP applications requiring high throughput.

Community Developments:

Active development of Triton-co-pilot projects to streamline model deployment and conversion processes, making Triton more accessible to developers.

4. Ollama 🛠️

Developer: Ollama Team
Specialty: Developer-friendly local LLM deployment

Developer-Centric Features:

• LLaMA Optimization: Specifically designed for LLaMA-based models
• Cross-Platform: Seamless operation on macOS, Windows, and Linux
• Zero-Setup Philosophy: Minimal configuration required for rapid prototyping
• CLI and API Support: User-friendly command-line interface with comprehensive API
• Local and Cloud Flexibility: Deploy models locally or scale to cloud environments

Performance Profile:

• Best For: Rapid prototyping, small teams, solo developers
• Learning Curve: Extremely accessible for developers new to LLMs
• Deployment Speed: Fastest time-to-deployment for LLaMA models
• Resource Requirements: Optimized for resource-conscious environments

Ideal Use Cases:

Perfect for developers creating language analysis tools, personal AI assistants, and research-focused applications. Its ease of use makes it popular among smaller teams and individual developers who need quick LLaMA model deployment.

Performance Comparison Matrix

Choosing the Right Solution: Decision Framework

Choose vLLM if:

• Memory efficiency is critical
• You're running large models (13B+ parameters)
• Cost optimization is a primary concern
• You need cutting-edge performance with user-friendly deployment

Choose TGI if:

• You're heavily invested in the Hugging Face ecosystem
• Production reliability and enterprise features are essential
• You need robust quantization support
• Kubernetes-native deployment is required

Choose Triton if:

• You're running diverse model types (not just LLMs)
• Enterprise multi-model deployment is needed
• You require model ensemble capabilities
• NVIDIA hardware optimization is crucial

Choose Ollama if:

• You're prototyping with LLaMA models
• Rapid deployment with minimal setup is priority
• You're working in small teams or as an individual developer
• Cross-platform compatibility is important

Performance Benchmarking Insights

Key Findings

1. vLLM's Rising Performance: Consistently competitive with TensorRT-LLM while maintaining superior usability
2. TGI's Specialized Strength: Exceptional performance on specific model types (MPT-30B showed 36% improvement)
3. Triton's Versatility: Strong across diverse workloads but slightly behind in pure LLM inference
4. Memory Efficiency Leader: vLLM's PagedAttention provides the best memory utilization

Future Outlook and Recommendations

The GPU inference server landscape is rapidly evolving, with each solution addressing different market needs:

For Startups and Scale-ups: vLLM offers the best balance of performance, cost efficiency, and ease of use.

For Enterprise Deployments: Triton provides the most comprehensive feature set for complex, multi-model environments.

For Hugging Face-Centric Teams: TGI remains the natural choice with its ecosystem integration and production readiness.

For Rapid Prototyping: Ollama continues to excel in developer velocity and accessibility.

Conclusion

There's no universal "best" GPU inference server—the optimal choice depends on your specific requirements, technical constraints, and organizational context. The good news is that all major solutions are actively developed and continuously improving, ensuring robust options regardless of your chosen path.

Consider running your own benchmarks with your specific models and infrastructure to make the most informed decision. The performance landscape is dynamic, and what works best today may evolve as these technologies mature.

Picture this: It's 3 AM. Your Slack is blowing up. The deployment is broken. Again.

Developer: "I just need to deploy my app! Why do I need 47 YAML files?!"

DevOps: "You can't just wing it! Where's your ServiceAccount? Your network policies? Your monitoring setup?!"

Developer: "I don't know what half of those words mean!"

DevOps: screams into the void

Sound familiar? If you've worked with Kubernetes, you've lived this nightmare. Developers want simplicity. DevOps wants control. Kubernetes gives you... complexity.

But what if I told you there's a new sheriff in town that's making everyone happy? 🤠

Enter Kro: The Kubernetes Peacekeeper 🏴‍☠️

Kro (Kube Resource Orchestrator) is like that cool mediator friend who helps feuding roommates finally get along. It's an open-source, Kubernetes-native tool that lets you create custom APIs that are actually... wait for it... simple to use.

Here's the magic: DevOps teams define the complex stuff once, developers get a clean, simple API to work with. Everyone wins!

How Kro Works Its Magic ✨

For DevOps Teams (The Platform Heroes):

You create a ResourceGraphDefinition that bundles all your best practices:

# One definition to rule them all
apiVersion: kro.run/v1alpha1
kind: ResourceGraphDefinition
metadata:
  name: my-application
spec:
  schema:
    kind: Application  # This becomes your new simple API!
    spec:
      name: string
      image: string | default="nginx"
      ingress:
        enabled: boolean | default=false

  resources:
    - id: deployment    # Your carefully crafted Deployment
    - id: service      # Your perfectly configured Service  
    - id: ingress      # Your security-compliant Ingress
    # + all your monitoring, RBAC, policies, etc.

You get to:

• ✅ Enforce security best practices

• ✅ Standardize across teams

• ✅ Include all the "boring" stuff (monitoring, RBAC, etc.)

• ✅ Sleep better at night

For Developers (The Feature Builders):

You get a beautifully simple API:

# That's it. That's the whole deployment.
apiVersion: kro.run/v1alpha1
kind: Application
metadata:
  name: my-awesome-app
spec:
  name: my-awesome-app
  image: my-app:v1.2.3
  ingress:
    enabled: true

You get to:

• ✅ Deploy with confidence

• ✅ Focus on your application logic

• ✅ Stop googling "kubernetes service yaml example" for the 847th time

• ✅ Actually ship features

The Secret Sauce: CEL Expressions 🌶️

Kro uses CEL (Common Expression Language) to wire everything together intelligently. It's like having a smart assistant that knows:

• "Oh, the service needs to point to the deployment? I got you."

• "User wants ingress? Let me create that AND wire it to the service."

• "Need to pass values between resources? On it."

The best part? Kro automatically figures out the dependency order. No more "Service created before Deployment" errors!

What This Actually Means 💡

Before Kro:

Developer: "I need to deploy my microservice" DevOps: "Okay, here's a 200-line YAML template. Don't forget the ServiceAccount, NetworkPolicy, PodDisruptionBudget, HorizontalPodAutoscaler, ServiceMonitor, and..." Developer: eye twitching intensifies

After Kro:

Developer: "I need to deploy my microservice" DevOps: "Cool, just set your app name and image in the Application API" Developer: "That's... it?" DevOps: "Yep! All the security and monitoring stuff is already baked in" Developer: "I... I love you"

Why This Changes Everything 🚀

1. Developers Deploy Fearlessly: No more broken deployments because someone forgot a label

2. DevOps Sleeps Soundly: Standards are enforced automatically

3. Platform Teams Become Heroes: Instead of gatekeepers, they're enablers

4. Compliance Teams Rejoice: Best practices are built-in, not bolt-on

5. Everyone Ships Faster: Less time debugging YAML, more time building features

"But What About Helm?" 🎭

I see you, Helm users! Yes, Helm charts can package resources together, but here's the thing: Helm is still about managing YAML templates. With Kro, you're creating actual Kubernetes APIs.

Think of it this way:

• Helm: "Here's a templated way to deploy complex YAML"

• Kro: "Here's a simple API that happens to create complex resources"

You can even use them together! Your ResourceGraphDefinition could deploy Helm charts if that's your jam. Kro plays nice with everyone. 🤝

Getting Started 🏁

Want to try Kro? It's surprisingly simple:

1. Install Kro in your cluster

2. Platform team creates ResourceGraphDefinitions

3. Developers use the shiny new APIs

4. Profit! (and inner peace)

The best part? It works with your existing tools and processes. No need to rip and replace everything.

The Bottom Line 📝

Kro doesn't just solve technical problems—it solves people problems. It gives developers the simplicity they crave while giving DevOps the control they need. It's like relationship therapy, but for Kubernetes teams.

#Kubernetes #DevOps #DeveloperExperience #Kro #PlatformEngineering #CloudNative #TeamWork #YAML #APIs

We trained our first expert model in Kubernetes kubectl commands using a fine-tuning process on a specialized dataset. The dataset consists of structured prompt-command pairs where natural language queries are mapped to their respective kubectl commands. This allows the model to generalize and generate accurate kubectl commands for a variety of Kubernetes management tasks.

Why Fine-Tune a Model for kubectl?

General-purpose AI models lack deep Kubernetes knowledge and struggle with domain-specific queries. Our goal is to fine-tune a model that can:

• Generate correct kubectl commands based on natural language input.

• Explain command syntax and usage with step-by-step reasoning.

• Identify potential errors and suggest fixes.

Our Decision-Making Process

1. Model Selection: We use LLaMA 3.2-3B-Instruct, which balances efficiency and accuracy. It runs efficiently with 4-bit quantization, making it feasible for fine-tuning on consumer-grade GPUs.

2. Fine-Tuning with Unsloth: We streamline model training with LoRA adapters, reducing memory usage and improving efficiency.

3. Dataset Curation: We use multiple datasets to cover different aspects of kubectl usage:

   • Basic Commands: ComponentSoft/k8s-kubectl

   • Advanced Scenarios: ComponentSoft/k8s-kubectl-35k

   • Chain-of-Thought (CoT) Explanations: ComponentSoft/k8s-kubectl-cot-20k

   • Instruction-based Training: sozercan/k8s-instructions

   • Troubleshooting & Debugging: eliashasnat/k8s-qa

4. Deployment with Ollama: Ollama was selected for easy packaging and distribution, making it seamless to integrate into terminal-based applications.

Training Process

1. Dataset Collection: We compiled a comprehensive dataset of kubectl commands mapped to natural language queries, covering a wide range of Kubernetes operations.

2. Preprocessing: We cleaned and structured the data to ensure consistency, removing redundant entries and normalizing formatting.

3. Fine-Tuning: We optimized the model using instruction-tuning techniques, allowing it to generate precise kubectl commands from user queries.

4. Validation: The trained model was rigorously tested against various Kubernetes scenarios to ensure accuracy and reliability.

5. Deployment: The final model was packaged as niradler/k8s-operator:latest for easy use within the KasK application.

Integrating the Model into KasK

KasK - Kubernetes Assistant Terminal App

KasK is an open-source terminal-based application designed to simplify your interaction with Kubernetes clusters. With KasK, you can ask natural language questions about your Kubernetes resources, and it will generate accurate kubectl commands to fetch the required details. The app also provides a JSON viewer to display and explore the command output in a structured and user-friendly way.

Features

• Natural Language Queries: Ask questions like "Show all running pods" or "List services in all namespaces," and KasK will generate the appropriate kubectl command.

• JSON Viewer: View the output of kubectl commands in a tree-like structure with search and filtering capabilities.

• Clipboard Integration: Copy selected JSON values to your clipboard for easy sharing or further use.

• Dark Mode Support: Enhanced readability with dark mode styles.

• Keyboard Shortcuts: Navigate and interact with the app efficiently using intuitive key bindings.

Usage

git clone https://github.com/niradler/kask.git
cd kask
pip install -r requirements.txt
python main.py

Write your query in the "Write your prompt here" text area. For example: Show all pods in the default namespace.

Click the "Prompt" button or press Enter to generate the kubectl command and view the output.

Use the JSON viewer to explore the output:

• Search for specific keys or values.

• Expand or collapse nodes.

• Copy selected values to your clipboard.

Use keyboard shortcuts for quick actions:

• x: Expand/Collapse all nodes.

• s or /: Focus on the search bar.

• c: Copy the selected value.

• q: Quit the application.

Requirements

• Python 3.8 or higher

• Ollama server with the niradler/k8s-operator:latest model:

  • Download model

  • Use the Modelfile

  • ollama create k8s-operator -f Modelfile

• kubectl installed and configured to access your Kubernetes cluster.

Features for the future:

• Limit commands (read-only) or review before execution.

• Compact table view.

• Tools integration.

• Chat memory.

• Model selector.

• Ollama configuration.

• UI/UX improvements.

• Submit prompt with keyboard.

With our first fine-tuned model powering KasK, we’re excited to continue improving Kubernetes management through AI-driven automation! 🚀