Kumar Maddali on agent permissions, ownership, and why AI can't be a black box

First Principles is a conversation series where I ask engineering and security leaders the questions that usually go unasked: how their teams actually behave, what they can't see, and what they've learned the hard way.

In this conversation, Kumar and I talked about how AI agents get permission to act, who owns an agent after it's approved, what it takes to explain an agent's decisions, and how to keep engineers motivated when many of their ideas get a no.

Watch on YouTube

Key moments

1. How agents get permission

   Agents reach other systems through tools or through other agents. Either way they need access: a system account, a service account, or specific credentials, scoped by role or by feature. He added that getting permissions right, and getting the right information to the right users at the right time, is where enterprises struggle.

   Getting the job done is only one small part of the whole agentic system.

2. Who owns an agent after it's approved

   Guardrails live in the harness built around the model, but accountability stays with people. His advice to engineers: understand what an agent produces instead of treating it as a black box. Systems thinking matters more than vibe coding.

   The ownership and accountability still should lie with the humans or the teams.

3. Explaining what an agent did

   Every action an agent takes should come with clear reasoning, and its decisions can be kept within defined limits, such as a schema.

   Explainability and observability is the key thing.

4. When engineers' ideas get a no

   A mandate without a purpose creates resistance. Explain the purpose and people come along, the way unit testing went from unpopular to standard practice. Apply the 80/20 rule and handle outliers case by case.

   If you make it a mandate without describing the purpose, it's a challenge.

5. The one thing he'd want to see

   AI productivity gains showing up as real business impact, and a culture where everyone thinks in those terms.

   It's not about the AI adoption. It's about how AI is accelerating your business.