Margin of Safety #38: The Best Enterprise Agents Probably Aren’t Fully Autonomous

Enterprise-ready agentic systems trade autonomy for control: scoped dependencies, human escalation, versioned behavior, and shared agent definitions over per-user agents

As Agentic Process Automation and/or Agentic Process Mining get more and more attention, we’re seeing a lot of speculation around what it’ll take to make them scale. We’re also seeing a lot of speculation around underlying software architectures that (we think) fail to fully recognize the implicit design tradeoffs in these hypothetical architectures. In this blog, we’ll go through some of the tradeoffs and gotchas that we see coming, and why we think that the best enterprise ready agentic automation product may not be the same as automagical AI agent that first comes to mind for many folks.

Autonomy

A key consideration is the level of autonomy given to the agent. A lot of talk is around goal-oriented agents (implying that these are Anthopic-style agents [refer to this post for more on agentic processes], in which an LLM is in control or some or all of the software’s control flow) and how they’ll take arbitrary action to achieve their goal. That sounds great, until you think about how good AI systems are at reward hacking(details here, see here for a fun example of LLMs trying it). If you set the goal of satisfied customers with a highly autonomous agent, you should expect to be giving out a lot of discount codes, customer credits, and maybe leadership phone numbers – after all, whatever it takes to make the customer happy! Worse, if you’re in a regulated industry, there are many goal definitions that, when coupled with arbitrary decision making, could result in a regulatory violation (eg, the ongoing bias lawsuit Mobley vs Workday).

Guarding against reward hacking for a highly autonomous system is hard. You either need the ability to perfectly quantify a good result in a timely fashion, or you need the ability to see around corners and figure out every possible way the system might ‘cheat’ your scoring. The former is often impossible in business contexts – for example, things that impact lifetime customer value can be slow and probabilistic to measure, meaning they only appear over time and in aggregate. The latter though implies an ability to avoid unknown unknowns in your system, something that’s a brave assumption for complicated processes.

So where does this leave us? We think that for important processes, business will achieve enterprise readiness by avoiding highly autonomous, broadly permissioned systems. Instead, we think wise enterprises will (at least in the near and medium term) adopt systems that:

• Are specifically scoped to interact with a vetted set of dependencies on a per-task basis – this helps ensure that unapproved decision making inputs are fully avoided, since the system has no way to access them. That in turn reduces the risk of prompt injection or other AI attacks while also reducing the potential for unexpected or untested sources of decision bias.

• Escalate versus YOLO when the conditions for a task fall sufficiently outside of tested boundaries – this helps in many of the same ways as the previous bullet. If you’ve only tested your digital car salesmen in conversations about consumer vehicles, do you really want it to be allowed to discuss a quote for a backhoe? Humans are less consistent with process execution than machines, but they’re often better at rapidly adapting to unexpected unknowns, even versus modern AI.

• Undergo versioned releases, in which major behavioral changes are tested under strict monitoring and/or with historical data (while noting that testing against historical data can be hard for multi-step processes, because you don’t necessarily know what would have happened if your action on an early step changed)

• Are designed from the ground up to support detailed logging, rich debugging, and emergency overrides in the case of bad behavior.

To be clear, these are very specific tradeoffs. They imply that enterprises will pay the cost of mining human resources to both oversee agentic processes (what good is logging if you never check it?) and also to provide judgement and fallback support as a final escalation point. They also imply that processes will be somewhat less adaptive – since they’d go through strict versioning – in exchange for being more clearly defined and tested.

But we’d argue there’s precedent for these preferences. If you look at traditional, human escalated processes, they often start off poorly defined and with a lot of latitude for individual humans to use their judgement. But as processes and companies mature, we typically observe an increase in process structure, planning and review and a corresponding decrease in real time adaptability. While this can be partly driven by cost – it’s cheaper to staff human employees if you don’t need them to do anything fancier than following a deeply proscriptive flow chart – we think it’s also driven by a desire for predictability and control. Such desires would apply to AI automated processes as well, and with similar results.

That said, we expect that we will see broadly permissioned, low guardrails agents in one key space – consumer products, and potentially user desktops (Claude desktop, agentic browsers, and voice assistants are nascent examples). These sorts of use cases move away from focusing on any one process and instead attempt to assist with most things a user could want. That’s distinctly at odds with the sort of repetitive consistency needs we see in process automation. For these use cases, we expect to see significantly more autonomy and, in exchange, more risk (this is, in our opinion, a large part of why Gartner has taken such a strong stance against agentic browsers – it is exceptionally difficult to secure them from being influenced by unwanted inputs, and it is equally difficult to derisk their broad and autonomous surface area).

Per-User Agents

Another suggestion that’s come up is the idea of per-user agents for things like customer support or engineering velocity. For this one, we’d like to see greater clarity on what a per-anything agent is. Does it mean that an automated system is producing distinct pieces of software which, given the same inputs [customer context and circumstance, technical project, etc.] would produce different outputs? Or does it mean that a single set of logic, which may vary its behavior based on things like customer context, is being individually applied to all customers?

For a practical example, imagine a chess engine with an “aggression” slider. Different users may have radically different experiences based on where they put the slider (or based on how the engine responds to their playing), but under the hood it’s still the same piece of software. If two otherwise identical users – some aggression preference, same moves – used the engine, they’d see markedly similar behavior. On the other hand, you can also imagine two completely different engines, perhaps both with an aggression slide. In that case, a set of users might get a completely different experience even for the same aggression levels because under the hood, it’s completely different systems which should be expected to act in completely different ways.

We think the extent to which behavior is per-user can (and should) be viewed on a spectrum, and would encourage specificity about where on this spectrum use cases fall. In our eyes, the spectrum looks something like:

• Fully bespoke, per-user agents (and codebase), aka two different chess engines: In this world, two otherwise identical users end up with fully distinct software and different experiences. As far as we know, there’s minimal precedent for this.

• Singular codebase (and agent), but an extreme degree of per-user customization, per-user execution, aka a single chess engine with a lot of sliders: This is again where many desktop agents fall. Different users can engage with Claude code via radically different patterns and receiving significantly varying experiences as a result, and their various interactions are largely isolated from each other. But at the end of the day, an update to the agent will affect all users.

• Singular agent, moderate to low per-user customization, multi-tenant execution, aka a single engine: This is where many enterprise process automation agents will fall. While you want some degree of user customization – for example, letting users choose how support agents engage with them (contact medium, speech versus text, language, etc) makes sense, but other things (account history, policy) need to be treated in a consistent manner across all users.

We expect the first case – two distinct engines, or two fully distinct agents – to be rare for any single use case. It’s a lot easier to build and maintain one good engine versus two, and the same applies to agents! As a result, we expect use cases to cluster on the second and third option, with degree of customization (aka the number of sliders) being driven by the desired degree of user flexibility. We’d also note that it doesn’t have to be a slider – for example, every time someone defines a claude.md file describing how they want Claude code to work for them, they’re customizing the highly configurable but also singular Claude code agent.

This has implications for agentic identity – we shouldn’t be thinking about overall agents having singular identities, but rather about how a process can be set up to step into a set of permissions appropriate for working with any given customer. This ties back in to the level of autonomy. Such an agent probably won’t be able to come up with a radically new policy for addressing long tail customer complaints – for example, it couldn’t deal with a lost, hard-to-replace delivery by going to Amazon and ordering the customer a replacement sourced from a competitor. But is that a bad thing? As above, our suspicion is that enterprises would prefer to formally approve such behaviors before seeing them take place in the wild (or production).

Conclusion

As agentic automation matures, the question is less whether agents can act autonomously, and more whether enterprises are willing to let them. In practice, scaling agentic process automation while satisfying enterprise desires for control and predictability will require embracing constraint over freedom: scoped permissions over broad autonomy, escalation to human operators over improvisation, and shared, versioned logic over fully bespoke behavior. This doesn’t limit the potential of agents but makes them potential usable in real-world environments where predictability, auditability, and trust matter more than speed, creativity, and cleverness.