Most AI problems are really human problems

The cause sits upstream: unread intent, missing oversight, thin context, loose language, unset expectations, no evals, fuzzy outcomes, and the wrong tool for the job. Here’s the AI checklist.

Anthropic recently told its growth team to hire more product managers, not fewer. You’ll be surprised at the reason why.

Claude Code had turned its engineers into a team shipping at triple its headcount, and the bottleneck moved from writing code to deciding what to build, which is why everything is a research preview—so they can get user feedback.

A Jevons Paradox moment to say the least.

The technology no longer the constraint; it now takes more time to plan and get to a shared understanding, according to Microsoft’s Travis Lowdermilk, than to build. We’re moving so fast, misalignment is the outcome, which could lead to project failure.

What’s new is the acceleration of code generation. AI doesn’t create the need for shared understanding. It puts even more pressure on it.
 — Travis Lowdermilk

Because of this they want the humans to be aligned and in control, which is a perfect time for user experience: put in enough thought to do it right.

Right now when a project stalls, the instinct is to blame the model and wait for the next one. It rarely helps; a 2025 MIT study found that roughly 95 percent of generative AI pilots deliver no measurable impact.

This is not new, just has a different target since it didn’t take AI to have failed software projects that cost a lot of money—it’s always fun to link to CIO for that—and in times of innovation this is to be expected because we underestimate the ambiguity we because we don’t have a shared perspective. All white space that happens in meetings and Slack channels, and not in code.

We’re still in early times so it’s also a learning and expectations gap: a capable tool that has known constraints inside an organization that hasn’t changed, AI or not.

Most AI failures are organizational, not technical—Ironically a failure of human in the loop making unrealistic decisions.

I have spent 11 years around AI, the last three and a half in generative AI. We built these systems and I have found that the bottleneck sits upstream, in the scaffolding around it: the user’s intent, who is accountable, what the model can see, how the work is specified, what the system promises, whether you can measure it, the outcome you are chasing, and the tool you picked. Not a single line of code in there as the reason.

And frankly, a lot of the above are just solid software development practices that are even more important in the era of AI.

Eight human problems, each wearing a technical hat. None waits on a better model or line of code; they wait for better human decisions.

Intent: Start From What People Are Trying to Do

Most AI efforts start from the capability and work backward to a use. The model can summarize, so we add a summarize button. It can chat, so we bolt on a chatbot. Nobody asked what the person was trying to accomplish; the technology proposed itself, and the team said yes.

That is how you ship a feature no one opens, and it’s the most basic of user experience first principles i.e. what are you solving for.

The discipline is to start from intent: the job the person is trying to do, in their words, before any mention of AI. I have written about what Will Manidis calls a tool-shaped object, something that produces the feeling of work without the work.

Capability-first AI is a factory for them: it feels productive and ships nothing anyone needed, but it looked great in a demo.

Nobody asked what the person was trying to accomplish; the technology proposed itself, and the team said yes.

Intent is what every later decision hangs on. You cannot match the tool to the job, set honest expectations, or define an outcome until you can state, plainly, what the person is here to do.

Get the intent wrong and the most capable model in the world will execute the wrong task perfectly. Start there, in plain language, before anything else.

Action items

• Write the intent as the user’s sentence. State what the person is trying to accomplish in their words, with no mention of AI, before you design anything. How might we’s are a great way to start so there’s alignment.
• Kill capability-first ideas. If the pitch starts with what the model can do rather than what someone needs, send it back until it names the job. That’s technology looking for a problem.

Fit: Use an Agent Only When the Work Needs One

The fastest way to fail with AI is to reach for the most powerful version of it. I call it flying down with a 747 to the corner store to grab a gallon of milk.

An autonomous agent that plans, calls tools, and acts on its own is impressive and expensive and hard to oversee. Most of the work people point it at is retrieval, summarization, or a fixed sequence of steps, jobs a simpler tool does more reliably in a deterministic approach and at a fraction of the cost and risk.

Gartner’s analysts have been direct: many of the use cases being sold as agentic today do not need an agent at all, and the firm estimates only about 130 of the thousands of vendors claiming agentic capability are the real thing.

The industry has a name for the rest: Agent washing.

Buying it means paying agent prices for assistant work. Slower, less accurate and more expensive than before.

The fastest way to fail with AI is to reach for the most powerful version of it without understanding the need.

The fix starts with a clear-eyed look at the task. Fit the tool to the job: use an agent when a decision needs to be made, automation when the steps are fixed, and an assistant when someone just needs an answer.

And when the work really is agentic, the next question is where it should live.

• Chat fits exploration, one-off questions, and tasks where the person wants to steer turn by turn.
• Ambient fits the rest: work that runs in the background, triggered by an event, surfacing only when it needs a decision or hits a limit you set.

Forcing everything through a chat box is the common error, so an agent that should run quietly ends up waiting for someone to open a window and type. Match the modality to the moment, and use each where it fits.

Reach for a background agent only when the task clears this bar. The first four decide whether the work is agentic at all; the last two decide whether it is safe and worth it.

• A real decision under uncertainty. The right next step is not known in advance. If you can draw the flowchart, it is automation, not an agent.
• Steps that depend on each other. The work runs several steps where each shapes the next, so the plan has to adapt as it goes rather than follow a fixed script.
• It has to act, not just answer. The task calls tools or takes actions in a system, not only returns text.
• Inputs too varied to hardcode. The situations differ enough that a fixed workflow would break or need constant patching.
• A bounded, overseen blast radius. Consequential actions sit behind permissions and a human checkpoint, so a wrong move is recoverable.
• Autonomy that earns its cost. The time or scale the agent saves outweighs the expense, latency, and risk of letting it run on its own.

Action items

• Match the tool to the task. Start with the MVP. Decision needed, agent; fixed steps, automation; retrieval, assistant. Most work is the last two.
• Match the mode to the moment. Use chat for exploration and turn-by-turn steering; run ongoing or background work ambient, on triggers, instead of behind a chat box the user has to remember to open.
• Demand proof of autonomy. If a vendor sells an agent, ask what it decides without a human; if the answer is nothing, it is not one.

Expectations: Show Users What to Give It and How Far to Trust It

A new hire gets an onboarding: here is what you will handle, here is what to bring to it, here is how sure you should be before you act. Most AI systems give the user none of that, on either side. They rarely say what to feed them to get a good result, and they return a confident paragraph whether the answer is solid or a guess.

The user is left to guess at the input and calibrate trust in the output with no signal for either.

Expectations run both ways. On the way in, a good system shows what a strong request looks like, what context to include, and what it will quietly ignore. Most do not, so a user types a vague line, gets a vague answer, and concludes the model is weak when the real gap was the prompt. Telling people what to give you is the cheapest quality lever there is.

Telling people what to give you is the cheapest quality lever there is.

On the way out, the same honesty applies.

The team behind Microsoft’s Guidelines for Human-AI Interaction, Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Eric Horvitz, and colleagues, put two rules first:

• Make clear what the system can do
• Make clear how well it does it

Most products ship the capability and hide the limits, then act surprised when users over-trust a wrong answer or abandon a useful one.

The over-trust is well documented: in one controlled study, trained pathology experts overturned their own correct judgment to follow the AI’s wrong call in about 7 percent of decisions, a pattern researchers call automation bias, and a system that never signals its own uncertainty makes it worse.

Setting expectations is mundane work on both ends, which is why it gets cut. Tell people what to bring. Show explainability. Name what the system cannot do before they find out the hard way.

None of it is glamorous, and all of it is the difference between a tool people trust and one they quietly stop using, mainly because they don’t understand the constraints.

Action items

• Tell users what to give it. Show what a good request looks like, what context to include, and what the system will ignore, so a vague input does not become a vague output you both blame on the model.
• Tell users what to trust. State what the system does well, does poorly, and how often it is wrong, and make an uncertain answer look different from a sure one.

Oversight: Decide What the AI Is Allowed to Touch

Most AI deployments have no owner. There is a sponsor who approved the budget and a vendor who sold the tool, but no single person accountable for what the system decides and what happens when it decides wrong.

That vacuum is fine in a demo; it is not fine when the agent has write access to customer records.

Oversight is not paperwork, it is the harness: the set of permissions that decides what the AI can and cannot do, so that doing the wrong thing is never available to it.

The most reliable way to stop an agent from taking a harmful action is to make sure it never had permission to take it. Gartner expects more than 40 percent of agentic AI projects to be canceled by the end of 2027, and one of the three named causes is inadequate risk controls.

A risk control is an oversight decision before it is a technical one. What is the agent allowed to touch? Who reviews its actions? Who can shut it off?

The most reliable way to stop an agent from taking a harmful action is to make sure it never had permission to take it.

This is not hypothetical. In 2025, Replit’s AI coding agent deleted a live production database during a code freeze, wiping records for more than 1,200 executives and 1,190 companies, despite repeated instructions to change nothing. I had a conversation with an UX expert this, and we aligned that it should have never happened because you have to put the same guardrails in for AI that you have to for people.

The model was not the failure.

The failure was that an agent in development could reach production and run a destructive command at all, which is a security problem. Replit’s own fix was to enforce separation between development and production like any normal system, an oversight decision made one disaster too late.

When those answers are missing, the project does not fail loudly. It stalls in a holding pattern, never trusted enough to leave the pilot. Security teams have known the fix for decdes under a plain name: least privilege.

Give the system the narrowest access that still lets it do its job, and the exposure of any mistake shrinks to something you can live with. Before you tune a prompt, decide what the AI is permitted to do just like you would do for a human, and name someone who owns that decision.

Action items

• Scope permissions to least privilege. Grant the narrowest access the task requires; everything the agent cannot reach is a harm it cannot cause.
• Name an accountable owner and a stop condition. One person answerable for the system in production, and a documented way to halt it, settled before the first deployment rather than after the first incident.

Context: Give the Model What It Cannot See

A model knows what is in the prompt and what is in its training. It does not know your pricing exceptions, your renewal calendar, the customer who threatened to leave last week, or the unwritten rule that finance signs off on anything over ten thousand dollars.

The MIT researchers found that pilots stall on brittle workflows and a lack of contextual learning, where the tools never absorb how the work is done because they don’t have context of the white space.

That is not a model defect. It is a context-supply problem, and the supply chain runs through your documentation, your data, and your willingness to expose both. It fails, because the machine gave answers no one would trust — it was playing blackjack when it couldn’t see either card the dealer was holding.

When the output is generic, the model is not failing. It is answering the only question it could see. This is the part most teams skip because it is unglamorous.

This is where I see most projects failing most — not enough context for the AI to hang its hat upon. I’m not alone in saying this.

Andrew Ng has spent years arguing that the fastest gains in applied AI come from improving the data, not the model. The same logic holds for context. A mediocre model with rich, current context will outperform a frontier model working blind, every day of the week.

Knowing full well every agent is a context experiment for what it is good enough, this is going to take time to learn. Until then, feed it the organization as much as possible, or accept that it is guessing.

Action items

• Inventory what the model can see. Write down the context it has and the context it needs; the gap is your roadmap.
• Put context what the model can’t see. A document no system can retrieve is context the model does not have. Then you know where the risk is.

Language: Write Instructions Tight Enough to Execute

Ask three people to summarize the customer feedback and you will get three different documents. Ask a model the same thing and you get a fourth. The problem is not comprehension. The problem is that the instruction never specified length, audience, format, what to include, or what to leave out. Ambiguous in, ambiguous out.

We spent decades teaching teams to write clear requirements for software that executes literally. AI is different in kind.

It is probabilistic, not deterministic: it predicts the most likely next words rather than running your instructions to the letter, and sometimes it will not get it right, literally by design. The model fills any gap you leave with a plausible guess, which is whywhy controlled language and controlled vocabularies, precise and structured and unambiguous, is now a core skill rather than a nicety. Most people don’t have that skill, acting like Michael Stipe at an R.E.M. concert.

The prompt is a specification, and most prompts are underspecified.

This is where a consistent structure across most of your documents earns its keep: A request with named fields, explicit constraints, and worked examples outperforms a paragraph of hopeful prose every time.

The discipline is old, and it is why you keep UX professionals around.

Technical writers, information architects, and content designers have spent years turning fuzzy intent into precise, structured language, which is exactly what a good prompt demands. Yuval Keshtcher and the content design field treat this as craft: designing the rules that govern language, not writing one string at a time.

AI just made that craft foundational for everyone else.

A controlled prompt is built, not typed. I use a format I have written about, CARE: Context, Ask, Rules, Examples. Context is the background the model needs. Ask is the request in one clear sentence.

Rules are the constraints it must respect, including format and tone. Examples show what good looks like. Leave any of the four blank and the model fills it with a guess. Name all four and the output stops drifting, because there is nothing left to infer.

Action items

• Write the prompt in CARE format. Context, Ask, Rules, Examples; a blank in any one is where the output starts to drift.
• Specify before you prompt. State the audience, length, and constraints up front; do not make the model infer them.

Evals: Measure Whether It Works Before You Ship It

Most teams cannot answer a simple question about their AI feature: is it better or worse than last week?

They shipped on a handful of prompts that looked good in a demo, and they have measured nothing since. So when they change the model, tweak the prompt, or swap a vendor, they are guessing.

The system might have improved.

It might have quietly broken.

Nobody can say.

Evals are the fix: a fixed set of real inputs with known-good outputs, scored the same way every time, so a change produces a number instead of a feeling. State what success is before you start.

Conventional software has had this for decades; we call it a test suite. I know, because I was leading test driven development as a Product Manager in 2002, and it accelerated development 50 percent because the developers knew exactly what success was.

AI needs it more, not less, because the output is probabilistic and a change that helps one case can break another you never thought to check. McKinsey found that a defined process for when model outputs need validation is one of the practices that most separates the rare high performers from everyone else.

Most AI teams have no such process.

This is a discipline problem, not a tooling one. The eval set is built from the messy real cases your users send, the edge cases that embarrassed you, the categories that matter to the business.

It is unglamorous to assemble and it is the only thing that turns “the new version feels better” into something you can defend. Without it, every release is a vibe and every regression is a surprise.

Action items

• Build an eval set from real cases. Collect actual user inputs, especially the failures and edge cases, with the output you would consider correct, and score every change against them.
• Gate releases on the score. Do not ship a model, prompt, or vendor change that drops the eval number, however good the demo looked.

Outcomes: Name the Number You Can Defend

Walk into most AI initiatives and ask what success looks like. You will hear improve productivity, or reduce costs, or, more candidly, leadership wants AI. None of those is an outcome. They are wishes with a budget attached, and they cannot be measured, which means they cannot be defended when the spending review arrives.

Gartner names unclear business value as a leading reason agentic projects get canceled, the same forecast that put the cancellation rate above 40 percent. The tools that lasted in every prior wave, the spreadsheet, the search box, the database, won because they made a specific job faster, not because they were new. AI is not exempt from that test.

They are wishes with a budget attached, and they cannot be measured, which means they cannot be defended when the spending review arrives.

A model that has no defined job to do well is a model nobody can tell is working. This is old product discipline. Marty Cagan has argued for years that teams should commit to outcomes, not output, the result you want rather than the feature you ship.

AI makes the discipline urgent: decide what changes in the business if this works, name the number, and you will know whether to scale it or kill it.

This is why designers and researchers still earn their place as products get built on top of AI. Someone has to decide which problem is worth solving, and the model will not do that for you other than predicting a guess with no judgement or taste.

That judgment is the work.

It also strains the way teams ship. Most development processes were built for predictable software on a schedule, and they do not fit the AI innovation cycle, which moves faster, loops more, and rewards teams that can reframe the problem mid-build instead of freezing it in a specification.

In an innovation fueled, probabilistic world, you’re going to encounter some bumps; plan accordingly.

Action items

• Write the outcome as a number. If you cannot state the metric that will move, you are not ready to build.
• Pick a job, not a category. Cut first-response time in support beats add AI to support every time.

Conclusion

None of this is an argument against AI. It is an argument about where the work is. When factories wired up electric motors, the gains arrived only after managers like Henry Ford redesigned the floor.

The motor was never the bottleneck, The layout was. AI is at the same moment.

The models are ready; the organization around it usually is not.

The eight failures are not technical, and that is the good news, because human problems are the ones you can fix without waiting for a vendor. Start from the user’s intent. Name the person accountable for the decision. Give the model the context it cannot infer. Specify the work in language tight enough to execute. Tell the user what to expect. Measure whether it works. Define the outcome as a number you can defend. Match the tool to the job instead of buying the most powerful one in the room.

Do that, and the model you have performs like the one you waited for.

Skip it, and no model progress will save the project, because what broke was never the model, it was human expectations of amazing technology that is still in early stages of maturity which we forget.

The hard part of AI was always going to be us. That is also the one part we get to control.

An AI Checklist

Run an AI effort through this before it ships; every item is a human decision, not a model setting, and each lists the best-practice documents that set the foundation, the ones you write once and reuse. Clear them and the AI problem never gets the chance to become the human one that sinks the project.

Intent

Can you write the user’s goal in one sentence, in their words, with no mention of AI or the model?

• A clear problem statement: the goal in one line, in the user’s words, with the non-goals named.
• Job story: the triggering situation, what the user wants to do, and the outcome they expect in a world where states are more important than rules.

Fit

Can you name the tool you chose, the simpler option you ruled out, and which of the four agent triggers the work meets: a real decision under uncertainty, interdependent steps, action rather than answers, and varied inputs?

• Tool decision record: the chosen tool, the simpler option you rejected, and why. Think Occam’s Razor, without Matthew McConaughey in Contact.
• Agent-fit assessment: which of the four triggers the work meets, and the bounded exposure that earns the autonomy.

Expectations

Can a first-time user point to where the interface states what the system can do, how often it is wrong as a generalized number, and how sure it is about the answer in front of them because there’s explainability?

• Capability-and-limits statement: what the system does well, does poorly, and refuses to do.
• Confidence-display specification: how certainty and the error rate surface in the interface.

Oversight

Can you list every system the AI is allowed to change, name the one person accountable, and show the documented way to stop it and who can trigger it?

• Least-privilege access matrix: every system the AI can read or change, and nothing beyond it.
• Accountability record: the one named owner, mapped to each action the AI can take.
• Stop-and-rollback run book: how to halt the system, who can trigger it, and how to undo.

Context

Have you listed the context the model needs, confirmed each item sits in a store it can retrieve at run time, and closed the gaps?

• Context inventory: each item the model needs, where it lives, and whether it is retrievable at run time.
• Gap list: the missing context, with an owner assigned to each item.

Language

Does the prompt carry all four CARE parts, context, ask, rules, and examples, with at least one worked example?

• CARE prompt template: context, ask, rules, and examples, versioned and reused.
• Worked-example set: representative inputs paired with the output you would accept.

Evals

Do you have a scored eval set of real cases with known-good answers, a numeric pass bar, and a standing rule that any change below the bar does not ship?

• Held-out eval set: real cases, including failures and edge cases, with known-good answers.
• Scoring rubric and pass bar: how each case is scored and the number a release must clear.
• Release-gate policy: the standing rule that anything below the bar does not ship.

Outcome

Have you named the single metric, its current baseline, the target value, and the date you will check it?

• Outcome statement: the single metric, its baseline, the target, and the date you will check it.
• Measurement plan: how the metric is captured and who reviews it.

If any box is unchecked, the gap is human, and so is the fix. That is the good news.

Resources

• The GenAI Divide: State of AI in Business 2025 — the primary MIT NANDA report: about 95 percent of enterprise generative AI pilots deliver no measurable profit-and-loss impact, and the barrier is a learning gap, not model quality.
• Claude Code turned every engineer into three. Now companies need more product thinkers — VentureBeat on the bottleneck shifting from writing code to deciding what to build as coding agents raise engineering throughput.
• Gartner: over 40 percent of agentic AI projects will be canceled by 2027 — escalating costs, unclear business value, inadequate risk controls, and the “agent washing” estimate.
• An AI coding tool wiped a company’s production database — the 2025 Replit incident: an agent deleted live data during a code freeze; the fix was enforced separation between development and production.
• Guidelines for Human-AI Interaction — Saleema Amershi, Dan Weld, Mihaela Vorvoreanu, Eric Horvitz, and colleagues at Microsoft Research; the first two guidelines are about setting expectations.
• Automation Bias in AI-Assisted Medical Decision-Making — a controlled study in which trained experts overturned their own correct calls to follow erroneous AI in about 7 percent of cases.
• The State of AI in 2025 — McKinsey’s global survey; a defined process for validating AI output is one of the practices that most distinguishes high performers.
• Outcomes Are Hard — Marty Cagan (SVPG) on moving from output to outcomes, framed around the problem to solve.
• A Chat with Andrew on MLOps: From Model-centric to Data-centric AI — Andrew Ng’s own talk arguing the fastest gains in applied AI come from improving the data, not the model.
• The key to great AI-generated content: well-structured, consistent language — the CARE prompt framework.
• Agentic Content Design: How to Build Your First Guidelines — Yuval Keshtcher (@yuvalkeshtcher) on content design as the craft of structured, rule-governed language, now applied to AI.
• What UX competencies in a Generative AI world could look like — why problem definition keeps designers and researchers central.
• Tool-Shaped Objects in UX, Part 1 — building from intent rather than capability, applying Will Manidis’s “tool-shaped object.”
• Tool-Shaped Objects in UX, Part 2 — Patrick Neeman’s field guide to what survives the shift to AI, including why conventional delivery processes strain against the innovation cycle.

Most AI problems are really human problems was originally published in UX Collective on Medium, where people are continuing the conversation by highlighting and responding to this story.