AI Agents Fail at the Boring Work. What AI Agents for Small Business Can Really Do
Why the tasks you most want to hand off are the ones agents fail most, and how to pick the ones they can handle.
Researchers at Carnegie Mellon built a fake small software company and gave AI agents real office jobs inside it. The best agent in their paper finished 30.3 percent of 175 tasks.
The surprise is where it failed. On software engineering tasks, it succeeded 37.68 percent of the time. On administrative tasks, 13.33 percent. On finance tasks, 8.33 percent.
Most people assume agents will take the boring admin first and the hard expert work later. The benchmark points the other way.
The promise behind AI agents for small business is not stupid. Agents can already do real work. They just break on the dull parts: vague instructions, missing context, and clicking through software built for humans.
I want to show you what the benchmarks actually measured. I want to show you the three mistakes small businesses make when they buy or build an agent. And I want to give you a 5-step readiness check you can run on one task this week.
Key takeaways
- In TheAgentCompany benchmark, the best agent in the paper completed 30.3 percent of office tasks. Admin and finance tasks scored lowest.
- Agents failed on things like a closable popup and not following up with a colleague, not on hard reasoning.
- Narrow, written, checked workflows work far better. Start with one boring task, a written process and a test set of real cases.
What the research on AI agents for business actually shows
The common belief: agents are smart enough now, so give them a job and they will do it. Intelligence is the bottleneck, and it is solved.
The evidence says intelligence is rarely the bottleneck. The job description is.
The fake company with real tasks
TheAgentCompany was built by a team led by Frank F. Xu and Graham Neubig at Carnegie Mellon. It simulates a small software company with internal tools: a code host, a file-sharing site, a project tracker and a team chat. Simulated colleagues answer questions in the chat.
The 175 tasks came from the kind of work people in such a company do. Write code, update a project board, fill in a spreadsheet, collect information from colleagues. Building them took 20 people more than two months and about 3,000 person-hours.
In the paper, the strongest setup was the OpenHands agent with Gemini 2.5 Pro. It completed 30.3 percent of tasks fully and scored 39.3 percent when partial progress counted. Each task took about 27 steps and more than 4 dollars in model costs.
What is an AI agent? An AI agent is a system where a language model decides its own next steps and uses tools, such as a browser, files or email, until a task is done. Anthropic draws a useful line: a workflow follows a predefined path written in code, while an agent directs its own process. Most things sold as agents to small businesses are workflows, and that is often a good thing.
Where the agents broke
The breakdown by department is the part almost nobody quotes. It sits in an appendix table.
The authors call this a gap between perceived difficulty for humans and for agents. Admin and finance tasks involved spreadsheets, collecting details from several people, and reading scanned images. Most humans would find that easier than writing code. The agents did not.
The authors name the reasons: trouble understanding documents, communicating with people, navigating complex software and handling tedious processes. Coding has huge amounts of public training data. Your invoicing process does not.
The platform numbers tell the same story. On tasks that required the file-sharing tool ownCloud, the best agent succeeded 12.86 percent of the time. A closable welcome popup alone kept stopping the text-based agent, which failed to click the x.
The failures were boring, not deep
The paper lists failures that a new hire would rarely make. In one task, an agent asked a colleague who to introduce itself to next. The colleague named a person. The agent never contacted her and marked the task as done.
In another, the agent could not find the right person in the team chat. So it renamed a different user to the name it was looking for. The authors call this deceiving oneself: faking a shortcut past the hard part.
Is this already outdated? Partly. The public leaderboard has moved since the paper. The top entry, from November 2025, is 42.86 percent. That is real progress, and it is still a minority of tasks. As of this writing, the leaderboard lists no newer entry.
Gartner sees the same pattern from the buyer side. In June 2025, it predicted that over 40 percent of agentic AI projects will be canceled by the end of 2027, citing costs, unclear business value and weak risk controls. Its 2026 Hype Cycle for Agentic AI cites a Gartner CIO survey: only 17 percent of organizations have deployed AI agents. More than 60 percent expect to within two years. And most deployments, Gartner notes, remain narrowly scoped.
In short: agents do not fail because they are not clever enough. They fail because the work around the task is unclear, undocumented or hidden behind an interface.
The three mistakes small businesses make with AI agents
Once you see where agents break, the common mistakes are easy to spot.
Mistake 1: Buying the label instead of the capability
In the same 2025 release, Gartner described "agent washing": rebranding chatbots, assistants and older automation tools as agents. It estimated that only about 130 of the thousands of agentic AI vendors are real.
That does not make every relabeled tool useless. A chatbot that answers opening hours is fine. It is just not an agent, and it should not be priced or trusted like one. Ask any vendor which decisions the system makes on its own, and what it does when it gets stuck.
Mistake 2: Handing over a job instead of a task
Here is an example. A solo founder tells a new agent: "Handle my booking requests." Some clients email, some message, some ask to move an existing slot. Prices depend on details only the founder knows.
That is not one task. It is a job made of ten tasks, half of them unwritten. And it involves back-and-forth with people, which is exactly where agents drop.
Salesforce researchers measured this in CRMArena-Pro, a benchmark of 19 business tasks in sales, service and quoting. Leading agents reached around 58 percent success when the request came in one message. With multi-turn conversations, success fell to about 35 percent. On workflow execution, a clearly defined process, top agents got above 83 percent in single-turn settings.
The clearer the process, the better the agent. The more it has to clarify with people, the worse.
Mistake 3: No human check where money or customers are involved
Anthropic and Andon Labs let Claude run a small office shop for about a month in 2025. The agent, called Claudius, ordered stock, set prices and talked to customers over Slack.
It found niche suppliers quickly. It also sold items below cost, was talked into many discount codes, gave some items away, and for a time told customers to pay into a Venmo account it had invented. Anthropic concluded it would not hire Claudius.
In phase two, with newer models, the shop did better. Among the most impactful changes was forcing Claudius to follow procedures, such as checking prices and delivery times before quoting them. Anthropic summed it up as rediscovering that bureaucracy matters. The agent also never had a payment interface, so a human signed off on purchases.
How to use AI agents: the Agent Readiness Check
None of this means small businesses should wait. It means the first agent should be small, written down and checked. Here is a 5-step check to run on one task before you automate it.
Step 1: Narrow it to one repeating task. Pick something that happens at least weekly, has one clear input and one clear output, and is cheap to get wrong. "Turn each new web inquiry into a CRM entry and a draft reply" passes. "Run my sales" does not.
Step 2: Write the process like you would for a new hire. List the steps, where each piece of information lives, what done looks like, and what to do when something is missing. If a colleague is part of the task, name who to ask and what to do with the answer. This document is the agent instruction. If you cannot write it, the agent cannot follow it.
Step 3: Build a test set from 20 real past cases. Take 20 examples where you know the right result. Run the agent on all of them and count how many it gets fully right. Decide the pass mark before you look, for example 18 of 20. This is what benchmarks do, at the size of your business. The AI productivity study article shows why your gut feeling is a poor substitute.
Step 4: Remove the interface obstacles. Give the agent direct access through an integration or an export, not a screen it has to click through. Make sure it can see the facts it needs, such as cost prices and stock levels. Popups and logins are where agents stall.
Step 5: Put a human check on money, customers and anything irreversible. Drafts go out only after approval. Discounts, refunds and payments need a yes from a person. Read a sample of the log every week, and look for tasks marked done that were not.
One more filter before you start: every agent should serve a goal you can name. If you cannot say which quarterly target a task moves, automate something else. Linking daily work to bigger goals is the idea behind Gaapp.io, a goal tracking app that shows what is due today and which bigger goal it serves. Disclosure: I am building it. A paper notebook works too. The same logic runs through the article on how to achieve goals: a plan with a when and a where beats a wish.
Ask yourself this week:
- Which task do I repeat every week that I could explain in one page?
- Do I have 20 past examples with a known right answer to test an agent on?
- Where would an agent mistake cost me money or a customer, and who checks that point?
Conclusion: the agent is only as good as the job description
Agents rarely fail at thinking. They fail at the parts nobody wrote down.
The research does not say AI agents are useless for small businesses. It says they are uneven. They handle clear, well-documented workflows well. They struggle with vague jobs, conversations, internal admin and clunky software.
That is good news for small teams. You do not need a bigger model. You need one narrow task, a written process, a test set and a human at the money. That costs an afternoon, not a budget.
What you learned today
- In TheAgentCompany benchmark, the best agent in the paper completed 30.3 percent of office tasks, with 13.33 percent on admin and 8.33 percent on finance.
- Agents fail on boring things: unclear instructions, multi-turn conversations, missing context and popups. In CRMArena-Pro, success fell from about 58 to about 35 percent with back-and-forth.
- Start with one narrow task, a written process, a test on 20 real cases and a human check on money and customers.
FAQ
What are AI agents for small business? AI agents for small business are AI systems that take a goal, plan steps and use tools such as email, a CRM or files. They complete work with limited supervision. Today, the useful ones handle narrow, repeating tasks with a written process, such as sorting inquiries or preparing drafts. A person still approves anything involving money or customers.
How do you use AI agents in a small business? Pick one repeating task with a clear input and output. Write the process as you would for a new hire. Test the agent on 20 past cases with known answers and set a pass mark first. Connect it directly to your tools, and keep a human approval step for payments, discounts and customer messages.
What are some examples of AI agents? Common examples include agents that turn web inquiries into CRM entries and draft replies, agents that reconcile invoices against orders, research agents that collect supplier prices, and coding agents that fix software issues. In benchmarks, agents did best on clearly defined workflows and software tasks, and worst on admin and finance work with many people involved.
What is agentic AI? Agentic AI describes AI systems that act toward a goal instead of only answering a prompt. They decide next steps, call tools and check results. In 2025, Gartner estimated that only about 130 of the thousands of agentic AI vendors are real. It called the relabeling of chatbots and older automation "agent washing."
Are AI agents worth it for small businesses? They can be, for the right task. Gartner predicts over 40 percent of agentic AI projects will be canceled by end of 2027, mostly due to costs, unclear value and weak risk controls. Small businesses avoid that by starting narrow, testing on real cases and measuring time saved before expanding.
About the author
Tristan Eilenburg. One person. AI systems. Global reach. I write about building systems that work, with evidence instead of hype.
- X: @MrB2Beast
- LinkedIn: Tristan Eilenburg
- Instagram: @mrb2beast
- Building Gaapp.io (Instagram: @gaapp.io)
I write one of these every week. If this one taught you something, the newsletter is where the next one lands first.
Sources
- Xu, F. F., Song, Y., Li, B., Tang, Y., Jain, K., Bao, M., Wang, Z. Z., Zhou, X., Guo, Z., Cao, M., Yang, M., Lu, H. Y., Martin, A., Su, Z., Maben, L., Mehta, R., Chi, W., Jang, L., Xie, Y., Zhou, S., and Neubig, G. (2025). TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks. arXiv:2412.14161 (v3, September 10, 2025). Presented at NeurIPS 2025. https://arxiv.org/abs/2412.14161
- TheAgentCompany. Leaderboard (accessed October 2026). https://the-agent-company.com/leaderboard and submissions: https://github.com/TheAgentCompany/experiments
- Gartner (2025, June 25). Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027. Press release. https://www.gartner.com/en/newsroom/press-releases/2025-06-25-gartner-predicts-over-40-percent-of-agentic-ai-projects-will-be-canceled-by-end-of-2027
- Kandaswamy, R. (2026, April 15). What the 2026 Hype Cycle for Agentic AI Reveals. Gartner. https://www.gartner.com/en/articles/hype-cycle-for-agentic-ai
- Huang, K.-H., Prabhakar, A., Thorat, O., Agarwal, D., Choubey, P. K., Mao, Y., Savarese, S., Xiong, C., and Wu, C.-S. (2025). CRMArena-Pro: Holistic Assessment of LLM Agents Across Diverse Business Scenarios and Interactions. arXiv:2505.18878. https://arxiv.org/abs/2505.18878
- Anthropic (2025, June 27). Project Vend: Can Claude run a small shop? (And why does that matter?). https://www.anthropic.com/research/project-vend-1
- Anthropic (2025, December 18). Project Vend: Phase two. https://www.anthropic.com/research/project-vend-2
- Erik S. and Zhang, B. (2024, December 19). Building effective agents. Anthropic Engineering. https://www.anthropic.com/engineering/building-effective-agents
