Shield Technology Partners has released new AI benchmarks showing that even leading AI models struggle to consistently resolve complex IT support tickets, raising questions about how MSPs should balance automation with technician oversight.
The evaluation tested leading AI models against 52 challenging support tickets that Shield’s AI Engineer had previously handled incorrectly. GPT-6 Astra with extra-high reasoning achieved the highest pass rate at 61.5%, while Claude Opus 5.5 max and Astra on high reasoning effort each passed 59.6% of cases.
The findings highlight a central challenge for MSPs adopting AI-powered service desks: Technical capabilities alone do not guarantee successful service delivery. Even when AI agents completed complex tasks, they sometimes failed to follow customer instructions or established procedures, underscoring the importance of human oversight as providers expand automation.
AI models struggle with complex IT support tickets
AI systems can already handle a range of routine helpdesk tasks end-to-end, including password resets, employee onboarding and offboarding, license renewals, and firewall repairs.
Forge is currently handling more than 50% of all actionable tickets for Shield’s IT service provider partners, known as ShieldCos.
In the new evaluation, GPT-6 Astra with extra-high reasoning passed 61.5% of tickets (32 of 52), while Claude Opus 5.5 max and Astra on high effort each passed 59.6% (31 of 52).
The results showed that high model intelligence does not necessarily translate into consistent service delivery. Agents sometimes completed technically difficult work, but failed to follow customer instructions or established procedures.
Accuracy, cost, and speed complicate AI model selection
The evaluation highlights the tradeoffs between model performance, processing time, and cost.
Claude Opus 5.5 high passed 53.8% of cases, taking an average of 13 minutes per ticket at a cost of $3.30. Gemini 3.8 Flash high passed 36.5% of cases, taking 24 minutes per ticket at a cost of $1.60.
The testing also showed that increasing Astra’s reasoning effort from low to extra high resulted in two additional successful evaluations, but increased processing time from under 10 minutes to nearly 20 minutes and cost from $5.10 to $7.60.
DeepSeek V4.1 Flash low passed 25 tickets at a cost of $0.13.
Shield notes that these findings provide data to guide model selection, balancing accuracy, time, and cost based on the requirements of individual tickets. The organization also noted that the reported costs and times do not include human verification, corrections, or follow-through – which can add to the overall expense of using lower-performing models.
For MSPs, the findings suggest that selecting AI models based on processing costs alone could create operational challenges. Lower-cost models may offer savings on individual tickets, but those advantages can diminish when technicians must review, correct, or repeat unsuccessful work.
Providers evaluating AI service desk tools will need to consider total resolution costs, escalation rates, and customer satisfaction alongside model performance.
Customer instructions expose gaps in AI performance
Shield’s evaluations show that AI capabilities can vary significantly, even within familiar IT workflows.
Agents may successfully perform complex technical tasks while overlooking specific customer requirements or document procedures.
One example details how a customer requested a phone call about a slow computer, failed updates, and printing issues. Rather than arranging a call, Claude Opus 5.5 attempted to schedule remote maintenance through email, citing an earlier ticket as justification.
Shield notes that this case demonstrates that evaluating AI performance requires more than determining whether an agent can complete a technical task. Success also depends on whether the work meets customer requirements and adheres to established procedures.
Shield says the findings are already helping frontier AI labs identify weaknesses in IT workflows, with researchers indicating they plan to use the evaluations to guide model improvements.
AI adoption reshapes MSP technician responsibilities
Shield sees an opportunity to expand technicians’ roles beyond resolving individual tickets toward defining how AI systems should handle work as AI agents take on more service desk responsibilities.
“You go from owning individual tickets to owning the process of a ticket,” said Jon Katz, co-founder and COO of ShieldCo LISS Technologies.
Technicians help establish ideal outcomes, document which tasks can be delegated, and determine when human intervention is necessary. Shield said agents must have the authority to act within agreed-upon limits, while engineers must be able to incorporate technicians’ feedback into system improvements.
The organization is also working with its ShieldCos to explore how IT service providers may need to reorganize as AI assumes more work. At one provider, some helpdesk capacity has shifted toward account management, projects, and managed AI services. Another provider is restructuring workflows to create a larger role for escalation engineers and introduce team leaders responsible for overseeing AI engineers.
For MSP leaders, these changes raise broader questions about workforce planning and service delivery. As AI takes on more routine support work, providers may need fewer resources dedicated exclusively to ticket resolution and more employees focused on escalation management, customer relationships, and AI oversight.
The transition could also create opportunities to redirect technical capacity toward higher-value services without necessarily expanding headcount.