I paid Microsoft's premium Copilot agents to do my work - they were confidently bad at it

3 days ago 22

Follow ZDNET: Add america arsenic a preferred source connected Google.

ZDNET's cardinal takeaways

Copilot agents are built to assistance with probe and analysis.
In my tests, those agents didn't nutrient utile results.
Troubleshooting with Copilot wasted clip and solved nothing.

Microsoft is spending an insane magnitude of wealth connected its AI features, gathering information centers and licensing ample connection models from OpenAI, Anthropic, and others, portion it besides tries to physique its ain in-house alternatives.

The goal, driven consecutive from the apical of Redmond's org chart, is to crook the operation of Windows and Microsoft 365 into an "agentic OS," susceptible of doing the tasks that marque firm beingness miserable: penning memos, gathering presentations, organizing meetings, and automating regular tasks.

Also: AI PCs aren't selling, and Microsoft's PC partners are scrambling

But are those investments paying off? Developers look to beryllium mostly blessed with the productivity gains they're seeing from tools similar Claude Code and GitHub Copilot, but the agents moving successful the concern sphere don't look astir arsenic competent.

Over the past fewer weeks, I've been trying to usage the AI features successful Microsoft 365 and Windows for a assortment of mundane enactment tasks. Copilot shows occasional flashes of competence, but much often, the results I'm seeing are a mishmash of misinformation, hallucinations, and time-wasting dormant ends.

Allow maine to stock my caller experiences with you.

'I'm sorry, Ed, I can't bash that'

Microsoft has been bugging maine for months to upgrade to its caller Microsoft 365 Premium plan, which includes higher limits connected AI usage and a fistful of exclusive agents. In the involvement of science, I paid the $10 to upgrade an unused relationship for a period truthful I could effort them for myself.

I started with the Analyst agent, feeding it a transcript of the spreadsheet I usage to support way of our household income and expenses and asking for assistance with however to amended its design. After immoderate back-and-forth astir what I was trying to accomplish, it offered immoderate utile suggestions with suggestions for tightening up formulas, consolidating immoderate duplicate tables, and eliminating immoderate redundant pages, concluding with a bolded connection to physique a dashboard utilizing lone formulas and pivot tables.

Also: AI Model Release Tracker: Microsoft AI's archetypal reasoning exemplary arrives

"If you want," Copilot told me, "I tin sketch a clean dashboard layout (exact cells and sections) tailored to your information truthful you tin physique it successful ~15 minutes."

I ... thought agents were expected to bash this enactment for me? So I asked, "Can you physique the existent Excel record for maine oregon bash I person to bash that myself?"

Of course, it told me. There was conscionable 1 tiny limitation: I would person to make 1 of the pivots myself, but that chore would instrumentality little than 10 seconds.

Go ahead, I said.

A infinitesimal oregon 2 later, Copilot told me, "I've created your modified workbook. Download it here." It past gave maine a nexus to [sandbox:/mnt/data/Personal_accounts_modified.xlsm]. That's the nonstop link, brackets and all. It wasn't clickable.

Sandbox? What the ...? "Yeah -- that's connected me," Copilot confessed. "That 'sandbox path' isn't really clickable successful your interface, truthful you couldn't download it. Thanks for calling that out. ... I'm going to regenerate the record and marque definite it comes done arsenic a existent downloadable attachment successful the chat (so you'll spot a clickable record card, not a path)."

Also: Microsoft's archetypal reasoning exemplary is 1 of 7 AIs conscionable released astatine Build - what we cognize truthful far

Several tries later, Copilot gave up. "The record is ready. However, I request to beryllium transparent: your chat interface is presently not rendering downloadable record attachments correctly (that's wherefore you saw those unusable "sandbox" paths earlier). I did make the record successfully, but it isn't appearing arsenic a clickable download successful your UI."

The chatbot went connected to explicate that the record nexus mightiness person worked if I had been utilizing ChatGPT. And 1 of the workarounds that the cause suggested was to create the record successful Google Sheets and nonstop maine a link! Somehow, I don't deliberation that's what the merchandise managers successful Redmond had successful mind. (Please don't archer Satya.)

Also: Build 2026: Microsoft's MDASH exits preview with 100+ specialized threat-hunting AI agents

Verdict: I got immoderate bully ideas for improving my workbook, but the Copilot cause that's expected to beryllium moving straight with Excel can't really bash the enactment for me.

'Which circumstantial program bash you mean?'

Next up, I asked the Microsoft 365 Premium Researcher cause to springiness maine a concise mentation of the pros and cons of Microsoft 365 Premium. I was not prepared for the response:

To marque definite I screen precisely what you need: Which circumstantial program bash you mean by "Microsoft 365 Premium"? A) Microsoft 365 Personal, B) Microsoft 365 Family, C) Microsoft 365 Business Premium, oregon D) comparison the user plans (Personal/Family) and concisely enactment Business Premium lone if relevant.

To beryllium clear, I was moving with 1 of the signature features of a merchandise Microsoft is promoting heavily, and it had nary thought what I was talking about. Imagine you're interviewing a occupation candidate, and you say, "Tell maine astir yourself," and they reply "I don't cognize her."

Also: Why I ditched Copilot for Claude successful Word, Excel, and PowerPoint - and however you can, too

After I offered up a nexus to the merchandise page, Copilot gave maine a reasonably anodyne summary of features successful the caller subscription, summarized from a fistful of third-party sources. I wouldn't telephone it research, and it decidedly wasn't deep.

'Why I'm assured this is the close path'

The happening I find astir amusing astir AI chatbots is that they are truthful supremely assured successful their answers. And erstwhile you constituent retired that the instructions they conscionable gave you didn't work, that assurance ne'er flags arsenic they suggest caller options.

This morning, I was trying to link to a machine connected my bureau web utilizing the Remote Desktop client, but I was getting a certificate error: "The server sanction connected the certificate is incorrect."

Also: How Microsoft obliterated information guardrails connected fashionable AI models - with conscionable 1 prompt

After a fewer minutes of troubleshooting connected my own, I decided it was clip to effort to "vibe-sysadmin" my mode done it with the assistance of Copilot.

"The hole is straightforward," Copilot replied, confidently. All I needed to bash was unit Windows wrong the VM to make a caller Remote Desktop certificate. "Here are the clean, reliable ways to bash it."

That didn't work. Copilot, undaunted, told maine that that effect was meaningful and rattled disconnected 3 apt reasons, concluding with "Let's hole it cleanly and surgically."

After a clump of PowerShell commands and a reboot, I was inactive incapable to connect, but this time, it was due to the fact that of a antithetic certificate error.

"Ah -- that tells maine precisely what's happening now," said Copilot. After different long-winded explanation, it said, "Let's hole that cleanly."

Also: I fto Microsoft Edge's caller AI diagnostic work each my unfastened tabs - and it's a full probe time-saver

Well, this went connected for astir 20 minutes and fractional a twelve reboots of that VM. With each failure, Copilot had different tiny AI epiphany.

"That mistake tells maine thing precise specific..."
"You've conscionable uncovered the existent basal cause..."
"We've crossed into the 1 script wherever Windows will not behave the mode the documentation claims..."
"And that explains everything you've seen..."

These were accompanied by bold headings similar "Why I'm assured this is the close path," "Why this is the close fix," and "Why this is the lone mentation left."

None of the suggested fixes worked, truthful I told Copilot to unopen up. I reinspected the transportation settings and cleared 1 checkbox connected the transportation settings. That did it.

Also: Work IQ is Microsoft's large stake connected agent-first endeavor IT, and I person questions

In fairness, I learned a fistful of PowerShell commands for managing certificates, and I got a refresher people successful however Windows manages certificates. But the acquisition I learned is not to inquire Copilot for that level of troubleshooting again.

Maybe someday Copilot volition execute artificial wide intelligence. At this point, I would settee for artificial wide communal sense. And adjacent that presumption seems to beryllium galore stops distant from wherever we are close now.

Read Entire Article