The old saying goes that, with tech, you should never buy the first generation of anything new. Wait for the devs to work out the kinks, then check back. We’re now two years into the AI “revolution,” and we’re being dragged into the third. AI should be the next big thing already; the ruffles should have been smoothed out, and the puzzle pieces should all fit. It’s not there yet. This year was big on AI, but next year will show the true promise of on-device artificial intelligence come alive. Where have we heard that one before?
AI has not lived up to many of the promises put forth by tech companies, both big and small. In 2024, AI-specific devices fell flat. AI on Mac or PC hasn’t made a strong impression either. There hasn’t been a wave of AI applications that use new laptop’s neural processors, and most applications rely on cloud computing. The main AI applications seem to be coders finding ways to kill their own industry. Otherwise, grifters are using AI to fill the internet with fakes, junk, and slop. On-device AI pushes regular consumers to write or summarize emails with AI. That doesn’t exactly sound like the killer AI app.
That’s why big tech is now pushing “agentic” AI. Companies promise large language models will do all your busywork for you seamlessly, non-intrusively. Perhaps, with agents, AI can come alive in 2025. We have only seen some demos of how this AI will work. Surveys show that current AI features don’t enthuse Apple and Android users. In essence, big tech needs agentic AI to take off. Without it, regular users will wonder what the fuss was for. We don’t know how these AI agents will work next year, but we know exactly how Silicon Valley will push it to users, whether we want them or not.
Nobody Has Cracked the AI Wearable
This year brought us a slew of AI wearables and handheld devices, like the Humane AI Pin and the Rabbit R1. Both devices launched far too soon, with obtuse software that effectively provided little more than quick access to an AI chatbot like ChatGPT.
There was an avalanche of bad products so big we didn’t have the time to cover it all. I’ve used Timekettle’s X1 Interpreter Hub, a pocket-sized translator stick that touts its AI translation capabilities. It could hold its own back and forth from English to Spanish in our tests. However, trying English to Urdu would start inputting random Pakistani celebrities or references to God in the middle of an interpretation. It was insulting and hilarious in equal measure to my Urdu-speaking colleague. It did worse in some other languages than the Google Translate app.
And it wasn’t just smaller brands that couldn’t meet the full promise of device-specific AI. Meta’s Ray-Ban glasses‘ AI image recognition features sometimes struggle to comprehend what’s in front of them. At least those glasses can still take pictures without needing cloud-based AI, something other devices can’t manage. The $700 Humane AI Pin didn’t live up to its lofty promises. Reviewers noted it would often fail to identify objects in front of it correctly, and even when it was accurate, it was hampered by poor battery life and heat issues. Humane later recalled the charging pack due to concerns over fire risks. Once valued at around $850 million, the company reportedly saw more returns than sales into the middle of the year.
The promise of device-specific AI was squashed again and again. The Rabbit R1 launched a few weeks after the Humane pin. CEO Jesse Lyu directly compared his $200 device to his rivals and claimed his “personalized operating system” and “Large action model” would be your true AI assistant. The launch was a disaster. Users quickly opened the LAM and found that the Android-based OS could run on phones. Most of its capabilities were facilitated through the cloud. The device could also connect to some outside apps, but white hat hackers and developers found they could access user data also available to internal Rabbit staff.
There has been more AI-centric hardware, like the Plaud NotePin, which offers AI-based transcription and note-taking. It works thanks to a limited use case. Inevitably, you will ask whether your current device can handle these same capabilities. Google has Pixel Recorder, and iPhones and Macs have voice memos with transcription capabilities.
To their credit, AI hardware developers have tried to improve their devices. In November, Rabbit updated its OS to allow “custom AI agents” with a Teach mode. This was essentially promised with the LAM half a year ago. The mode is still in beta, but the problem remains that the device does not have direct access to the apps you want it to use.
In December, Humane started promoting its CosmOS, “built from the ground up for AI,” to devices outside the AI Pin. They want to put it in cars, use it for smart home tech, and even stick it in your TV to analyze on-screen action. The “intelligent conductor” will essentially operate like any other agentic offering, digging into your devices and information to perform tasks on your behalf.
The switch from “AI device” to “AI agent device” was seamless. The promise of these devices failed to impress, but they now use the same hype strategy for agentic AI. We expect more of these kinds of devices at CES 2025 next month. They’ll use the same language for “AI assistant,” but it will be in the new Agentic flavor of the week. The jury is out on whether they’ll be good, but it doesn’t look good if these devices can’t figure out something your phone doesn’t already do.
The ‘AI PC’ Has Yet to Materialize
Chipmakers like Intel and Qualcomm hammered home the point about their neural processors or NPUs. That was the story with Qualcomm’s Snapdragon X Elite and X Plus chips. Microsoft christened any PC with Qualcomm’s ARM-based chip, a “Copilot+ PC.” All those “AI PCs” with Intel’s Meteor Lake were left out in the cold.
I sat in front of Intel in January and asked one of the company’s senior VPs, Sachin Katti, whether the initial run of “AI PCs” was truly capable of running AI on-device. Yes, they could, he told me. The only issue was the lack of apps. For the first time in the history of tech, the technology outpaced the available applications. It was up to the developers to meet demand, he said.
The biggest AI apps in 2024 were chatbots—like Perplexity, Claude, ChatGPT, and more—none of which required on-device AI processing. Then came Copilot+. It was the turning point for ARM-based chips on PC with the new Qualcomm Snapdragon X Elite and X Plus. Each chip had an NPU capable of 45 TOPS, or trillions of operations a second (a derived value that’s arguably not great at describing AI capabilities). None of those previous Intel chips met the requirements to be Copilot+. It wouldn’t be until AMD’s Strix Point and Intel’s Lunar Lake months later that Team Blue and Team Red could claim the coveted Copilot+ moniker.
Using those features was another matter. The PCs shipped with the new Copilot button for instant access to Microsoft’s favored chatbot. However, the only on-device AI features included were a few AI image generators and live captions on video calls or in videos. Microsoft’s premiere AI feature, Recall, was supposed to give your PC “photographic memory” by screenshotting everything you did and then transcribing it with AI.
Microsoft delayed the feature just before many OEMs planned to release their first laptops. Security researchers proved that screenshot transcriptions could be accessed without any real security layer. Microsoft only allowed Windows 11 beta testers access to the feature in November. Judging by the latest beta build, Recall still requires some fine-tuning. It works. If you’re okay with your life and some potentially sensitive info being screenshotted, it’s handy for those with bad memories.
Then you get to Apple, and the current AI features arrived so late in 2024 that it was better if they were all delayed until 2025. The latest macOS Sequoia 15.2 stable build arrived in December, bringing the Image Playground and ChatGPT integration with Siri to Macs. At the very least, you only need an M-series Mac to access these features, unlike the iPhone, which requires an iPhone 15 Pro or iPhone 16 model.
If you have an older Apple device, you’re not missing anything. Image Playground creates cartoonish images of you or your friends with faces that look like a cross between a lazy caricature artist and big-head mode in an old-school video game. ChatGPT Integration offers little more than a typical Google search. It also makes it difficult to find past chats through the built-in widget, which is now prominently on the top toolbar.
The NPUs for these devices can only run simplistic or background AI tasks. For more complex AI tasks, like running the top-end AI models promoted by these companies, you need a GPU. A Nvidia GeForce RTX 4090 can do upwards of 1,300 TOPS, 26 times what today’s top-end on-chip NPUs can do. In December, Nvidia launched the $250 Orin Nano, which was built specifically for running AI applications locally. The processor promises 67 TOPS.
While AI Hits ‘the Wall,’ Agentic AI Needs to Take Up the Slack
The latest and greatest Gemini models are available to new Chromebook Plus owners, so I’ve become acquainted with Google’s on-device AI, even beyond phones. In December, Google brought out Gemini 2.0, the advanced mode for Gemini Advanced subscribers. You would have to be a very dedicated user to tell the difference between models. The new version should have better coding and language ability, but if you only use it for text, the difference is that 2.0 Pro will be more verbose than 1.5 Pro.
A big reason AI is becoming “agentic” is “the wall.” In AI circles, it’s the colloquial term for how providing more training data to AI results in diminishing returns. OpenAI cofounder Ilya Sutskever, who hasn’t minced words about his former employer, told a conference crowd in Vancouver that AI developers are running out of data to train AI models, saying, “We have to deal with the data that we have. There’s only one internet.” That’s not to say AI models can’t improve. Sutskever, now a co-founder of the startup AI Labs, previously told Reuters that the age of “scaling” is over and that now is the time of “discovery.”
Newer models, like OpenAI’s GPT-o1 model, are designed with better reasoning in mind. But better benchmarks don’t necessarily result in better results for a base user. If you’re not already impressed with today’s AI models, you probably won’t be with next year’s big releases. That’s why OpenAI is promoting Altera AI agents, and reports hint Sam Altman’s big AI company will launch an autonomous AI agent codenamed “Operator.”
That’s why agents have to take off. Anthropic, the makers of Claude, offered us a taste of what this entails in a demo released in October. Demos show how users could ask Claude 3.5 Sonnet to access Google Chrome, type out a Google search, and then add an event to the users’ calendar.
We’re trying something fundamentally new.
Instead of making specific tools to help Claude complete individual tasks, we’re teaching it general computer skills—allowing it to use a wide range of standard tools and software programs designed for people. pic.twitter.com/42u8VeTvXd
— Anthropic (@AnthropicAI) October 22, 2024
It’s an entertaining demo, though you’re offering the AI a deep look into your personal life. Anthropic noted that the AI accidentally stopped the company’s screen recording at one point, which was all on its own. If the AI fails in any one part of a long chain of tasks, it can cause a cascade of issues for the entire prompt. Imagine if it books the wrong flight for you or puts the wrong time on your calendar for when you’re supposed to pick up your mother from the airport.
Late last year, I speculated about the rise of AI on PC. This was before Microsoft brought the Copilot key kicking and screaming into this world. I wondered what it would be like if AI could take over my PC and control settings without digging through Windows settings. Imagine telling your PC to bring up the controls for your laptop’s brightness setting without needing to surf through either Windows or whatever bloatware was first included on your device. What if it could do this without an internet connection, using models housed on-device so I don’t have to worry about outside agencies accessing my emails or calendars?
Settings aren’t sexy, but making it easier for users would be a boon. Apple has promised that Apple Intelligence will instead be the kind of everyday-life assistant. It wants you to imagine if every iPhone, iPad, or Mac user had a butler capable of diving into your emails, pulling out the necessary information, and turning that into a calendar event.
Agentic AI Has Privacy Implications, and We Don’t Know How Big Tech Will Address It
Agentic applications give AI access to a lot of your sensitive information. This isn’t the sort of AI that can be handled on-device; it requires cloud processing. Apple promises to keep your information safe with a private cloud computing structure that creates a firewall between your information and the company’s servers.
So far, Microsoft’s agent initiatives have focused on their enterprise end, specifically for those using 365 apps in business settings. It promotes a Copilot Studio for businesses to create their in-house AI agents.
As its FAQ states, OpenAI has direct access to your chat logs on ChatGPT, but it claims it’s limited to “authorized personnel.” Google has not spelled out its privacy plans for when Gemini goes agentic, but the company does have access to your activity, including your chats. It claims it uses this information to “improve Google products and machine-learning technologies.”
Agentic AI is coming. Over time, it will slide onto our phones, computers, and other devices under the banner of “experimental” or “beta” features. Major chipmakers will continue to tout the TOPS value of their new CPUs, and Google, Microsoft, and Apple will try to outrace each other with their AI-based assistants. It will be the same old, in the endless march of hype.
Read the full article here
