Tag: Qualcomm Snapdragon X processors

On-device or on-premises artificial intelligence–what does it offer

simonmackay 11/09/2024 Uncategorized No Comments

A significant direction for artificial intelligence, especially advanced AI like generative AI, is to have the AI processes performed on the same device or within the same premises as the users who will benefit from it.

This can be considered as part of edge computing because it involves the pre-processing of data before it is sent to a cloud-driven AI platform or post-processing of data coming back from a cloud-driven AI platform.

What is desireable about this is energy efficiency for cloud-based AI, reduced data transfer requirements or assurance of user privacy, corporate confidentiality and data sovereignty due to the minimum amount of data processed in an online environment. Apple even takes this further by running a private cloud specific to each Apple platform user to cater for more intense processing that can’t be performed on the device itself.

On-device and on-premises AI relies primarily on a smaller language model compared to the large language models that cloud-based AI services like ChatGPT rely on. Here they are focused on the data that exists on or is likely to come in to the machine or the logical network. Here, this cam allow for improved data management or permit a custom language model that represents personal or corporate desires.

Why on-device and on-premises AI

Samsung Galaxy AI press image courtesy of Samsung

Samsung Galaxy AI representing on-device artificial intelligence on Android mobile devices

A key desire is data security and end-user privacy. Here the data never leaves the device or premises for artificial-intelligence / machine-learning processing. This satisfies business and industry compliance expectations like privacy, corporate confidentiality and data sovereignty requirements.

Another benefit is improved performance and personalisation when it comes to artificial-intelligence processing and machine learning. Here, the processing takes place on a local machine thus avoiding the use of oversubscribed cloud computing services that can underperform under load. The AI language model ends up being highly personalised thus becoming lean.

Apple iPhone 16 press image courtesy of Apple

Apple iPhone 16 Series – first iOS device with on-device AI processing

There is reduced energy consumption compared to sending the data out to cloud-computing data centres. You also see efficient use of telecommunications links due to smaller amounts of data being sent using them as well as seeing efficient use of cloud-computing services. Another key benefit to see is improved service resilience because you aren’t heavily dependent on online resources – what if the network link fails.

Hybrid (cloud+on-device / on-premises) AI setups can allow for more sophisticated artificial-intelligence / machine-learning processing and working with multiple custom environments. This is due to having a lot of the data handling done locally before it is submitted or after results are received.

On-device AI processing

Lenovo Yoga Slim 7X 2-in-1 laptop with Qualcomm Snapdragon X Elite silicon press image courtesy of Lenovo

Lenovo Yoga Slim 7X 2-in-1 laptop with Qualcomm Snapdragon X Elite silicon – implementing on-device AI under Windows 11 for ARM microarchitecture

On-device AI is about having the AI data handled by the same device that is to make use of the data. This is facilitated through either a third processor called a neural processing unit (NPU) or a very powerful general processor that sets aside processor cores for neural processing to answer AI tasks.

A good analogy to think of are some NAS units that have a graphics processor in addition to their primary CPU. Here, these devices use the graphics processor for accelerated datatype translation like converting multimedia files in to other formats. or similar processing tasks.

There will also be an expectation to have a lot of RAM and storage capacity on these devices. This is something that is being answered easily thanks to Moore’s Law where cost of increased storage and RAM is being reduced significantly.

Such setups can be facilitated either on regular computers or mobile computing devices like smartphones and tablets.

On-premises AI processing

New Dell XPS 13 with Intel Lunar Lake Core Ultra processor press image courtesy of Dell

Dell has offered an XPS 13 laptop with Intel Lunar Lake Core Ultra CPU which has on-device AI processing for Windows 11 under IA microarchitecture

The on-premises AI approach would rely on a server or NAS on the same logical network as the end-users to process the AI data. This would come in to its own with on-premises or hybrid cloud computing setups where the desire is to keep the important data on the user’s premises.

This could represent a server or NAS that uses artificial intelligence and machine learning to make sense of a data set stored therein; or a server or NAS could perform AI tasks for client computers that don’t have on-device AI abilities. This can even lead to the creation of local chatbots that supply answers based on locally-held organisational data.

The trends associated with on-device and on-premises AI

Apple Intelligence writing tools on MacOS screenshot courtesy of Apple

Even MacOS is now supporting on-device artificial intelligence on the latest Macs with Apple silicon.

2024 has effectively become the year of general-purpose on-device AI processing with both the mobile-platform devices that run mobile operating systems and the regular computers that run desktop operating systems.

Some of the premium Android smartphones, tablets and smartwatches from the likes of Samsung and Google that are introduced in 2024 are being equipped with AI functionality. These implement Qualcomm Snapdragon mobile ARM64 processors and use this technology for voice-to-text, advanced search, machine translation, photo editing and similar functionality. Apple is introducing this kind of on-device AI to their latest iPhones and iPads powered with their latest silicon as part of Apple Intelligence, their branding of on-device AI. Here, this offers AI-driven inbox management, document and recording summarisation, image editing, AI-driven emojis and similar functions.

As well, during this year, Microsoft built in to Windows 11 on-device AI functionality which comes alive on computers that have neural-processing units. This is marketed as CoPilot+ and is being offered on laptop computers that use Qualcomm Snapdragon X (ARM64) silicon or, shortly, Intel Lunar Lake Core Ultra and AMD Strix Point (IA-64) silicon. These offer video transcription and captioning, image creation and editing, video editing, document summarisation amongst other things. This has been underscored by a deluge of CoPilot+ AI-capable laptops being launched or given their first outing at the Internationaler Funkaustellung 2024 in Berlin with some of the units equipped with Intel silicon and others with Qualcomm Snapdragon X silicon.

Apple is also offering a similar kind of artificial intelligence for the latest Macintosh computers with the latest Apple M-series silicon. This will offer the same kind of features as their iOS and iPadOS implementations but with a richer interface. For all the Apple operating systems, there is support for hybrid ChatGPT operation with a “private cloud” arrangement to protect users’ data.

QNAP and Synology are working on equipping newer NAS units and newer versions of the NAS operating systems for artificial intelligence with AI being seen as part of a NAS’s feature set. But this will primarily be about managing or indexing data held on these devices themselves but someone even prototyped a NAS-based local ChatGPT setup as a proof of concept about on-premises generative AI setups which would then be about secure AI operations.. There will be the idea of using business or enthusiast grade NAS units as part of edge-computing setups to permit pre-processing of data before submitting to cloud-based AI.

Conclusion

On-device and on-premises artificial intelligence including hybrid setups such as edge-based AI or private cloud AI is expected to be a key turning point for this technology. This will most likely be due to a call for secure private and bespoke data handling requirements coming about and to keep generative AI technology relevant for most users.

Microsoft makes ARM-64 regular computers more legitimate

simonmackay 30/05/2024 Computer Hardware Design, Computer Systems, Current and Future Trends, Desktop Computers, Laptop, Notebook and Netbook Computers No Comments

Through the late 1980s and the 1990s, there were a range of regular personal desktop computers that used various forms of processors that implemented RISC (Reduced Instruction Set Computing) technology.

The most common of these were the Apple Mac computers that used Motorola PowerPC silicon and existed before Apple implemented Intel silicon with examples like the original iMac. As well, the Sony PS3, Microsoft XBox 360 and Nintendo Wii games consoles implemented PowerPC RISC silicon at the heart of these devices so you may have played with this technology without knowing it.

But there were some computers with niche appeal like the SGI Indigo or the Sun Microsystems SPARC-based workstations that used limited-appeal high-power RISC chipsets. Acorn even ran a range of computers pitched to the education sector in the form of the Archimedes and RISC PC that were the first to implement today’s ARM RISC technology.

These systems were more about maximum graphics and multimedia power or high-load workstation-class computing that was to be achieved in an efficient manner. But Intel and Microsoft had brought a very similar level of power to computers based on their traditional i86-based CISC (Complex Instruction Set Computing) processors.

Through the 2000s and 2010s, Apple implemented Intel i64-based (64-bit i86 microarchitecture) silicon in their lineup of MacOS-based regular computers and adapted to this new microarchitecture. Now they have licensed ARM-64 (64-bit ARM RISC) microprocessor technology and used that to build their own M-Series system-on-chip processors.

Qualcomm Snapdragon X Elite processor press image courtesy of Qualcomm

Qualcomm Snapdragon X Elite processor chip – maturing ARM64 RISC computing for the Windows-based regular computer

Now Microsoft is developing their desktop operating system, application software and software-development tools to also work with ARM-64 RISC microarchitecture. They are also partnering with Qualcomm to work on a series of Snapdragon ARM64-based system-on-chip microprocessors for use in laptops that run Windows with this effort coming to maturity this year in the form of the Snapdragon X system-on-chip processor.

This is because ARM-based RISC computing is being used in portable and low-power computing setups like mobile-platform smartphones and tablets. It is also being implemented in set-top boxes, smart TVs, network-attached storage devices and similar devices where a low-profile or flexible design is being preferred. For portable devices, this will be about longer battery life such as many days on a charge or being equipped with smaller batteries. For statioary devices, it will be about compact or flexible power-efficient device designs.

Dell Inspiron 14 Plus CoPilot+ laptop with Qualcomm Snapdragon X Elite silicon

As well, Apple and Microsoft are moving towards ARM64 microarchitecture for their regular-computer hardware and software to slow down the decline in business and consumer interest in this client-side computer class.

For Microsoft, Windows 11 has made it possible to emulate a 64-bit Intel operating environment on ARM64 computers like those using the Qualcomm Snapdragon X system-on-chip. This would allow most of today’s software and games to run on these computers. As well, Qualcomm and Microsoft are driving the on-device AI abilities associated with Snapdragon X by marking these computers as “CoPilut+ computers”.

These computers will answer most mainstream computing tasks especially on highly-portable or “all-in-one” computers. But to see stronger appeal, there may be market pressure to have a wide range of core games or creative software developed for or ported to ARM64 platforms.

Lenovo Yoga Slim 7X 2-in-1 laptop with Qualcomm Snapdragon X Elite silicon

Creative software that was written for or ported to Apple Macs using M-Series silicon could just as easily be developed for ARM-based Windows computers. This has also increased the validity of regular computers using ARM64-based technology amongst the creator / prosumer community. Similarly games studios that wrote for games consoles that have used PowerPC or ARM technology as well as i86/i64 technology desktop computers will also be able to adapt easily to this new reality.

At the moment, the regular-computer scene will still end up as a “horses for courses” environment with Intel/AMD-based silicon being for high-power computing tasks like core gaming or certified workstations, or where the highest level of hardware and software compatibility is desired. That is while the ARM64-based computers will hold their ground for an increasing amount of mainstream computing tasks.

But I would still consider the ARM64-based computers as being a viable alternative to i64-based Intel or AMD powered regular computers that run Windows or Linux as their operating systems.