Generative AI on a Phone?

Generative-AI models such as ChatGPT are typically large and run only in the cloud, but Qualcomm recently demonstrated a smaller image-generation model called Stable Diffusion performing inference on a smartphone.

Linley Gwennap

Generative-AI models such as ChatGPT are typically quite large and run only in the cloud, but Qualcomm recently demonstrated a smaller image-generation model called Stable Diffusion performing inference on a smartphone containing its Snapdragon 8 Gen 2 processor. The demo is supposed to show that despite having relatively few resources, a phone can run large AI models. But even after the company made compromises to image quality, the model still required nearly 15 seconds to complete.

Generative AI is an emerging category of neural networks. Most AI models transform a complex input, such as an image or sound waves, into a simple text output such as the type of object in the image (cat, house) or the words spoken. Generative models instead transform a short text input into a complex output such as an image or paragraphs of text. These models may produce different outputs even when using the same input prompt.

By now, almost everyone has heard of ChatGPT, a free (for now) service that can create detailed and human-like responses to simple text prompts. Microsoft has invested heavily in OpenAI, the company that developed ChatGPT, and has already incorporated the technology into a beta version of its Bing search engine. ChatGPT is a massive model that takes weeks to train, but updating it takes less effort, and running the model (inference) requires much less compute power. Image-generation models such as Stable Diffusion and Dall E are smaller than ChatGPT but still large by smartphone standards, and they require extensive compute power to inference.