Distilled Giants: Why We Must Rethink Small AI Development

In recent years, the race to develop increasingly larger AI models has captivated the tech industry. These models, with their billions of parameters, promise groundbreaking advancements in various fields, from natural language processing to image recognition. However, this relentless pursuit of size comes with significant drawbacks in the form of high costs and significant environmental impact. While small AI offers a promising alternative, providing efficiency and lower energy use, the current approach to building it still requires substantial resources. As we pursue small and more sustainable AI, exploring new strategies that address these limitations effectively is crucial.

Small AI: A Sustainable Solution to High Costs and Energy Demands

Developing and maintaining large AI models is an expensive endeavor. Estimates suggest that training GPT-3 costs over $4 million, with more advanced models potentially reaching high-single-digit millions. These costs, including necessary hardware, storage, computational power, and human resources, are prohibitive for many organizations, particularly smaller enterprises and research institutions. This financial barrier creates an uneven playing field, limiting access to cutting-edge AI technology and hindering innovation.

Moreover, the energy demands associated with training large AI models are staggering. For example, training a large language model like GPT-3 is estimated to consume nearly 1,300 megawatt hours (MWh) of electricity—equivalent to the annual power consumption of 130 U.S. homes. Despite this substantial training cost, each ChatGPT request incurs an inference cost of 2.9 watt-hours. The IEA estimates that the collective energy demand of AI, data centers, and cryptocurrency accounted for nearly 2 percent of global energy demand. This demand is projected to double by 2026, approaching the total electricity consumption of Japan. The high energy consumption not only increases operational costs but also contributes to the carbon footprint, worsening the environmental crisis. To put it in perspective, researchers estimate that training a single large AI model can emit over 626,000 pounds of CO2, equivalent to the emissions of five cars over their lifetimes.

Amid these challenges, Small AI provides a practical solution. It is designed to be more efficient and scalable, requiring much less data and computational power. This reduces the overall costs and makes advanced AI technology more accessible to smaller organizations and research teams. Moreover, small AI models have lower energy demands, which helps cut operational costs and reduces their environmental impact. By utilizing optimized algorithms and methods such as transfer learning, small AI can achieve high performance with fewer resources. This approach not only makes AI more affordable but also supports sustainability by minimizing both energy consumption and carbon emissions.

How Small AI Models Are Built Today

Recognizing the advantages of small AI, major tech companies like Google, OpenAI, and Meta have increasingly focused on developing compact models. This shift has led to the evolution of models such as Gemini Flash, GPT-4o Mini, and Llama 7B. These smaller models are primarily developed using a technique called knowledge distillation.

At its core, distillation involves transferring the knowledge of a large, complex model into a smaller, more efficient version. In this process, a “teacher” model—large AI model—is trained on extensive datasets to learn intricate patterns and nuances. This model then generates predictions or “soft labels” that encapsulate its deep understanding.

The “student” model, which is small AI model, is trained to replicate these soft labels. By mimicking the teacher’s behavior, the student model captures much of its knowledge and performance while operating with significantly fewer parameters.

Why We Need to Go Beyond Distilling Large AI

While the distillation of large AI into small, more manageable versions has become a popular approach for building small AI, there are several compelling reasons why this approach might not be a solution for all challenges in large AI development.

• Continued Dependency on Large Models: While distillation creates smaller, more efficient AI models and improves computational and energy efficiency at inference time, it still heavily relies on training large AI models initially. This means building small AI models still requires significant computational resources and energy, leading to high costs and environmental impact even before distillation occurs. The need to repeatedly train large models for distillation shifts the resource burden rather than eliminating it. Although distillation aims to reduce the size and expense of AI models, it doesn’t eliminate the substantial initial costs associated with training the large “teacher” models. These upfront expenses can be especially challenging for smaller organizations and research groups. Furthermore, the environmental impact of training these large models can negate some of the benefits of using smaller, more efficient models, as the carbon footprint from the initial training phase remains considerable.
• Limited Innovation Scope: Relying on distillation may limit innovation by focusing on replicating existing large models rather than exploring new approaches. This can slow down the development of novel AI architectures or methods that could provide better solutions for specific problems. The reliance on large AI restricts small AI development in the hands of a few resource-rich companies. As a result, the benefits of small AI are not evenly distributed, which can hinder broader technological advancement and limit opportunities for innovation.
• Generalization and Adaptation Challenges: Small AI models created through distillation often struggle with new, unseen data. This happens because the distillation process may not fully capture the larger model’s ability to generalize. As a result, while these smaller models may perform well on familiar tasks, they often encounter difficulties when facing new situations. Moreover, adapting distilled models to new modalities or datasets often involves retraining or fine-tuning the larger model first. This iterative process can be complex and resource-intensive, making it challenging to quickly adapt small AI models to rapidly evolving technological needs or novel applications.

The Bottom Line

While distilling large AI models into smaller ones might seem like a practical solution, it continues to rely on the high costs of training large models. To genuinely progress in small AI, we need to explore more innovative and sustainable practices. This means creating models designed for specific applications, improving training methods to be more cost- and energy-efficient, and focusing on environmental sustainability. By pursuing these strategies, we can advance AI development in a way that is both responsible and beneficial for industry and the planet.