The Open Source Promise of Large Language Models

The release of the open-source LLM Vicuna-13B was a watershed moment in the world of AI. It showed that open-source LLMs were not only possible, but they could also be just as powerful as closed-source models. This has led to a surge of innovation in the open-source LLM community, with developers developing new training techniques, improving the quality of models, and making them more accessible to a broader range of users.

By Jay Limburn

Technology is evolving at break-necking speeds, and companies and individuals are building their own Machine Learning (ML) models like never before. Predictive AI models can now underpin mission-critical tasks, giving organizations a window into the future instead of a rundown of the past while helping them operate quicker and leaner.

When ChatGPT was launched to the public in November 2022, there was an historic surge of interest in large language models (LLMs). The topic of LLMs is becoming inescapable, as AI models are being adopted rapidly throughout industry and academia. LLMs exhibit remarkable understanding and generative capacity, propelling language tasks to new heights.

Recently, significant progress has been made on prompt-driven models, e.g., ChatGPT1 and GPT-3.5. Following instructions or prompts in natural language, they can generate professional and contextual responses conversationally. However, the further prevalence of instruction-following models is primarily impeded by the closed-source restriction and high development costs.

The Prometheus Effect

In Greek mythology, Prometheus was a Titan who stole fire from the gods and gave it to humanity. This act of defiance profoundly impacted human civilization, allowing humanity significant progress from hunter-gatherers to farmers and eventually to the modern world.

Akin to the fire Prometheus brought to humanity, the open-source community got its hands on a robust foundation model for the first time at the beginning of March. When Meta’s LLaMA was leaked, it came with no instructions or prompt tuning. None were needed; the mortals knew the power of what was handed to them. Innovation spread like wildfire across the open-source LLM community.

Shortly after LLaMA became publicly available, a group of developers created their own open-source LLM called Vicuna-13B. Vicuna-13B was trained on a massive dataset of text and code, and it achieved comparable performance to ChatGPT. The release of Vicuna-13B was a watershed moment in the world of AI. It showed that open-source LLMs were not only possible, but they could also be just as powerful as closed-source models.

One of the most promising research areas is the development of low-rank adaptation (LoRA). Lora is a technique that allows developers to fine-tune LLMs at a fraction of the cost and time. This means anyone with a computer can train an LLM without spending millions of dollars on expensive hardware.

The development of LoRA and other open-source LLMs is profoundly impacting the AI community. It democratizes AI, making it accessible to everyone, not just the big tech companies. This is the Prometheus Effect. Just as Prometheus gave fire to humanity, the open-source community is giving us the power of AI.

Innovation burns bright

With every significant technological milestone, human capacity becomes augmented. The advent of the internet brought us the multi-million dollar garage startup—where innovators could spark change with little more than an idea. What will the AI revolution bring?

It’s already unfolding in front of our eyes. While first-movers still hold a slight edge, the innovation gap is closing astoundingly fast. When it comes to LLM training and experimentation, a single person with a gaming laptop can achieve in an evening what previously took an entire research organization months.

Open-source models are not only more customizable, they are evolving faster, while becoming pound-for-pound more capable. They are doing things with $100 and 13B parameters that incumbents struggle to do at $10M and 540B. And they are doing so in weeks, not months.

New milestones and significant developments are separated by mere days. A few weeks after the LLaMA leak, hackers got variants running on TI-84 calculators. Within a month the open source community developed instruction tuning, quantization, quality improvements, multimodality, and RLHF… each improvement building on the next.

The future of AI is open source

Many questions remain in this constantly evolving AI landscape. Will closed-source models constantly struggle to keep up with the pace of innovation in the open source community? Are open source LLMs set to become the standard for AI and be used in various applications, from healthcare to education and customer service? What are the risks of businesses adopting open source models?

What we do know is the future of LLMs is being driven by the open-source community. They are more agile, more innovative, and more focused on the needs of users. They are also more democratic, allowing everyone to participate in developing this transformative technology.

The open source community is already making a significant impact on the field of LLMs. They have developed new training techniques, improved the quality of models, and made them more accessible to a broader range of users.

As the open source community continues to grow and evolve, it is clear that it will play a leading role in the future of LLMs. The Prometheus Effect is just the beginning. The open source community is about to unleash a wave of innovation that will transform the world.

Disclaimer: The views and opinions on this site are solely of the authors, they do not reflect nor represent the views of their employers.