Article List

Training foundation models at scale is constrained by data. Whether working with text, code, images, or multimodal inputs, the public datasets are sat...

Embeddings are a numerical representation of text. They are fundamental to the transformer architecture and, thus, all Large Language Models (LLMs). I...

In neural networks, some neurons end up outputting near-zero activations across all inputs. These so-called “dead neurons” degrade model capacity beca...

In the first part of this series, we covered the fundamentals of instruction fine-tuning (IFT). We discussed how training LLMs on prompt-response pair...

Large Language Model (LLM) inference at scale is challenging as it involves transferring massive amounts of model parameters and data and performing c...

Large Language Models (LLMs) can be thought of as knowledge bases. During training, LLMs observe large amounts of text. Through this process, they enc...

Instruction Fine-Tuning (IFT) emerged to address a fundamental gap in Large Language Models (LLMs): aligning next-token prediction with tasks that dem...

Here’s something fun to start with: Open ChatGPT and type, “Use all the data you have about me and roast me. Don’t hold back.”...

In recent years, Large Language Models (LLMs) have mostly improved by scaling. This has primarily involved increasing the size of the LLMs and the dat...

As foundation models scale to billions or even trillions of parameters, they often exhibit training instabilities, particularly vanishing and explodin...

Mixture-of-Experts (MoEs) architectures offer a promising solution by sparsely activating specific parts of the model, reducing the inference overhead...

Retrieval-augmented generation (RAG) is a technique for augmenting the generative capabilities of a large language model (LLM) by integrating it with...