Generative AI, responsible for creating plausible text, images, and code that mimics human ability, owes its existence to the Transformer model. This scientific development, pioneered by a group of eight Google AI researchers in June 2017, was outlined in an 11-page research paper that marked the beginning of the generative AI era. Previously, researchers built specialized models for tasks like summarizing, translating, and searching; the Transformer unified these functions into a single structure capable of performing a wide variety of tasks. It radically sped up and augmented how computers understand language, allowing LLMs to parse and write fluently by processing entire sequences at once and analyzing all their parts.
A key concept of the Transformer architecture is "self-attention," which enables LLMs to understand relationships between words. This allows the software to capture context and patterns better, leading to more accurate text generation and faster training of LLMs. The Transformer's power extends beyond language, as its inventors discovered it could recognize and predict repeating patterns in pixels, computer code, music, and even DNA in proteins, powering tools like Dall-E, Midjourney, Stable Diffusion, and GitHub CoPilot.
Despite its advancements, LLMs have limitations, notably the phenomenon of "hallucination," where they generate fabricated information such as made-up numbers, names, dates, quotes, or even non-existent web links and articles. This inherent predictive nature means LLMs are pattern-spotting engines that guess the next best option in a sequence, rather than search engines looking up facts. A high-profile incident in New York saw a lawyer use ChatGPT to create a brief riddled with made-up judicial opinions and legal citations. Researchers are working on limiting hallucinations through "grounding," which involves cross-checking LLM outputs against web search results and providing citations, and through techniques like "beam search" and "reinforcement learning by human feedback (RLHF)," but understanding and predicting these errors remains a significant challenge. Goldman Sachs estimates generative AI could impact the equivalent of 300 million full-time workers across major economies.
