San Francisco, California · 11 June 2020 — OpenAI announced the release of GPT-3 (Generative Pre-trained Transformer 3) on 11 June 2020, opening API access to a 175-billion-parameter language model that represented a more than 100-fold scale increase over its predecessor, GPT-2. The model demonstrated an ability to perform a wide range of natural language tasks — including translation, summarisation, question-answering, and code generation — from only a few examples in the prompt, without requiring task-specific fine-tuning.
The release followed a 75-page research paper published on arXiv on 28 May 2020 by a team of 31 engineers and researchers at OpenAI, led by Tom Brown and Dario Amodei. The paper, titled "Language Models are Few-Shot Learners," argued that scaling up language models dramatically improved performance on a broad set of NLP benchmarks, particularly under few-shot conditions where the model is given only a handful of examples at inference time.
Architecture and Training
GPT-3 used the same decoder-only transformer architecture as its predecessors, relying on the attention mechanism introduced in the 2017 paper "Attention Is All You Need" by Vaswani et al. The model's improved performance was attributed primarily to its vastly increased scale rather than architectural innovation. GPT-3 had a context window of 2,048 tokens and used 16-bit precision parameters, requiring approximately 350 GB of storage.
The training data was composed of several large text corpora:
| Dataset | Tokens | Weight in Training Mix |
|---|---|---|
| Common Crawl (filtered) | 410 billion | 60% |
| WebText2 | 19 billion | 22% |
| Books1 | 12 billion | 8% |
| Books2 | 55 billion | 8% |
| Wikipedia | 3 billion | 3% |
The Common Crawl portion was filtered using fuzzy deduplication via Apache Spark's MinHashLSH to remove near-duplicate documents and improve data quality. The total training corpus encompassed hundreds of billions of words drawn from web pages, books, and encyclopedic sources.
Lambdalabs estimated that training GPT-3 on a single GPU would have cost approximately $4.6 million and taken 355 years, though the actual training used large clusters of GPUs in parallel, reducing wall-clock time to weeks.
Few-Shot Learning: A Paradigm Shift
The most significant finding of the GPT-3 paper was the model's few-shot learning capability. Traditional NLP models required fine-tuning on task-specific datasets — a process involving gradient updates to the model's weights. GPT-3, by contrast, could perform many tasks with only a text prompt containing a few examples, or in some cases zero examples (zero-shot).
OpenAI defined three inference regimes:
- Zero-shot: The model is given a natural language instruction with no examples.
- One-shot: The model is given a single example.
- Few-shot: The model is given a small number of examples (typically 10–100).
In all three regimes, no gradient updates are performed — the model's weights remain frozen. This represented a significant departure from the fine-tuning paradigm that had dominated NLP since the introduction of BERT in 2018.
Benchmark Performance
GPT-3 was evaluated across more than 50 NLP benchmarks. Key results included:
- Translation: GPT-3 achieved competitive BLEU scores on several language pairs in few-shot settings, despite never being explicitly trained for machine translation. Performance was strongest for high-resource languages.
- Question-answering: On the TriviaQA benchmark, GPT-3 achieved 64.3% accuracy in the few-shot setting, approaching human-level performance.
- Arithmetic: The model could perform 2-3 digit addition and subtraction with high accuracy, though performance degraded on more complex calculations.
- News article generation: In a human evaluation study, 80 participants were asked to judge whether ~200-word articles were written by humans or GPT-3. Participants correctly identified AI-generated text only 52% of the time — barely above random chance.
- Code generation: GPT-3 demonstrated ability to generate functional code in Python, CSS, JSX, and other programming languages from natural language descriptions.
API Release and Eourcing
On 11 June 2020, OpenAI opened a waitlist for API access, describing it as a general-purpose "text in, text out" interface. The API was initially available by invitation only, with OpenAI stating the staged rollout was intended to help "explore the strengths and limits" of the technology.
On 22 September 2020, Microsoft announced it had acquired an exclusive license to GPT-3's underlying model. Under the agreement, OpenAI continued to operate a public API allowing users to send text and receive model output, but only Microsoft gained access to GPT-3's source code for integration into its own products and services. The deal followed Microsoft's $1 billion investment in OpenAI in 2019.
Risks and Limitations
The GPT-3 paper devoted significant attention to potential harmful applications, including:
- Misinformation generation: The model's ability to produce coherent, persuasive text at scale raised concerns about automated disinformation.
- Phishing and social engineering: GPT-3 could generate convincing phishing emails or social engineering pretexts.
- Academic integrity: The model's essay-writing capability posed challenges for educational institutions assessing student work.
- Toxic language: GPT-3 occasionally produced toxic or biased content, reflecting patterns in its training data. A University of Washington study found toxicity levels comparable to GPT-2, though OpenAI implemented filtering strategies to reduce harmful output.
- Environmental impact: Training large models consumed substantial energy, a concern later highlighted by AI ethics researchers Timnit Gebru and Emily M. Bender in a 2021 paper.
Impact and Legacy
GPT-3's release catalysed a wave of innovation in the AI ecosystem. Within months, developers built applications ranging from automated customer service chatbots to code completion tools and creative writing assistants. The API's general-purpose design meant that a single model could power dozens of different applications, reducing the barrier to entry for NLP-powered products.
The model also intensified debates about the scaling hypothesis in AI — the proposition that simply increasing model size, data, and compute would continue to yield qualitative improvements in capability. GPT-3's few-shot performance provided empirical support for this hypothesis, influencing the direction of AI research investment for years to come.
By November 2021, OpenAI removed the waitlist for API access, making GPT-3 broadly available. In January 2022, OpenAI introduced InstructGPT, a fine-tuned version that better followed user instructions and produced fewer factual errors, which became the default model on the API.
Sources
- Brown, T. et al., "Language Models are Few-Shot Learners," arXiv:2005.14165, 28 May 2020
- OpenAI, "OpenAI API," , 11 June 2020
- Wikipedia, "GPT-3,"
- VentureBeat, "OpenAI debuts gigantic GPT-3 language model with 175 billion parameters," 11 June 2020
- NVIDIA Developer Blog, "OpenAI Presents GPT-3, a 175 Billion Parameters Language Model," 2020
FIRAT Editorial Board
Institutional Research Desk · Foresight Institute of Research and Translation
The collective editorial and research translation board of FIRAT, synthesising peer-reviewed evidence, policy briefs, and division milestones across our seven foundational research pillars.
