NLP

GPT-3

Generative Pre-trained Transformer 3 (GPT-3) is an autoregressive language model that uses machine learning to mimic the production of human-like text. The benefit is that the quality of the text is so high that it is difficult to predict if it was written by a human or not.

Project Background

  • Project: GPT-3
  • Author: OpenAI
  • Initial Release: 2020
  • Type: Autoregressive Transformer Language Model
  • Dataset Common Crawl Dataset (composition): 410B tokens and 60% weight in training mix  
  • WebText2 Dataset: 19B tokens and 22% weight in training mix
  • Books1 Dataset: 12B tokens and 8% weight in training mix
  • Books2 Dataset: 55B tokens and 8% weight in training mix
  • Wikipedia: 3B tokens and 3% weight in training mix
  • GitHub: /gpt-3 with 10.8 stars and 4 contributors
  • Twitter: /gpt3_

Applications

  • Use by inexperienced AI researchers to build and explore language modeling systems across functions
  • GPT-3 is used to convert natural language into formal computer code 
  • GPT-3 was used to create an article on the harmlessness of AI, by Andrew Mayne
 
Scroll to Top