AI Research
The reports worth your hour: investment-bank research, VC state-of-AI decks, lab and academic papers, each with an AI Lounge summary and the three numbers to remember.
Training language models to follow instructions with human feedback (InstructGPT, 2022): the RLHF recipe behind ChatGPT
Reinforcement learning from human feedback fine-tunes language models to follow instructions better than larger base models
Research paper
Language Models are Few-Shot Learners (GPT-3, 2020): 175B parameters and the discovery of in-context learning
Large language models demonstrate few-shot learning capabilities through in-context examples without gradient updates
Research paper
Scaling Laws for Neural Language Models (2020): loss falls as a power law in parameters, data and compute
Language model loss scales predictably as a power law with model size, data, and compute
Research paper
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2018)
Bidirectional Transformer pre-training with masked language modeling achieves state-of-the-art results across multiple NLP tasks
Research paper
Attention Is All You Need (2017): the Transformer paper that started the modern AI era
Self-attention mechanisms enable parallel training and superior translation quality while reducing computational costs compared to recurrent and convolutional architectures
Research paper
Deep Residual Learning for Image Recognition (ResNet, 2015): skip connections make 100+-layer networks trainable
Skip connections enable training of very deep networks by allowing layers to learn residual functions instead of direct mappings
Research paper
Adam: A Method for Stochastic Optimization (2014): the default optimizer for nearly every neural network since
Adaptive learning rate optimizer with momentum and bias correction becomes standard for training deep neural networks across domains
Research paper
Generative Adversarial Nets (2014): Goodfellow's generator-vs-discriminator game that created modern generative AI
Generator and discriminator networks trained adversarially learn to produce realistic data without explicit likelihood estimation
Research paper
Efficient Estimation of Word Representations in Vector Space (word2vec, 2013): words become vectors, and king − man + woman = queen
Dense word embeddings capture semantic relationships, enabling analogies like king minus man plus woman equals queen
Research paper
ImageNet Classification with Deep Convolutional Neural Networks (AlexNet, 2012): the result that launched deep learning
Deep CNN trained on GPUs reduces ImageNet error from 26% to 15%, launching modern deep learning era
Research paper
