Skip to content
OpenAI News·· 2019-04-23AI score6

OpenAI Proposes Sparse Transformer: Modeling Ultra-Long Sequences with Sparse Attention

Generative modeling with sparse transformers

AI briefing

OpenAI developed the Sparse Transformer, a deep neural network that set records on the task of predicting the next item in sequences of text, images, or sound. By improving the algorithm of the attention mechanism, it can extract patterns from sequences 30 times longer than before.

Open original openai.com