#Jordan 1 attention attention series

However, for now, they typically are are each a separate linear transformation of the same input matrix. The similarity of a given query to the keys determines how much information from each value to retrieve for that particular queryįor now, don't worry too much about where Q, K, and V come from, I'll talk about that more extensively in part 2 of the post.

#Jordan 1 attention attention series

We then take those queries and match them against a series of keys that describe values that we want to know something about. The general idea is that we submit a query for each token in the input sequence. These inputs can all be described in terms of a 'information retrieval' system of sorts. There are three main inputs to the attention mechanism: queries (Q), keys (K), and values (V). Visual depiction of the attention operation, where n is the number of tokens in the input sequence and d is the dimensionality of those tokens. Note: here I ignore the softmax and sqrt(d k) scaling term in the attention equation for visual simplicity, but I'll discuss those later. For example, if we were dealing with a sequence of words the dimensionality d would be the dimension of the word embeddings for the n tokens in the sequence. There's a lot going on here, but I'll break things down step by step.īelow is the above formula shown with the shape of the tensor inputs and outputs, where n is the number of tokens in the input sequence and d is the dimensionality of those tokens. So after all that, what's the equation for the attention operation? Attention Equation Formula for the attention operation. For instance, it can also be used with imaging data, where typically an image is represented as a sequence of patches that are then used as tokens in a sequence (more details on vision transformers in my post here). Attention applies to any type of data that can be formatted as a sequence. For example, if we take the sentence the cat jumped over the lazy dog, we would expect the network to learn to associate cat with jumped and lazy with dog (for more on attention for natural language processing (NLP) tasks see my post here).

The key concept behind self attention is that it allows the network to learn how best to route information between pieces of a an input sequence (known as tokens). In part 2 next week I'll cover multi-head attention and how it is put to use in transformer architectures. In part 1 of this multipart post, I'll cover the attention mechanism itself, and build an intuition for the why and how attention works.

Multi-head attention allows for the neural network to control the mixing of information between pieces of an input sequence, leading to the creation of richer representations, which in turn allows for increased performance on machine learning tasks. In general, the feature responsible for this uptake is the multi-head attention mechanism. Transformers have become nearly ubiquitous for a variety of machine learning tasks.