Positional Encoding in Transformer
With the advancement of large language models (LLMs), the significance of the context length they can handle is increasingly apparent. Let’s take a look at the evolution of positional encoding over the years to enhance the context processing capability of LLMs. Vanilla Positional Encoding Why does Transformer need positional encoding? Actually, Transformer contains no recurrence and no convolution. To help the model to ultilize the order of the sequence, Vanilla Transformer (vaswani et al., 20171) introduced the concept of positional encoding and adopted a simple yet effective approach, using sine and cosine functions to generate positional encodings. This method allows the model to effectively capture the positional information of words in the sequence without adding additional parameters. ...