Scribd
Upload
3 views5 pages

Position Encoding

Positional encoding is crucial for Transformers as it provides sequence order information, allowing the model to distinguish between different token arrangements. Without it, self-attention treats inputs as unordered sets, losing the ability to understand order-dependent meanings, which is essential for tasks like language processing and time series analysis. Common types of positional encoding include sinusoidal, learned embeddings, and relative positional encoding, each with unique advantages for modeling sequences.

Uploaded by

yzyjack18
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
3 views5 pages

Position Encoding

Positional encoding is crucial for Transformers as it provides sequence order information, allowing the model to distinguish between different token arrangements. Without it, self-attention treats inputs as unordered sets, losing the ability to understand order-dependent meanings, which is essential for tasks like language processing and time series analysis. Common types of positional encoding include sinusoidal, learned embeddings, and relative positional encoding, each with unique advantages for modeling sequences.

Uploaded by

yzyjack18
Copyright
© All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd

Effect of Positional Encoding (PE)

Transformers process tokens in parallel, so unlike RNNs/CNNs they have no


inherent notion of order. Positional encoding injects sequence order
information so the model can reason about who comes before whom.

What happens without positional encoding?

 Permutation invariance: Self-attention treats inputs as a set, not a


sequence.
 Sentences like

“the dog bit the man” vs “the man bit the dog”
become indistinguishable if tokens are the same but reordered.

 The model can learn content similarity but not order-dependent


meaning.

 Tasks that rely on sequence order (language, time series, trajectories)


break.

What positional encoding adds


Positional encoding adds a position-dependent vector to each token
embedding:
x t =et + pt

This enables the model to:

 Distinguish absolute position (start vs end of sequence)

 Infer relative position (token A is before token B)

 Learn directionality and ordering patterns

Common types of positional encoding

1. Sinusoidal (fixed)

 Uses sine/cosine waves at different frequencies

 Deterministic, no extra parameters

 Enables extrapolation to longer sequences

 Original Transformer choice

2. Learned positional embeddings

 Trainable vectors per position


 Often better empirical performance

 Limited to max training length

3. Relative positional encoding

 Encodes distance between tokens instead of absolute index

 Improves generalization for long contexts

 Common in modern LLMs and robotics/planning models

Intuition (interview-friendly)

Positional encoding breaks the permutation symmetry of self-attention,


allowing the Transformer to model ordered structure instead of treating
tokens as an unordered set.

Why this matters in practice

 Language modeling: grammar, syntax, causality

 Time series: trends, delays, temporal dependencies

 Trajectory & behavior modeling (e.g., robotics, autonomy): motion


history, intent prediction

Without PE → attention answers “what is similar?”


With PE → attention answers “what is similar and where/when?”

One-liner for interviews

Self-attention alone is permutation-invariant; positional encoding injects


order information so Transformers can model sequences instead of bags of
tokens.

If you want, I can also:

 Compare absolute vs relative PE in attention equations

 Show why sin/cos lets the model infer relative distance

 Connect this directly to trajectory prediction or behavior planning


(very relevant for Nuro)

You might also like