Generative AI notes
- AI imitates human behavior by using machine learning to interact with environment and
executes tasks without explicit directs on what to output.
- Generative AI is a category within AI that creates original content like text , images, music,
audio and videos. Eg Microsoft copilot, chatgpt
- Language model is a probabilistic model of a natural language.
Language models are used to perform NLP task like sentiment or classying natural language
text, summarizing text, comparing multiple text sources for semantic similarity.
- Transformer models are a neural network that learns context and thus meaning by tracking
relationships in sequential data like the words in a sentence.
- These transformer models consist of 2 components
Encoder that generates the meaning/ semantic representation from the sequence of data
Decoder generates the new language sequences
The sequence of text are broken down into tokens (individual words)
The encoder processes these tokens using a technique called attention.
This is to determine the relationship between these tokens.
The output from this encoder is a collection of vectors (multi-valued numeric arrays) referred
to as embiddings.
- The decoder works on a new sequence using text tokens and uses embeddings generated by
the encoder to generate the
- In practice, the specific implementations of the architecture vary – for
example, the Bidirectional Encoder Representations from Transformers
(BERT) model developed by Google to support their search engine uses
only the encoder block, while the Generative Pretrained Transformer (GPT)
model developed by OpenAI uses only the decoder block.
- Attention is a technique used to examine a sequence of text tokens and
try to quantify the strength of the relationships between them.
