Modern large language models (LLMs) rely on two primary components: perceptrons and attention mechanisms. Perceptrons handle most tasks, while attention manages sequence information. However, perceptrons must learn a vast array of complex tasks, which can be challenging. Engrams are introduced to assist with these tasks by allowing perceptrons to focus more on abstract linguistic concepts. Engrams are described as a specific type of memory, though they are actually embeddings with additional steps.
Engrams function by embedding a hashed sequence of multiple tokens rather than a single token. For example, the word 'Demonize' might be split into tokens 'De', 'mon', and 'ize', each with its own embedding. This means the concept of 'Demonize' is spread across unrelated embeddings, making it difficult for models to associate these tokens with the concept. Engrams address this by embedding a sequence of tokens, such as a bigram or trigram, which helps the model better understand the context.
The source explains that engrams are essentially embeddings but with a larger table size. For instance, a typical embedding table might have 8,000 rows for a vocabulary of 8,000 tokens, but an engram table for bigrams would require 64 million rows. This is impractical, so models use hashing and multiple tables to manage collisions. Additionally, engrams use confidence gates to handle rare sequences, ensuring that only common sequences are prioritized. This approach allows models to focus on typical sequences while still relying on attention for rare or out-of-distribution cases.
Source: huggingface
