What changed
Hugging Face has announced the release of a neural network that generates GTA V gameplay frames in real-time without a game engine. The model, called Summaryformer, is a 630M-parameter transformer trained from scratch. It generates one frame at a time based on a starting image and player input, with no game engine underneath. The weights and complete inference code are open under Apache-2.0, and the model can be played in the browser.
What this means in practice
The model uses a unique architecture called Summaryformer to manage its context. It stores each frame twice, at two sizes, and the model writes both itself. A frame is stored as a record with raw tokens and summary tokens. The summary tokens are a compact description of the frame and are used by later frames. The model uses a single key/value buffer to store the context, with different regions for raw and summary frames. The buffer holds 6,528 keys when using the trained window. The model is inference-only and cannot be trained.
Limitations and availability
The model is limited by its memory management. The architecture gives the model a window of 24 latent frames at a low resolution. The model is not available for training, and the release is inference-only. The model was trained on 100 million images, 4 million videos, and 200,000 very long videos. The model is available on Hugging Face Space under the Apache-2.0 license.
