What changed

Hugging Face has announced the release of a neural network that generates GTA V gameplay frames in real-time without a game engine. The model, called Summaryformer, is a 630M-parameter transformer trained from scratch. It generates one frame at a time based on a starting image and player input, with no game engine underneath. The weights and complete inference code are open under Apache-2.0, and the model can be played in the browser.

What this means in practice

The model uses a unique architecture called Summaryformer to manage its context. It stores each frame twice, at two sizes, and the model writes both itself. A frame is stored as a record with raw tokens and summary tokens. The summary tokens are a compact description of the frame and are used by later frames. The model uses a single key/value buffer to store the context, with different regions for raw and summary frames. The buffer holds 6,528 keys when using the trained window. The model is inference-only and cannot be trained.

Limitations and availability

The model is limited by its memory management. The architecture gives the model a window of 24 latent frames at a low resolution. The model is not available for training, and the release is inference-only. The model was trained on 100 million images, 4 million videos, and 200,000 very long videos. The model is available on Hugging Face Space under the Apache-2.0 license.

Source facts for Hugging Face releases 630M-parameter GTA V world model: Play it in your browser · Hugging Face Space · Weights and code · Architecture note This is not the game. You give it one start image and hold WASD, and it generates the game one frame at a time while you play. There is no game engine underneath, the whole thing is a neural network.
Original AINewsTrend explainer based on the cited source. This graphic summarizes source statements; it is not an independent benchmark.

Sources and further reading

  1. huggingface: GTA in a neural network: an open world model, and the Summaryformer