Hugging Face released a guide on September 9, 2026, showing how to create a dataset of datasets. The guide collects metadata from up to 1,000 repositories.
It uses Python and the Hugging Face API to gather information about datasets, including their names, downloads and tags. The script saves the original information in a RAWDATA folder and processes it into a structured format.
The guide explains how to use Python's built-in tools to interact with the Hugging Face Hub API. It requests metadata from up to 1,000 repositories sorted by downloads over the last 30 days. The script saves the raw response in RAWDATA/response.json and processes it into structured fields like languages, task categories and formats.
"We're collecting the public metadata of these repositories," said tegridydev, the author of the guide. "This helps users understand the data and its context without accessing the actual content." The guide includes a complete script that users can run without changing anything, making it accessible for beginners.
The guide also highlights the importance of keeping raw data for verification. It explains that the script saves the original response in RAWDATA/response.json, allowing users to check what Hugging Face returned if something looks wrong in the processed dataset. The script also records the request, collection times and script version in collection.json.
The guide does not say how the collected data will be used beyond its initial processing. It raises the question of whether the data should be expanded to include more repositories or additional metadata. The script is designed to be run in a terminal, with instructions for different operating systems.
Source: huggingface
