AMD has introduced LogsLop, a summarization tool designed to make large, repetitive log files more manageable for human readers and artificial agents. The tool addresses the challenge of sifting through lengthy log files that often contain redundant information, which can be overwhelming and difficult to process, even for advanced language models. By removing duplicates and focusing on key messages, LogsLop streamlines the diagnostic process for system issues. The tool is available as a PyPI package and can be installed with 'pip install logslop'.
LogsLop works by normalizing numbers in log messages to common placeholders, tokenizing text, and using Jaccard similarity to identify and eliminate redundant lines. The tool is particularly useful in scenarios like multi-process training, where repeated error messages can obscure critical issues. For example, in a PyTorch training scenario with eight processes, LogsLop reduced an output of 9393 lines to just 34 representative messages, making it easier to locate the root cause of errors. The tool also allows users to customize its behavior, such as merging day and month abbreviations into shared tokens, through direct code modifications or agent-assisted coding.
The development of LogsLop was supported by AI tools, which helped with coding syntax and testing, while developers focused on defining desired behaviors and ensuring scalability. The tool is hosted in the amd/logslop repository on GitHub, with demo scripts and sample logs available for testing. Users can experiment with adapting the tool to suit their specific needs, such as modifying string normalization or implementing alternative similarity metrics.
Source: amd
