Fastdup

fastdup is a Python tool for finding problems in large image and video datasets before they reach a training run. Three lines of code point it at a folder of images, and it reports broken or corrupted files, duplicate and near duplicate pairs, outliers, images that are too dark, too bright or blurry, and clusters of visually similar content. Results come back as pandas data frames and visual galleries, so findings can be filtered, reviewed and acted on. It is built for computer vision teams dealing with datasets too large to inspect by eye, and it is designed to run on ordinary CPU hardware rather than requiring a GPU. Installation is a single pip command, with support for macOS, Linux and Windows under WSL2. The project comes from Visual Layer, founded by the authors of XGBoost, Apache TVM and Turi Create, and the same analysis underpins the company's commercial platform, which adds dataset visualization, semantic and visual search, smart clustering, quality analysis, tagging across clusters, images and objects, an API and a self hosting option. The library itself is free. Visual Layer publishes no price list and no pricing page for the hosted platform, so commercial deployments are quote based and require contacting the company. One licensing detail matters before adopting it. Although the project is described as open source, the repository license is Creative Commons Attribution NonCommercial NoDerivatives 4.0, which does not permit commercial use or modified redistribution. Teams planning to embed it in a product should settle terms with the vendor first.