FastDup is a tool to find identical files within directory trees, with a focus on the ability to scan very large trees and files in reasonable amounts of time. The use of clever techniques and optimizations to avoid comparisons allows fastdup to significantly outperform many existing solutions, and has a benefit of being entirely I/O bound (insignificant CPU usage). It is released under a modified BSD license, allowing all use, redistribution, and modification for non-commercial purposes.
License: BSD License (revised)
Changes:
This release includes vast improvements in memory usage, especially when scanning large numbers of files, and more informative and active output. Numerous internal changes were also made with the aim of cleanliness and future expandability.
More...