UNIX for Dummies Questions & Answers

have you tried with xargs ?

Code:

find /somefolder -type f -size -7M | xargs -iSourceFile cp -v SourceFile /path/to/some_other_folder

One way to test the theory that there are duplicate filenames.

Search for say the first 5 duplicate filenames then re-search the folder tree to find out where they are.

Code:

find /somefolder -type f -size -7M -exec basename {} \; | sort | \
uniq -d | head -5 | while read filename
do
        find /somefolder -type f -name "${filename}" -exec ls -lad {} \;
done

zedex -> that returned 173 files, so not sure exactly what happened

methyl -> that returned only 8 duplicate file names. I searched for duplicatfile.that.it.found.csv and indeed it only found 2 examples, which is what your script thought there would be.

Now I'm really baffled

@unclecameron
My script limited the number of times we searched for duplicates (or it would have run for hours and hours).

This seems to be a design error. You are copying files from multiple directory trees and some of those filenames are not unique. This does not mean that the data in the duplicate filenames is identical (the actual data content is a matter for local knowledge) and there is a real danger of arbitarily overwriting one file with another file which happens to have the same name.

We seem to have reached stalemate. The "rsync" option in an earlier post seems sensible, but you do not wish to preserve the directory tree for some reason unknown.
I don't recall a post where you had a reason for this exercise.

In similar circumstances I have created a prefix for every copied filename composed from the source directory name. Taking into account good local knowledge of the filenames this is not impossible. You could for example substitute "__" for every "/" in a full hirearchial filename to create a unique filename in your copy directory.

The reason is I am having trouble with a process I need to run which iterates through around 88K files, running a process on each of them, and speed is the key, and it turns out in testing I need to run an optimum 6 threads at the same time on a split up sort of the files by size, and also that if the directory structure is preserved within those sorts it adds around 30% to the processing time. So I start the laborious process of cp'ing them as files with no path into /somedir and then running md5sum's against them to correlate to their original file/path. It's really a pain, but I can't modify the binary app I'm running against them, hence the pain, and I need to overall processing time to be under a specific time, so we've added mem, cpu's, servers, more RAID volumes, and mounted /tmp on ssd's to make it happen, all of which helped. This is a dynamic set of files which changes so we have to re-sync/re-sort every few days.

An alternative approach.
A sort by file size followed by a deal out to 6 lists should take about 10-15 mins as long as we use "du" not "ls".
I don't have your Operating System so do check the commands.

Code:

counter=0
du -ak /somefolder|sort -n -r|while read size filename
do
        counter=`expr ${counter} + 1`
        if [ ${counter} -gt 6 ]
        then
                counter=1
        fi
        if [ ! -f "${filename}" ]
        then
                continue
        fi
        echo "${filename}" >> "/tmp/selection_${counter}"
done
#
wc -l /tmp/selection_*