For avoiding duplicates, you might want to pull just the file names into a list, and generate MD5 sums of all the files in that list. If two files have the same MD5, they are identical (with a probability which is close enough to certainty for most practical purposes). Remove duplicate MD5s, then copy the remaining files. (If the file format makes it unlikely that two different files will have exactly the same size, that might be good enough, and a lot quicker than MD5 to calculate.)