Shell Programming and Scripting

@Don Cragun
This is why I might cut the first 64 digits, because it looks a bit too nasty with the sha256sum, imagine sha512sum.

Code:

f888084c9ee5af98bb46df1c059b8cc2bcc4d9c43691e139f5bce44c64926e3a  ./drum/soundz4/GLASS CAVES – Out Of Control.mp3

as always learning

@Aia for heavens sake, sed coming up......yeah, I will train this....thanks, I will report later.

harsh.....

---------- Post updated 09-07-15 at 06:01 PM ---------- Previous update was 09-06-15 at 10:35 PM ----------

@Aia The quadcore works a lot with the xargs -0 version right now. I just started to read the first chapter of 63 for sed, so sorry, that this is one of the commands I hoped to avoid until now. First try results in one single file, the destination.mp3 in this case. hm....

@Don Cragun
one directory, 252 subdirectories, nearly 5000 tunes, different versions, kbps, length, put them in one folder.
My script from last year, well lets call it a snippet of a script, was hopping from one directory as the outset to move (cp) to destination. Then I took the destination to hop to the third new directory and so on. Surely not a hundred times, but it got a result of 30GB for snippets of tunes, one second of hihat, 3 seconds of openhihat, all together 26k files, or items. Therefore I used this construction

Code:

find /home/origin -type f -exec sha256sum {} \; | sort -k 64 >  /tmp/runner;txt | cp -nr /home/origin /home/destination && cat  -n /tmp/runner.txt | cut -d " " -f1; exit 0;

and guess what, it worked.

And I am aware, that the real 19 inch server rack is not sorted like this, they would put any file listed in alphabetical order, for each letter one drive, may even for different formats (.jpeg or .png) different mirrors. It is just to have it in one directory and later on on a pluggable drive.

@Aia It stopped after 575 songs, unable to call stat for "-" (no underliner), could not find directory. Running now version two, with sed. That goes well as root, I'll try it as well as user. Resulting in 4429 tunes, 22 GB. Orignally it was 22.9 GB. Now I could work with diff or midnight commander.

I am very glad that the command line:

Code:

find /home/origin -type f -exec sha256sum {} \; | sort -k 64 >  /tmp/runner;txt | cp -nr /home/origin /home/destination && cat  -n /tmp/runner.txt | cut -d " " -f1; exit 0;

is working for you. Assuming that the /tmp/runner;txt in the above is a typo and that what you really used was /tmp/runner.txt AND assuming that the cp completes successfully; are you aware that the above is equivalent to the three completely independent command lines:

Code:

cp -nr /home/origin /home/destination
find /home/origin -type f -exec sha256sum {} \; | sort -k 64 | tee /tmp/runner.txt | cut -d " " -f1
exit 0

which means, as I said before, that all of the stuff that you're doing with the checksums has absolutely no affect on the set of files copied to your destination directory.

Note also, that unless some of your song titles contain more than 62 "words", the second command line is equivalent to the slightly simpler:

Code:

find /home/origin -type f -exec sha256sum {} \; | sort | tee /tmp/runner.txt | cut -d " " -f1

which sorts with the checksum as the primary sort key and the song title as the secondary sort key. But, if some titles do have more than 62 "words" and you use sort -k 64, your primary sort key is the 63rd "word" in your song titles, the secondary sort key is the checksums, and the tertiary sort key is the rest of the song title. I do not understand why the 63rd word in your song titles should be so significant???

Presumably the reason why the size of the destination directory is smaller than the size of the source file hierarchy is because:

@Don Cragun: You are right, not only the typing error of " ; " but the length of the field for the checksum. Browsing the result there are no hidden files like

Code:

 ./mediaartlocal

, that tiny thumbnails, and probably some links. So thanks a lot to
@Aia and you,
even pushing me into the streamline editor. I will probe it some more times with diferent values. Really thanks a lot for your attention.