Shell Programming and Scripting

On my website I host a lot of files, and when people view the site, currently each time the page loads, I have PHP generating the md5 sums for the files right then and there. It was fine when my site was small, but now that's obviously very inefficient. Now I'd like to start generating MD5 sums when a user uploads a file, and storing it in a text file along with other information to be read from there instead.

I know how to incorporate this into my PHP script, however I still have the problem of the hundreds of files that have already been uploaded, that won't have MD5 information stored in their .desc file.

Here's my site's structure:

files/HTC Eris/Rom maker/rom.zip
files/HTC Eris/Rom maker/rom.zip.desc
files/HTC Eris/Rom maker/rom.zip.dlcnt
files/HTC Incredible/Rom maker2/rom2.zip
files/HTC Incredible/Rom maker2/rom2.zip.desc
files/HTC Incredible/Rom maker2/rom2.zip.dlcnt

...and so forth.

The .desc file already stores certain information such as username, IP, and a file description. I want to append the md5 information to the end of the .desc file. The .dlcnt file is irrelevant, it stores amount of times downloaded.

So basically I'm trying to write a bash script to do this. Trouble is... I know squat about bash scripting!

All I've been able to come up with so far is this:

Code:

#!/bin/bash

find -name *.zip >> files.lst
find -name *.apk >> files.lst
find -name *.exe >> files.lst
find -name *.png >> files.lst
find -name *.tar.gz >> files.lst
find -name *.img >> files.lst
find -name *.xml >> files.lst

for i in $( cat files.lst );
do
		md5sum $i >> $i.desc;
done

I'm sure this is a really horribly inefficient way to do this, but it's all I know.

First problem is that a lot of the directories and files have spaces in them. That screws up the script right there. I don't know how to change the names to have a \ in them.

Second problem is I'm not exactly sure how to output the md5sum to the .desc file. Obviously what I have written up there doesn't work, it's not proper use of the variable... but I'm not sure what to do?

Can anyone give me a hand?

find and xargs are good friends; one finds files and prints them, the other converts lines into command arguments. They're such good friends they even have a mutually-agreed-on format for communicating names into xargs without messing up on spaces: nulls. -print0 will tell find to print using nulls as seperators, and --null will tell xargs to read them likewise.

You can specify more than one extension with find's -or option, as well.

And, wherever you have 'for LINE in `cat file` you can do 'while read LINE ; do something ; done < file', much more efficient. It can also read from a pipe.

So you can cram this all into one efficient statement:

Code:

find /path  '(' -name '*.type' -or -name '*.type2' -or -name '*.type3' ')' -print0 |
        xargs --null md5sum |
        while read CHECKSUM FILENAME
        do
                echo "${CHECKSUM} ${FILENAME}" >> "${FILENAME}.desc"
        done

find seems to need brackets around the things in -or to use it both, otherwise it just ends up looking for the last one.

EDIT: Ok, I see you modified your original post, let me give that a shot.

Thanks for the reply.

Ok, I did:

Code:

find Files/ -name '*.zip' -or '*.apk' -or '*.exe' -print0 |
        xargs --null md5sum

Which returns:

Code:

grdlock@mainbox:~/Temp$ ./md5script2
find: paths must precede expression: *.apk
Usage: find [-H] [-L] [-P] [-Olevel] [-D help|tree|search|stat|rates|opt|exec] [path...] [expression]
d41d8cd98f00b204e9800998ecf8427e  -

The syntax of the find command isn't working. So just for testing purposes I took out the -or switches and tried it just for .zip files, and it worked, it outputted the md5's of the .zip files, however, I'm not sure how I would write those md5's to the .desc files using your example, instead of just outputting them?



---------- Post updated at 10:03 PM ---------- Previous update was at 10:02 PM ----------




Ok, wow, that's working.

The only problem is for some reason the -or switch isn't working. It still says "find: paths must precede expression: *.apk" for anything with the -or option.

That's really no big deal though, you're been more than helpful already! It won't take too much extra time to make a script for each file extension, or just run it multiple times or something.

Either way I really appreciate the help. Also, thanks for the explanation of how everything works as well!

I made a mistake. -or -name '*.ext', not just -or '*.ext' I should have copy-pasted it from my terminal instead of rewriting it...

Edited my original post to fix the issue.

Figured it out... it was:

find . -name '*.zip' -or -name '*.apk'

Anyways, I really appreciate your help with this.



---------- Post updated at 10:49 PM ---------- Previous update was at 10:25 PM ----------




Oh, just thought I'd include one more thing incase anyone ever reads this thread...

I had to also add this to the find statement:

find files/ -name '*.zip' -print0 -or -name '*.apk' -print0 -or -name '*.exe' -print0

I had to stick the print0 after each filename, otherwise it was only returning the last file type in the output.