Shell Programming and Scripting

Hi Guys,
I have a backup program that creates incremental backups and generates a MD5 hash at the same time.
Each server backup has its own sub folder.

Each backup file has a corresponding .md5 file containing a hash and a file name like this.

3410efed13b087322de8036145230a55 *COMPANYSRV_F_VOL-b008-i052.spi

I copy these files off site periodically to a linux based NAS Drive.
The version of linux is a bit cut down but I have managed to find and install a package manager and I now have md5deep working.

My question is how can I script md5deep to validate all files in the sub folders and output any errors. Or just output everything and I can search the text for errors.
I'm surprised I can't find this as it would seem exactly what it is meant to do.

Thanks in advance
David

Update
=========
I suppose I should say what I have tried so far.
The obvious command to start with is this with the md5 and backup files in the current dir.

Code:

md5deep -m  COMPANYSRV_F_VOL-b008-i052.md5 *

I assume the * on the end is if there are multiple hashes contained within the md5. In my case there is only one hash contained in each md5.

This lists the one file if the hash is correct.
I purposely corrupted the file and tried again and there was no output.

What I really need is a success message or a list of corrupted files from the whole directory tree.

Update 2
=========
I think the command I need is going to look something like this.

Code:

find . -name "*.md5" -exec cat {} \; | md5deep -x *.md5

That is....
Find all files in the current directory (find . -name)
With a name that ends in .md5 ("*.md5")
Pipe the contents of the files to md5deep (-exec cat {} \; | md5deep)
Show the file names that don't have a matching hash. (-x *.md5)

However this doesn't seem to read the standard input and just does a md5deep -x *.md5.

If I have three files with correct matching hash files it returns two.
I'd expect to see none when the hashes match.

Did you solve this issue?

Hi Neo,
No I still have not solved this problem.
Do you have any ideas?
Thanks
David

What is your problem? Where are you stuck? I guess the trailing asterisk in your first command might be too many.



Please post the contents and structure of the .md5 files, as well as the directory structures that you use.

Hi RudiC,
I can't seem to test a single folder at the moment so we don't need to worry about sub folders yet.

The trailing * is to select the files contained within the MD5. There is only one file in each MD5 so an * is going to cover it. I get an error if I leave it out.

My MD5's files are a one to one match with the files they are a hash of.

To test I have created a folder with 3 backup files and three matching md5 files
For example one of the backup files is named COMPANYSRV_F_VOL-b008-i066.spi
The contents of the matching MD5 named COMPANYSRV_F_VOL-b008-i066.md5 is
ea068f65d93cb8cccb3a44b642192758 *COMPANYSRV_F_VOL-b008-i066.spi

Couldn't be more straightforward so far.
I want to list files where the hash no longer matches or simply return success or no errors found.

I want to run a command that will test each file against it's hash.
My backups are incremental and range in size from a few 10's ok KB to Hundreds of GB. My Test files are under 20Mb.

My tests above returned the names of two files where I would expect none.
I don't expect I just happen to pick two corrupted files for my test.
It's a long time since I piped anything to the cat so I may not have constructed that command correctly which is why I brought my question to a scripting forum.

Previously I have run this from a windows machine but it copies a TB of data across my LAN and that takes too long.

Thanks for your assistance
David

Hello Again,
After more google research I found that the MD5 files created on my windows system had one extra character apart from the * in front of the file name, that is a Carriage Return.

I also don't think I need the functionality of md5deep as "find" will search sub directories anyway.

I upgraded md5sum on my NAS device as I has some old unknown version.
Now if I use the following command

Code:

find . -name "*.md5" -exec cat {} \; | dos2unix | md5sum -c

I have included a pipe through dos2unix that will convert the windows line termination to unix line termination.

I modified one of my MD5 files to cause a failure.

The Result

Code:

COMPANYSRV_F_VOL-b008-i068.spi: OK
COMPANYSRV_F_VOL-b008-i067.spi: FAILED
COMPANYSRV_F_VOL-b008-i066.spi: OK
md5sum: WARNING: 1 computed checksum did NOT match

This is exactly what I need. I should now be able to schedule this command and send myself the output in an email.

Hope this helps someone else.
David

Thanks for sharing.