Visit Our UNIX and Linux User Community

Running md5sum on a list of files

Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Running md5sum on a list of files
# 1  
Old 04-03-2009
Running md5sum on a list of files


I would like to run md5sum on a list of files saved in a text file, and save the result in another file. (ie. md5sum `cat list.txt` > md5list.txt)

I have tried several things, but I am always confronted to the same problem: some of the filenames have spaces.

I have run sed on the list to escape all the spaces, but shell still considers them as parameter separators.

How would you do that ?

Thanks !
# 2  
Old 04-03-2009
I was reading other posts while waiting for a reply, and found the solution in one of them: xargs.

The right command is:

xargs md5sum < list.txt >md5list.txt
# 3  
Old 04-03-2009
Originally Posted by SDelroen
The right command is:

xargs md5sum < list.txt >md5list.txt

...even that will break for filenames with spaces, use a while loop instead, a bit slower but safer...

while IFS= read line 
    md5sum "$line" 
 done < list.txt > md5list.txt

Don't forget "quoting".
# 4  
Old 04-05-2009
True, it breaks if you have spaces.

The quoting also breaks if you have double quotes in your filenames, which was also my case, that's why all special chars were escaped in my list.txt file.

By the way, it might be useful to someone to know how to do that, cause it took me quite a long time to find out:

find here -type f | sed "s:[\ \',\"]:\\\&:g" > list.txt

If your filenames include other special chars, you can add them in the left part of the sed expression.
# 5  
Old 04-06-2009
Actually I wasn't thinking of all the unhealthy characters in the filenames, your post mentioned space, the most common unwanted character, so I advised accordingly. Think of filenames with newlines embedded in them, dealing with them will be a different chapter.
I'd strongly recommend to rename those bad filenames asap, to avoid future troubles.
# 6  
Old 04-06-2009
Yeah, you're right, spaces are the most annoying, but I also had a few others that would mess with quoting...

I'd be happy to rename all those messy files, but sometimes you just can't prevent people doing what their OS allows them to do... =\

Previous Thread | Next Thread
Test Your Knowledge in Computers #244
Difficulty: Medium
Networking research in the early 1970s by Robert E. Kahn and Vint Cerf let to the formulation of the Transmission Control Program (TCP), and its specification in December 1974 in RFC 699.
True or False?

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

[md5sum] script

I am getting No such file or directory if my variable contains white spaces... Is there a way to fix this? This works x="1.md5" md5sum -c "$x" This, does not x="23\ 5\ 6\ 7\ 8\ 9\ 10.md5" md5sum -c "$x" md5sum: '23\ 5\ 6\ 7\ 8\ 9\ 10.md5': No such file or directory How do I fix... (1 Reply)
Discussion started by: soichiro
1 Replies

2. Shell Programming and Scripting

Compare two md5sum

Hello, First of all I want to apologize because i'm not a admin or coder and maybe all my efforts to write only this small script in my life would need one week full time reading man pages and forums but... I don't have the money to offer me to get this time and the script I want to do seems... (5 Replies)
Discussion started by: toscan
5 Replies

3. Shell Programming and Scripting

Md5sum on crontab

Hi all, I have to verify the integrity of a ISO image that is downloaded periodically on my PC. In order to do that, I've written a bash script that use the command md5sum with the aim to match the generated code with the default one created with the SIO image. The problem is that, if I start... (1 Reply)
Discussion started by: Mr. Piros
1 Replies

4. Shell Programming and Scripting

Compare files in directories with md5sum

And not to start. I can compare files, that's easy. The problem is that I compare files in a directory, and check if these files exist in another directory. The problem is that the file names are not the same. So I have to compare with "md5sum" or something similar. How I can do? All this in... (7 Replies)
Discussion started by: Jomeaide
7 Replies

5. Shell Programming and Scripting

Md5sum is running very slowly

Hi, I am trying to get the hash values of md5 of a string. I am on Redhat Linux. using the 25-27 field in the file I need to generate the md5 and append it at the end of the record as a new field. I have tried the below code but its painfully slow. can you please suggest any alternatives or... (21 Replies)
Discussion started by: ahmedwaseem2000
21 Replies

6. Shell Programming and Scripting

Making a script to copy files not seen before (using md5sum)

Hello, I would like to make a script that searches through a SRC folder and copies only files it's never seen before to a DEST folder. SRC = /user/.phonesync/photos-backup DST = /usr/.phonesync/photos-new So basically, I'd start with a: md5sum /user/.phonesync/photos-backup/* >... (29 Replies)
Discussion started by: nbsparks
29 Replies

7. Shell Programming and Scripting

how to get a md5sum in perl

hi All: i write a adduser script in perl , but I don't know how to deal with the password , for it stored as md5. and i don't use the shell command passwd. give me some advice...thanks (1 Reply)
Discussion started by: kingdream
1 Replies

8. Programming

md5sum and execve

Hello there! Is there a way to use execve() to run md5sum function? for example execve("md5sum <filename>, NULL,NULL);" thanx! (2 Replies)
Discussion started by: nicos
2 Replies

9. UNIX for Dummies Questions & Answers

the file: MD5SUM

i downloaded a Linux distribution from a FTP site today, and i found there is a file named MD5SUM in the same directory, with the following contents: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 c9a4d963a49e384e10dec9c2bd49ad73 valhalla-SRPMS-disc1.iso 41b03d068e84d2a17147aa27e704f79b ... (1 Reply)
Discussion started by: samprax
1 Replies

10. UNIX for Dummies Questions & Answers

What is md5sum???

Hi all, I am kinda puzzled. When and Why do we use md5sum? I've read man pages for mp5sum, but didn't get anything out of it. Please, can someone explain this to me in couple of words. Thank you all. (1 Reply)
Discussion started by: solvman
1 Replies

Featured Tech Videos