This is not a typical question. I have a fully working script but I'm interested in optimizing it.
I frequently back up photos and movies from my digital camera and cell phone to both my home/work desktops, laptops, wife's netbook, and my home NAS and often end up with multiple versions of the same files in folders of varying completeness. I previously wrote a very slow script that checked if files with the same length were identical. It was also limited to a single directory. I recently wrote this much faster script that digs recursively through a directory tree (with find instead of ls), creates a field containing size, checksum, basename and full path name, sorts them, and deletes all but the first when size and checksum are identical. The script uses pipes to avoid arrays and executes quite fast. The alphabetically first basename gets kept. I use a comma to separate the size and checksum part (to test if identical) and a backslash before the path name so I can pass everything on the pipe and separate after. There is a dash between size and checksum that was useful for debugging and I don't think impacts speed. The path ends up being part of the sort, but that is harmless since it is after the basename. The parameter substitutions are robust for either comma or backslash occurring in the file name or path (God forbid!).
Since it works so well, I'd like to use it as a general tool. Is md5sum good enough? Does adding size (was a legacy optimization from my first script) really add any benefit in collision resistance or execution speed in the sort command? Any ideas on how to make this script faster or more robust?
Edit: sha1sum is about 13% slower than md5sum but probably eliminates collision concerns.
Mike
Last edited by Michael Stora; 08-30-2010 at 04:44 PM..
Hi all,
I am kinda puzzled. When and Why do we use md5sum? I've read man pages for mp5sum, but didn't get anything out of it. Please, can someone explain this to me in couple of words.
Thank you all. (1 Reply)
i downloaded a Linux distribution from a FTP site today, and i found there is a file named MD5SUM in the same directory, with the following contents:
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
c9a4d963a49e384e10dec9c2bd49ad73 valhalla-SRPMS-disc1.iso
41b03d068e84d2a17147aa27e704f79b ... (1 Reply)
Hi,
I am using isql and putting the output in a file in a shell script in Linux.
However in my output i am getting the banner and a SQL prompt(at the start and the end of query output)
+---------------------------------------+
| Connected! |
| ... (6 Replies)
Is it possible to call the unix command md5sum from within a C program. I am trying to write a C program that scans a directory and computes the MD5Sum of all the files in the directory. Whenever I use md5sum 'filename' I get the error 'md5sum undeclared'. Is there a header file or some library... (3 Replies)
Hi,
I currently have a shell script that takes an RPM and scp's it to a set of remote servers and installs it.
What I would like to be able to do is make the script get the md5sum of the RPM locally (so get the md5sum of the rpm from where im running the script) and then scp the rpm to the... (0 Replies)
hi All:
i write a adduser script in perl , but I don't know how to deal with the password , for it stored as md5. and i don't use the shell
command passwd. give me some advice...thanks (1 Reply)
I am trying to download a file and make the filename of the file be the md5sum of the URL. I know to use wgets to download the file but I do not know how to do the rest...any help would be appreciated. (2 Replies)
Hi there,
I found something very weird!
Should I report that as a bug or is it me misusing the command?
I've got a file with a backslash in its name.
I know it's a horrible policy but it's not me.
The file came from a mac computer because this is a backup server.
Anyway, when using... (8 Replies)
Hello.
I'm writing a script where every file you create will generate a md5sum and store it into a text file.
Say I create 2 files, it'll look like this in the text file:
d41d8cd98f00b204e9800998ecf8427e /helloworld/saystheman
d41d8cd98f00b204e9800998ecf8427e /helloworld/test
I... (3 Replies)
Hello,
First of all I want to apologize because i'm not a admin or coder and maybe all my efforts to write only this small script in my life would need one week full time reading man pages and forums but...
I don't have the money to offer me to get this time and the script I want to do seems... (5 Replies)
Discussion started by: toscan
5 Replies
LEARN ABOUT OPENSOLARIS
gthumb
gthumb(1)gthumb(1)NAME
gThumb - an image viewer and browser for GNOME
SYNOPSIS
gthumb [--help] [--version] [-f] [--fullscreen] [-s] [--slideshow] [--import-photos] [directory] [filename] ...
DESCRIPTION
With gThumb you can browse your hard disk viewing thumbnails of image files. It also lets you view single files (including GIF anima-
tions), add comments to images, organize images in catalogs, print images, view slideshows, set your desktop background, and more.
OPTIONS -f, --fullscreen
Start in fullscreen mode.
-s, --slideshow
Automatically start a slideshow.
--import-photos
Automatically import digital camera photos.
--help Output help information and exit.
--version
Output version information and exit.
Usual other GNOME options apply as well.
EXAMPLES
To run this program the standard way type:
gthumb
To view the content of a directory type:
gthumb /path/to/directory
To view all images with jpg extension type:
gthumb *.jpg
AUTHORS
Written by Paolo Bacchilega <paolo.bacch@tin.it>.
This manual page was originally written by Remco van de Meent <remco@debian.org>, for the Debian GNU/Linux system, now is maintained by
Paolo Bacchilega.
ATTRIBUTES
See attributes(5) for descriptions of the following attributes:
+-------------------------------------+-----------------------------+
| ATTRIBUTE TYPE^ATTRIBUTE VALUE | |
+-------------------------------------+-----------------------------+
|Availability^SUNWgnome-img-organzier | |
+-------------------------------------+-----------------------------+
|Interface stability^Volatile | |
| | |
+-------------------------------------+-----------------------------+
GNOME Applications May 2002 gthumb(1)