Md5sum is running very slowly


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Md5sum is running very slowly
# 1  
Old 01-29-2014
Md5sum is running very slowly

Hi,

I am trying to get the hash values of md5 of a string. I am on Redhat Linux. using the 25-27 field in the file I need to generate the md5 and append it at the end of the record as a new field.

I have tried the below code but its painfully slow. can you please suggest any alternatives or help me tune it?

Code:
awk -F"\¬" '{ print $25" " $26 " " $27 }' /var/IBM/CMA/LandingArea/Analysis/Add.txt | while read x ; do echo $x |md5sum ; done

# 2  
Old 01-29-2014
Do you have a compiler installed?
# 3  
Old 01-29-2014
Quote:
Originally Posted by Corona688
Do you have a compiler installed?
which compiler are you referring to? I do have c++ installed
# 4  
Old 01-29-2014
Code:
$ cat md5line.c

#include <openssl/md5.h>
#include <unistd.h>
#include <stdio.h>
#include <string.h>

int main()
{
        int n;
        MD5_CTX c;
        unsigned char buf[1024], out[MD5_DIGEST_LENGTH];

        while(fgets(buf, 1024, stdin))
        {
                MD5_Init(&c);
                MD5_Update(&c, buf, strlen(buf)-1);
                MD5_Final(out, &c);

                for(n=0; n<MD5_DIGEST_LENGTH; n++)
                        printf("%02x", out[n]);
                fputs("\n",stdout);
        }

        return(0);
}

$ gcc md5line.c -o md5line -lssl # May need "openssl-dev" or something like that

$ printf "%s\n" a b c d e

a
b
c
d
e

$ echo "a" | md5sum # This includes the \n!
60b725f10c9c85c70d97880dfe8191b3  -

$ printf "a" | md5sum # Does not add \n
0cc175b9c0f1b6a831c399e269772661  -

$ printf "%s\n" a b c d e | ./md5line # My code also ignores \n
0cc175b9c0f1b6a831c399e269772661
92eb5ffee6ae2fec3ad71c777531578f
4a8a08f09d37b73795649038408b5f33
8277e0910d750195b448797616e091ad
e1671797c52e15f763380b45e841ec32

$

# 5  
Old 01-29-2014
Quote:
Originally Posted by Corona688
Code:
$ cat md5line.c

#include <openssl/md5.h>
#include <unistd.h>
#include <stdio.h>
#include <string.h>

int main()
{
        int n;
        MD5_CTX c;
        unsigned char buf[1024], out[MD5_DIGEST_LENGTH];

        while(fgets(buf, 1024, stdin))
        {
                MD5_Init(&c);
                MD5_Update(&c, buf, strlen(buf)-1);
                MD5_Final(out, &c);

                for(n=0; n<MD5_DIGEST_LENGTH; n++)
                        printf("%02x", out[n]);
                fputs("\n",stdout);
        }

        return(0);
}

$ gcc md5line.c -o md5line -lssl # May need "openssl-dev" or something like that

$ printf "%s\n" a b c d e

a
b
c
d
e

$ echo "a" | md5sum # This includes the \n!
60b725f10c9c85c70d97880dfe8191b3  -

$ printf "a" | md5sum # Does not add \n
0cc175b9c0f1b6a831c399e269772661  -

$ printf "%s\n" a b c d e | ./md5line # My code also ignores \n
0cc175b9c0f1b6a831c399e269772661
92eb5ffee6ae2fec3ad71c777531578f
4a8a08f09d37b73795649038408b5f33
8277e0910d750195b448797616e091ad
e1671797c52e15f763380b45e841ec32

$

Thanks for the code. I did try to compile but I get the error message as shown below. I have no c\c++ skills to be able to debug it. can you please help me?

Code:
g++ -c -O -fPIC -Wno-deprecated -m64 -mtune=generic -mcmodel=small md5tester.cpp  -o libmd5teser.so
md5tester.cpp:1:25: error: openssl/md5.h: No such file or directory
md5tester.cpp: In function ‘int main()':
md5tester.cpp:9: error: ‘MD5_CTX' was not declared in this scope
md5tester.cpp:9: error: expected `;' before ‘c'
md5tester.cpp:10: error: ‘MD5_DIGEST_LENGTH' was not declared in this scope
md5tester.cpp:12: error: invalid conversion from ‘unsigned char*' to ‘char*'
md5tester.cpp:12: error:   initializing argument 1 of ‘char* fgets(char*, int, FILE*)'
md5tester.cpp:14: error: ‘c' was not declared in this scope
md5tester.cpp:14: error: ‘MD5_Init' was not declared in this scope
md5tester.cpp:15: error: invalid conversion from ‘unsigned char*' to ‘const char*'
md5tester.cpp:15: error:   initializing argument 1 of ‘size_t strlen(const char*)'
md5tester.cpp:15: error: ‘MD5_Update' was not declared in this scope
md5tester.cpp:16: error: ‘out' was not declared in this scope
md5tester.cpp:16: error: ‘MD5_Final' was not declared in this scope

# 6  
Old 01-29-2014
How about using perl

Code:
#!/usr/bin/perl -w
use Digest::MD5 qw(md5 md5_hex md5_base64);

open my $DAT, $ARGV[0] or die "Could not open $ARGV[0]: $!";

while (my $line = <$DAT>) {
  chomp $line;
  my @fld = split /¬/, $line;
  print $line . "¬" . md5_hex($fld[24]." ".$fld[25]." ".$fld[26] . "\n") . "\n";
}

close $DAT;

Save as addms5sum.pl and call it like this:

Code:
$ ./addmd5sum.pl /var/IBM/CMA/LandingArea/Analysis/Add.txt > /var/IBM/CMA/LandingArea/Analysis/Add_fixed.txt


Edit: I've included the new line in the md5sum value, remove . "\n" from the md5_hex() call above if you don't need it.
# 7  
Old 01-29-2014
Quote:
Originally Posted by Chubler_XL
How about using perl

Code:
#!/usr/bin/perl -w
use Digest::MD5 qw(md5 md5_hex md5_base64);

open my $DAT, $ARGV[0] or die "Could not open $ARGV[0]: $!";

while (my $line = <$DAT>) {
  chomp $line;
  my @fld = split /¬/, $line;
  print $line . "¬" . md5_hex($fld[24]." ".$fld[25]." ".$fld[26] . "\n") . "\n";
}

close $DAT;

Save as addms5sum.pl and call it like this:

Code:
$ ./addmd5sum.pl /var/IBM/CMA/LandingArea/Analysis/Add.txt > /var/IBM/CMA/LandingArea/Analysis/Add_fixed.txt

Now I am getting this error
Code:
Use of uninitialized value in concatenation (.) or string at ./addmd5sum.pl line 9, <$DAT> line 10398.

Login or Register to Ask a Question

Previous Thread | Next Thread

6 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Checking Unix Performance - Why is a process running slowly?

Hi Please can someone explain to me how they would go about monitoring the performance of a process in Unix. Lets say that a user is running a process in Unix but it seems to be taking a long time, whereas it completed a lot quicker yesterday. How would you go about investigating what is causing... (1 Reply)
Discussion started by: Sunny Sid
1 Replies

2. SuSE

SLES 10 SP2 possible kernel problem, / slowly filling up

Hello Guys I first though about posting this to emergency but cause I fixed my issue with an reboot its not as important, more is important to me what caused this situation Some facts: OS: SLES 10 x64 SP2 (Virtualized Vmware ESX 3.5) / vmware tools status OK Soft: Oracle10g LVM... (1 Reply)
Discussion started by: kl1ngac1k
1 Replies

3. Solaris

Sun StorageTek Common Array Manager 6.0 works very slowly

Hi! I have Sun StorageTek 2540 FC array and CAM works very slowly - I can wait for software response even more than 2 minutes... I run this software on Windows machine with Firefox Web Browser but speed is terrible... How can I make it works at least a little bit faster?.. (2 Replies)
Discussion started by: Sapfeer
2 Replies

4. Shell Programming and Scripting

Running md5sum on a list of files

Hello, I would like to run md5sum on a list of files saved in a text file, and save the result in another file. (ie. md5sum `cat list.txt` > md5list.txt) I have tried several things, but I am always confronted to the same problem: some of the filenames have spaces. I have run sed on the... (5 Replies)
Discussion started by: SDelroen
5 Replies

5. Solaris

Delete and copy file(s) slowly(!?)

Hi all! I have to monitor space in V890 machine, Solaris 10 weekly, because there is Oracle DB on it with many datafiles which have been taken offline to make enough size. Sometime, one or more datafiles are big, they are 20GB, 40GB etc.. The problem I have encountered is the processing of... (5 Replies)
Discussion started by: trantuananh24hg
5 Replies

6. UNIX for Dummies Questions & Answers

What is md5sum???

Hi all, I am kinda puzzled. When and Why do we use md5sum? I've read man pages for mp5sum, but didn't get anything out of it. Please, can someone explain this to me in couple of words. Thank you all. (1 Reply)
Discussion started by: solvman
1 Replies
Login or Register to Ask a Question