parse csv file, sha1 hash and output | Unix Linux Forums | Shell Programming and Scripting

  Go Back    


Shell Programming and Scripting Post questions about KSH, CSH, SH, BASH, PERL, PHP, SED, AWK and OTHER shell scripts and shell scripting languages here.

parse csv file, sha1 hash and output

Shell Programming and Scripting


Tags
awk, awk trim, bash, csv, md5sum, sha1sum, trim, trim awk

Closed Thread    
 
Thread Tools Search this Thread Display Modes
    #1  
Old 01-28-2009
FreddyG FreddyG is offline
Registered User
 
Join Date: Jan 2009
Last Activity: 28 January 2009, 12:37 PM EST
Posts: 7
Thanks: 0
Thanked 0 Times in 0 Posts
parse csv file, sha1 hash and output

I have a file, not really a csv, but containing delineated data just the same. Lets call that file "raw_data.txt". It contains data in the format of company name:fein number like this:


Code:
first company name:123456789
second company name:987654321

what i need to do is read this file, apply sha1 to the fein field and then output that into a new file, we'll call "hashed_data.txt". So hashed_data.txt should contain data like


Code:
first company name:f7c3bc1d808e04732adf679965ccc34ca7ae3441
second company name:bfe54caa6d483cc3887dce9d1b8eb91408f1ea7a

i have tried looping thru the file with awk like so


Code:
read FILE
while read line
do
        awk 'BEGIN{FS=OFS=":"}{print $1, $2}'
done < $FILE

but I'm stuck on how to run the second field thru sha1sum
Sponsored Links
    #2  
Old 01-28-2009
danmero danmero is offline Forum Advisor  
 
Join Date: Nov 2007
Last Activity: 12 October 2014, 10:11 AM EDT
Location: H3X
Posts: 2,151
Thanks: 10
Thanked 117 Times in 110 Posts
bash solution.

Code:
IFS=":" && \
while read name fein
        do echo "$name:"$(sha1 -qs $fein)
done < file > newfile

Note: You should read the sha1sum man

Last edited by danmero; 01-28-2009 at 10:30 AM.. Reason: Add note
Sponsored Links
    #3  
Old 01-28-2009
FreddyG FreddyG is offline
Registered User
 
Join Date: Jan 2009
Last Activity: 28 January 2009, 12:37 PM EST
Posts: 7
Thanks: 0
Thanked 0 Times in 0 Posts
thanks! checking it out now
    #4  
Old 01-28-2009
rwuerth rwuerth is offline
Registered User
 
Join Date: Jan 2009
Last Activity: 19 May 2014, 10:57 AM EDT
Posts: 139
Thanks: 13
Thanked 15 Times in 13 Posts
if your echo supports '-n' have you tried to put that into your echo statement?


Code:
echo -n "$name:"$(sha1sum <<< $fein)

I'm not sure if it will or wont work, I just don't know if it's been tried.
Sponsored Links
    #5  
Old 01-28-2009
FreddyG FreddyG is offline
Registered User
 
Join Date: Jan 2009
Last Activity: 28 January 2009, 12:37 PM EST
Posts: 7
Thanks: 0
Thanked 0 Times in 0 Posts
i think the command I'm actually going for is sha1sum. When i modified the first snipit you posted to this it seemed to work


Code:
IFS=":" && \
while read name fein
        do echo "$name:"$(sha1sum <<< $fein)
done < raw_data.txt > hashed_data.txt

however, it's getting a newline feed in the hash as well

the accurate hash of
123456789
should be
f7c3bc1d808e04732adf679965ccc34ca7ae3441


Code:
echo -n 123456789 | sha1sum

will produce this hash

so need a way to trim the newline feed out of there.


Code:
tr -d '\n'

but where would it go ion the above snipit?
Sponsored Links
    #6  
Old 01-28-2009
FreddyG FreddyG is offline
Registered User
 
Join Date: Jan 2009
Last Activity: 28 January 2009, 12:37 PM EST
Posts: 7
Thanks: 0
Thanked 0 Times in 0 Posts
Quote:
Originally Posted by rwuerth View Post
if your echo supports '-n' have you tried to put that into your echo statement?


Code:
echo -n "$name:"$(sha1sum <<< $fein)

I'm not sure if it will or wont work, I just don't know if it's been tried.
seems to just remove the new line feed between the lines, IE ouput is now


Code:
first company name:179c94cf45c6e383baf52621687305204cef16f9  -second company name:a1b42d633e975efc2f665bda21f94e419c1b6074  -

Sponsored Links
    #7  
Old 01-28-2009
rwuerth rwuerth is offline
Registered User
 
Join Date: Jan 2009
Last Activity: 19 May 2014, 10:57 AM EDT
Posts: 139
Thanks: 13
Thanked 15 Times in 13 Posts
Okay take the -n out of the echo.

try this before the echo statement


Code:
fein=`echo $fein | tr -d /\n/`

Or as I think about it, just


Code:
fein=`echo -n $fein`

Sponsored Links
Closed Thread

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Parse Logfile output variable Ikon Shell Programming and Scripting 11 12-12-2008 09:37 AM
To parse through the file and print output using awk or sed script cdfd123 Shell Programming and Scripting 4 03-03-2008 11:07 AM
How to parse through a file and based on condition form another output file sivasu.india UNIX for Advanced & Expert Users 6 02-28-2008 03:59 AM
parse through one text file and output many sophiadun UNIX for Dummies Questions & Answers 14 02-20-2008 05:08 AM
Parse ping output palm101 Shell Programming and Scripting 3 11-10-2007 11:16 PM



All times are GMT -4. The time now is 12:16 PM.