Sponsored Content
Full Discussion: Hashing URLs
Top Forums Shell Programming and Scripting Hashing URLs Post 302872493 by twjolson on Friday 8th of November 2013 12:49:22 PM
Old 11-08-2013
Hashing URLs

So, I am writing a script that will read output from Bulk Extractor (which gathers data based on regular expressions). My script then reads the column that has the URL found, hashes it with MD5, then outputs the URL and hash to a file.

Where I am stuck on is that I want to read the bulk extractor output, line by line. I want to take the 2nd column, and add that as a variable named "url". I then want to take that URL, and hash it with MD5 and assign that to the variable "hash".

I then want to output $url and $hash to an output file, both on the same line.

But, when I use :
Code:
while read line
	do
		url=`awk '{print $2}'`
		hash=`awk '{print $2}' | md5sum`
		echo $url
	done < $iname/url.txt

to walk through the input file, line by line, it globs them all together. $url has EVERY URL, and $hash is a hash value of ALL the URLs.

I know this snippet of code works - I've used it before. So, what is going on?

Here is a snippet of the input :
Code:
1691	http://www.pof.com/inbox.aspx_	\007\0000\0001\0002\0003\000(\0004_\020\035http://www.pof.com/inbox.aspx_\0200Online Dating 
1874	http://www.pof.com/sendmessage.aspx	WasHTTPNonGet_\020#http://www.pof.com/sendmessage.aspx[401152043.2\020R\011_
1927	http://www.pof.com/inbox.aspx?messagesent=1&Guid=63929064&SID=dnia5geyks5fjvr2hfesqwbg#in	01152043.2\020R\011_\020Yhttp://www.pof.com/inbox.aspx?messagesent=1&Guid=63929064&SID=dnia5geyks5fjvr2hfesqwbg#in¡\000F\020\014\020\021¬\000H\000\025\000I\000\025
2101	http://www.pof.com/viewallmessages.aspx?sender_id=41561852&message_id=17458470377&Guid=63929064&SID=dnia5geyks5fjvr2hfesqwbg_	7\000C\000D\000E\000G\000J\000M_\020|http://www.pof.com/viewallmessages.aspx?sender_id=41561852&message_id=17458470377&Guid=63929064&SID=dnia5geyks5fjvr2hfesqwbg_\020.POF.com Free O

 

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Hashing or MD5

Hi, how can one find that which encryption algorithm the system is using for keeping the user password in the /etc/passwd or /etc/shadow file. Is it 1: Hashing ( which considers only first 5 letters of password) 2: MD5 (Which allows arbitry length passwords) Thanks, ~amit (0 Replies)
Discussion started by: amit4g
0 Replies

2. Shell Programming and Scripting

Perl Hashes, reading and hashing 2 files

So I have two files that I want to put together via hashes and am having a terrible time with syntax. For example: File1 A apple B banana C citrusFile2 A red B yellow C orangeWhat I want to enter on the command line is: program.pl File1 File2And have the result... (11 Replies)
Discussion started by: silkiechicken
11 Replies

3. AIX

How to : Find Which hashing algorithem used in AIX Box ?

hello Friends , How can i identify the hashing algo used by shadow file in aix box >??? Thanks AVKlinux (1 Reply)
Discussion started by: avklinux
1 Replies

4. UNIX for Dummies Questions & Answers

How to analyze file hashing

What command should I use to analyze file hashing of fixed flat files. How much work does it take for multiple flat files. (3 Replies)
Discussion started by: jbjoat
3 Replies

5. UNIX for Dummies Questions & Answers

file hashing utility in unix

I am looking for a utility that does file hashing in unix. ...Please let me know of any good easy to use utility (3 Replies)
Discussion started by: jbjoat
3 Replies

6. Programming

Linear hashing implementation in C language

Hi, I'm looking for linear hashing implementation in C language. Please help. PS: I have implement this on Ubuntu 10.04 Linux on 64 bit machine. (1 Reply)
Discussion started by: sajjar
1 Replies

7. UNIX for Advanced & Expert Users

password hashing algorithms

I'm collecting some info on the password hashing algorithms in use on various Unix systems. So far I have: no $ legacy unix crypt $1$ MD5 $2$ Blowfish on BSD $2a$ alternate Blowfish on BSD $md5$ Sun's alternate MD5 $3$ a Microsoft hash $4$ not used? $5$ RedHat proposed Sha-256... (2 Replies)
Discussion started by: Perderabo
2 Replies

8. Solaris

[solved] Password hashing

Hello, I'm having an issue with my password hashing. In /etc/shadow all the passwords hashes start with $1$. The security people want me to change it so the password hash starts with $5$ or $6$. So this is what I did to fix this. I changed CRYPT_DEFAULT for 1 to 6 CRYPT_DEFAULT=6When I create a... (0 Replies)
Discussion started by: bitlord
0 Replies

9. Solaris

Hashing password with bcrypt in Solaris 10

Hi, Our security audit person generated a report for Solaris-10 servers and mentioned this suggestion - "All passwords should be hashed using bcrypt. Solaris 10 supports this blowfish-based hash algorithm with the identifier 2a. To verify this, ensure the password hashes start with $2a$.... (2 Replies)
Discussion started by: solaris_1977
2 Replies
Template::Plugin::URL(3)				User Contributed Perl Documentation				  Template::Plugin::URL(3)

NAME
Template::Plugin::URL - Plugin to construct complex URLs SYNOPSIS
[% USE url('/cgi-bin/foo.pl') %] [% url(debug = 1, id = 123) %] # ==> /cgi/bin/foo.pl?debug=1&amp;id=123 [% USE mycgi = url('/cgi-bin/bar.pl', mode='browse', debug=1) %] [% mycgi %] # ==> /cgi/bin/bar.pl?mode=browse&amp;debug=1 [% mycgi(mode='submit') %] # ==> /cgi/bin/bar.pl?mode=submit&amp;debug=1 [% mycgi(debug='d2 p0', id='D4-2k[4]') %] # ==> /cgi-bin/bar.pl?mode=browse&amp;debug=d2%20p0&amp;id=D4-2k%5B4%5D DESCRIPTION
The "URL" plugin can be used to construct complex URLs from a base stem and a hash array of additional query parameters. The constructor should be passed a base URL and optionally, a hash array reference of default parameters and values. Used from with a template, it would look something like the following: [% USE url('http://www.somewhere.com/cgi-bin/foo.pl') %] [% USE url('/cgi-bin/bar.pl', mode='browse') %] [% USE url('/cgi-bin/baz.pl', mode='browse', debug=1) %] When the plugin is then called without any arguments, the default base and parameters are returned as a formatted query string. [% url %] For the above three examples, these will produce the following outputs: http://www.somewhere.com/cgi-bin/foo.pl /cgi-bin/bar.pl?mode=browse /cgi-bin/baz.pl?mode=browse&amp;debug=1 Note that additional parameters are separated by '"&amp;"' rather than simply '"&"'. This is the correct behaviour for HTML pages but is, unfortunately, incorrect when creating URLs that do not need to be encoded safely for HTML. This is likely to be corrected in a future version of the plugin (most probably with TT3). In the mean time, you can set $Template::Plugin::URL::JOINT to "&" to get the correct behaviour. Additional parameters may be also be specified to the URL: [% url(mode='submit', id='wiz') %] Which, for the same three examples, produces: http://www.somewhere.com/cgi-bin/foo.pl?mode=submit&amp;id=wiz /cgi-bin/bar.pl?mode=browse&amp;id=wiz /cgi-bin/baz.pl?mode=browse&amp;debug=1&amp;id=wiz A new base URL may also be specified as the first option: [% url('/cgi-bin/waz.pl', test=1) %] producing /cgi-bin/waz.pl?test=1 /cgi-bin/waz.pl?mode=browse&amp;test=1 /cgi-bin/waz.pl?mode=browse&amp;debug=1&amp;test=1 The ordering of the parameters is non-deterministic due to fact that Perl's hashes themselves are unordered. This isn't a problem as the ordering of CGI parameters is insignificant (to the best of my knowledge). All values will be properly escaped thanks to some code borrowed from Lincoln Stein's "CGI" module. e.g. [% USE url('/cgi-bin/woz.pl') %] [% url(name="Elrich von Benjy d'Weiro") %] Here the spaces and ""'"" character are escaped in the output: /cgi-bin/woz.pl?name=Elrich%20von%20Benjy%20d%27Weiro An alternate name may be provided for the plugin at construction time as per regular Template Toolkit syntax. [% USE mycgi = url('cgi-bin/min.pl') %] [% mycgi(debug=1) %] AUTHOR
Andy Wardley <abw@wardley.org> <http://wardley.org/> COPYRIGHT
Copyright (C) 1996-2007 Andy Wardley. All Rights Reserved. This module is free software; you can redistribute it and/or modify it under the same terms as Perl itself. SEE ALSO
Template::Plugin perl v5.16.3 2011-12-20 Template::Plugin::URL(3)
All times are GMT -4. The time now is 11:02 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy