Sponsored Content
Top Forums UNIX for Advanced & Expert Users Solution for the Massive Comparison Operation Post 302429219 by raghav288 on Sunday 13th of June 2010 01:58:08 PM
Old 06-13-2010
Another Idea for the same solution..

Hi

Thanks for the solution.. We had come up with a solution for comparing the huge data..

Since we are comparing huge data of flat file records, the follwing can be done

A hash function may be used like you mentioned below for each rows on the flat files, making the comparison easier.

But Is there a utility hash function in unix same as that of orahash in oracle that wud encrypt each new row uniquely within a few set of characters or numbers.

Then we cud use only those hashed codes to compare with the old hash codes of the prev day file and which wud make processing faster too...
 

5 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Looking for AWK Solution for column comparison in a single file

- I am looking for different kind of awk solution which I don't think is mentioned before in these forums. Number of rows in the file are fixed Their are two columns in file1.txt 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 I am looking for 3... (1 Reply)
Discussion started by: softwarekids23
1 Replies

2. Shell Programming and Scripting

Column operation : cosne and sine operation

I have a txt file with several columns and i want to peform an operation on two columns and output it to a new txt file . file.txt 900.00000 1 1 1 500.00000 500.00000 100000.000 4 4 1.45257346E-07 899.10834 ... (4 Replies)
Discussion started by: shashi792
4 Replies

3. Homework & Coursework Questions

having massive trouble with 5 questions about egrep!

Hi all! I need help to do a few things with a .txt file using egrep. 1. I need to list all sequences where the vowel letters 'a, e, i, o, u' occur in that order, possibly separated by characters other than a, e, i, o, u; consisting of one or more complete words, possibly including punctuation. ... (1 Reply)
Discussion started by: dindiqotu
1 Replies

4. Shell Programming and Scripting

Massive Copy With Base Directory

I have a script that I am using to copy around 40-70k files to a NFS NAS. I have posted my code below in hopes that someone can help me figure out a faster way of achieving this. At the end of the script i need to have all the files in the list, copied over to the nas with source directory... (8 Replies)
Discussion started by: nitrobass24
8 Replies

5. Shell Programming and Scripting

Massive ftp

friends good morning FTP works perfect but I have a doubt if I want to transport 10 files, I imagine that I should not open 10 connections as I can transfer more than 1 file? ftp -n <<!EOF open caburga user ephfact ephfact cd /users/efactura/docONE/entrada bin mput EPH`date... (16 Replies)
Discussion started by: tricampeon81
16 Replies
Data::Serializer::Cookbook(3pm) 			User Contributed Perl Documentation			   Data::Serializer::Cookbook(3pm)

NAME
Cookbook - Examples of how to use Data::Serializer DESCRIPTION
Data::Serializer::Cookbook is a collection of solutions for using Data::Serializer. CONVENTIONS
Unless otherwise specified, all examples can be assumed to begin with: use Data::Serializer; my $serializer = Data::Serializer->new(); Some examples will show different arguments to the new method, where specified simply use that line instead of the simple form above. CONVENTIONS for Raw Access Fort hose who want a straight pass through to the underlying serializer, where nothing else is done (no encoding, encryption, compression, etc) there is Data::Serializer::Raw(3). These begin like this: use Data::Serializer::Raw; my $raw_serializer = Data::Serializer::Raw->new(); Encrypting your data You wish to encrypt your data structure, so that it can only be decoded by someone who shares the same key. Solution $serializer->secret('mysecret'); my $encrypted_hashref = $serializer->serializer($hash); ... (in other program) ... $serializer->secret('mysecret'); my $clear_hash = $serializer->deserializer($encrypted_hash); Note: You will have to have the Crypt::CBC module installed for this to work. Compressing your data You wish to compress your data structure to cut down on how much disk space it will take up. Solution $serializer->compress(1); my $compressed_hashref = $serializer->serializer($hash); ... (in other program) ... my $clear_hash = $serializer->deserializer($compressed_hash); Note: You will have to have the Compress::Zlib module installed for this to work. Your mileage will vary dramatically depending on what serializer you use. Some serializers are already fairly compact. You want to read in data serialized outside of Data::Serializer You need to write a program that can read in data serialized in a format other than Data::Serializer. For example you need to be able to be able to process data serialized by XML::Dumper. Solution use Data::Serializer::Raw; my $xml_raw_serializer = Data::Serializer::Raw->(serializer => 'XML::Dumper'); my $hash_ref = $xml_raw_serializer->deserialize($xml_data); You want to write serialized data in a form understood outside of Data::Serializer You need to write a program that can write out data in a format other than Data::Serializer. Or said more generically you need to write out data in the format native to the underlying serializer. For our example we will be exporting data using XML::Dumper format. Solution ues Data::Serializer::Raw; my $xml_raw_serializer = Data::Serializer::Raw->(serializer => 'XML::Dumper'); my $xml_data = $xml_raw_serializer->serialize($hash_ref); You want to convert data between two different serializers native formats You have data serialized by php that you want to convert to xml for use by other programs. Solution use Data::Serializer::Raw; my $xml_raw_serializer = Data::Serializer::Raw->(serializer => 'XML::Dumper'); my $php_raw_serializer = Data::Serializer::Raw->(serializer => 'PHP::Serialization'); my $hash_ref = $php_raw_serializer->deserialize($php_data); my $xml_data = $xml_raw_serializer->serialize($hash_ref); Keeping data persistent between executions of a program. You have a program that you run every 10 minutes, it uses SNMP to pull some counters from one of your routers. You want your program to keep the counters from the last run so that it can see how much traffic has passed over a link since it last ran. Solution # path to store our serialized data # be paranoid, use full paths my $last_run_datafile = '/full/path/to/file/lastrun.data'; #We keep our data as a hash reference my $last_data = $serializer->retrieve($last_run_datafile); #Pull in our new data through 'pull_data()'; my $new_data = query_router($router); #run comparison code run_comparison($last_data,$new_data); $serializer->store($new_data); AUTHOR
Neil Neely <neil@neely.cx>. COPYRIGHT
Copyright (c) 2001-2011 Neil Neely. All rights reserved. This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself. SEE ALSO
Data::Serializer(3) Data::Serializer::Raw(3) perl v5.12.4 2011-08-16 Data::Serializer::Cookbook(3pm)
All times are GMT -4. The time now is 04:04 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy