Sponsored Content
Top Forums UNIX for Advanced & Expert Users Solution for the Massive Comparison Operation Post 302429219 by raghav288 on Sunday 13th of June 2010 01:58:08 PM
Old 06-13-2010
Another Idea for the same solution..

Hi

Thanks for the solution.. We had come up with a solution for comparing the huge data..

Since we are comparing huge data of flat file records, the follwing can be done

A hash function may be used like you mentioned below for each rows on the flat files, making the comparison easier.

But Is there a utility hash function in unix same as that of orahash in oracle that wud encrypt each new row uniquely within a few set of characters or numbers.

Then we cud use only those hashed codes to compare with the old hash codes of the prev day file and which wud make processing faster too...
 

5 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Looking for AWK Solution for column comparison in a single file

- I am looking for different kind of awk solution which I don't think is mentioned before in these forums. Number of rows in the file are fixed Their are two columns in file1.txt 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 I am looking for 3... (1 Reply)
Discussion started by: softwarekids23
1 Replies

2. Shell Programming and Scripting

Column operation : cosne and sine operation

I have a txt file with several columns and i want to peform an operation on two columns and output it to a new txt file . file.txt 900.00000 1 1 1 500.00000 500.00000 100000.000 4 4 1.45257346E-07 899.10834 ... (4 Replies)
Discussion started by: shashi792
4 Replies

3. Homework & Coursework Questions

having massive trouble with 5 questions about egrep!

Hi all! I need help to do a few things with a .txt file using egrep. 1. I need to list all sequences where the vowel letters 'a, e, i, o, u' occur in that order, possibly separated by characters other than a, e, i, o, u; consisting of one or more complete words, possibly including punctuation. ... (1 Reply)
Discussion started by: dindiqotu
1 Replies

4. Shell Programming and Scripting

Massive Copy With Base Directory

I have a script that I am using to copy around 40-70k files to a NFS NAS. I have posted my code below in hopes that someone can help me figure out a faster way of achieving this. At the end of the script i need to have all the files in the list, copied over to the nas with source directory... (8 Replies)
Discussion started by: nitrobass24
8 Replies

5. Shell Programming and Scripting

Massive ftp

friends good morning FTP works perfect but I have a doubt if I want to transport 10 files, I imagine that I should not open 10 connections as I can transfer more than 1 file? ftp -n <<!EOF open caburga user ephfact ephfact cd /users/efactura/docONE/entrada bin mput EPH`date... (16 Replies)
Discussion started by: tricampeon81
16 Replies
cdb(5)								File Formats Manual							    cdb(5)

NAME
cdb - Constant DataBase file format DESCRIPTION
A cdb database is a single file used to map `keys' to `values', having records of (key,value) pairs. File consists of 3 parts: toc (table of contents), data and index (hash tables). Toc has fixed length of 2048 bytes, containing 256 pointers to hash tables inside index sections. Every pointer consists of position of a hash table in bytes from the beginning of a file, and a size of a hash table in entries, both are 4-bytes (32 bits) unsigned integers in little-endian form. Hash table length may have zero length, meaning that corresponding hash table is empty. Right after toc section, data section follows without any alingment. It consists of series of records, each is a key length, value (data) length, key and value. Again, key and value length are 4-byte unsigned integers. Each next record follows previous without any special alignment. After data section, index (hash tables) section follows. It should be looked to in conjunction with toc section, where each of max 256 hash tables are defined. Index section consists of series of hash tables, with starting position and length defined in toc section. Every hash table is a sequence of records each holds two numbers: key's hash value and record position inside data section (bytes from the begin- ning of a file to first byte of key length starting data record). If record position is zero, then this is an empty hash table slot, pointed to nowhere. CDB hash function is hv = ((hv << 5) + hv) ^ c for every single c byte of a key, starting with hv = 5381. Toc section indexed by (hv % 256), i.e. hash value modulo 256 (number of entries in toc section). In order to find a record, one should: first, compute the hash value (hv) of a key. Second, look to hash table number hv modulo 256. If it is empty, then there is no such key exists. If it is not empty, then third, loop by slots inside that hash table, starting from slot with number hv divided by 256 modulo length of that table, or ((hv / 256) % htlen), searching for this hv in hash table. Stop search on empty slot (if record position is zero) or when all slots was probed (note cyclic search, jumping from end to beginning of a table). When hash value in question is found in hash table, look to key of corresponding record, comparing it with key in question. If them of the same length and equals to each other, then record is found, overwise, repeat with next hash table slot. Note that there may be several records with the same key. SEE ALSO
cdb(1), cdb(3). AUTHOR
The tinycdb package written by Michael Tokarev <mjt@corpit.ru>, based on ideas and shares file format with original cdb library by Dan Bernstein. LICENSE
Public domain. Apr, 2005 cdb(5)
All times are GMT -4. The time now is 10:12 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy