Sponsored Content
Top Forums UNIX for Advanced & Expert Users Solution for the Massive Comparison Operation Post 302429219 by raghav288 on Sunday 13th of June 2010 01:58:08 PM
Old 06-13-2010
Another Idea for the same solution..

Hi

Thanks for the solution.. We had come up with a solution for comparing the huge data..

Since we are comparing huge data of flat file records, the follwing can be done

A hash function may be used like you mentioned below for each rows on the flat files, making the comparison easier.

But Is there a utility hash function in unix same as that of orahash in oracle that wud encrypt each new row uniquely within a few set of characters or numbers.

Then we cud use only those hashed codes to compare with the old hash codes of the prev day file and which wud make processing faster too...
 

5 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Looking for AWK Solution for column comparison in a single file

- I am looking for different kind of awk solution which I don't think is mentioned before in these forums. Number of rows in the file are fixed Their are two columns in file1.txt 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 I am looking for 3... (1 Reply)
Discussion started by: softwarekids23
1 Replies

2. Shell Programming and Scripting

Column operation : cosne and sine operation

I have a txt file with several columns and i want to peform an operation on two columns and output it to a new txt file . file.txt 900.00000 1 1 1 500.00000 500.00000 100000.000 4 4 1.45257346E-07 899.10834 ... (4 Replies)
Discussion started by: shashi792
4 Replies

3. Homework & Coursework Questions

having massive trouble with 5 questions about egrep!

Hi all! I need help to do a few things with a .txt file using egrep. 1. I need to list all sequences where the vowel letters 'a, e, i, o, u' occur in that order, possibly separated by characters other than a, e, i, o, u; consisting of one or more complete words, possibly including punctuation. ... (1 Reply)
Discussion started by: dindiqotu
1 Replies

4. Shell Programming and Scripting

Massive Copy With Base Directory

I have a script that I am using to copy around 40-70k files to a NFS NAS. I have posted my code below in hopes that someone can help me figure out a faster way of achieving this. At the end of the script i need to have all the files in the list, copied over to the nas with source directory... (8 Replies)
Discussion started by: nitrobass24
8 Replies

5. Shell Programming and Scripting

Massive ftp

friends good morning FTP works perfect but I have a doubt if I want to transport 10 files, I imagine that I should not open 10 connections as I can transfer more than 1 file? ftp -n <<!EOF open caburga user ephfact ephfact cd /users/efactura/docONE/entrada bin mput EPH`date... (16 Replies)
Discussion started by: tricampeon81
16 Replies
HASH(3pub)						       C Programmer's Manual							HASH(3pub)

NAME
hash_create, hash_destroy, hash_install, hash_lookup, hash_uninstall, hash_iter - generic hash tables SYNOPSIS
#include <publib.h> Hashtab *hash_create(unsigned long (*fun)(void *), int (*cmp)(const void *, const void *)); void hash_destroy(Hashtab *ht); void *hash_install(Hashtab *ht, void *data, size_t size); void *hash_lookup(Hashtab *ht, void *data); int hash_uninstall(Hashtab *ht, void *data); int hash_iter(Hashtab *ht, int (*doit)(void *, void *), void *param); DESCRIPTION
These functions implement generic hash tables. The table is created by hash_create and destroyed by hash_destroy. The fun argument is a pointer to the hashing function, which must convert a datum to an unsigned long, which is then converted to an index into the hashing ta- ble. cmp is a qsort(3)-like comparison functions, used to compare to (wannabe) hash table elements. hash_install installs a new datum into the table. A pointer to the data and the size of the data are given as the arguments. If the size is 0, only the pointer value is copied to the table. Otherwise a copy of the data is made into dynamically allocated memory. hash_lookup attempts to find a datum in the hash table. A pointer to another datum is given as the argument. The comparison function should compare equal (return 0) the desired datum and this datum (but the argument needn't be a fully initialized datum, although that is up to the writer of the comparison function). There cannot be two elements in the hash table that are equal (the comparison function returns 0 for them). It is up to the user to handle collisions. hash_uninstall removes an element from a table. The argument is a pointer to a datum that identifies the element. hash_iter goes through every element in the hash table and calls the doit function for each. The first argument it provides to doit is the element in question, the second is whatever was given to hash_iter as param. If doit returns -1 or 0 for any element in the hash table, hash_iter immediately returns without going through the remaining elements in the hash table. Any other return value from doit is ignored. RETURNS
hash_create returns a pointer to the new hash table, or NULL if it fails. hash_install returns a pointer to an element in the table (either the installed one, or one that was already installed, if one tries to install the same datum twice). hash_uninstall returns 0 if it found the element in the array, or -1 if it didn't. hash_lookup return a pointer to the element it finds, or NULL if it doesn't find anything beautiful. hash_iter returns -1, 0, or 1. If hash_iter receives a return value of -1 or 0 for some element from doit, hash_iter immediately returns -1 or 0, respectively. In all other cases hash_iter returns 1. SEE ALSO
publib(3), qsort(3), bsearch(3) AUTHOR
Lars Wirzenius (lars.wirzenius@helsinki.fi) Publib C Programmer's Manual HASH(3pub)
All times are GMT -4. The time now is 12:24 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy