Sponsored Content
Top Forums UNIX for Advanced & Expert Users Solution for the Massive Comparison Operation Post 302428527 by raghav288 on Thursday 10th of June 2010 07:49:18 AM
Old 06-10-2010
Solution for the Massive Comparison Operation

Hi

We have 50 million records in mainframes DB2. We have a requirement to Record the Change Data Capture(CDC) records.

i.e New Records or Updated Records that were added into the DB2.
Unfortunately we dont have any column indicators to give the details of the changes made to the records.

So we decided to import the same into flat files into UNIX box everyday and then compare the previous day's file and get the changed rows using UNIX text processing functionalities.

The problem is everyday huge data changes happen approx 40 million

So can anyone give me a solution for better handling of the same or even the hardware requirement for UNIX Server for faster processing of the huge data comparison.

Thanks
 

5 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Looking for AWK Solution for column comparison in a single file

- I am looking for different kind of awk solution which I don't think is mentioned before in these forums. Number of rows in the file are fixed Their are two columns in file1.txt 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 I am looking for 3... (1 Reply)
Discussion started by: softwarekids23
1 Replies

2. Shell Programming and Scripting

Column operation : cosne and sine operation

I have a txt file with several columns and i want to peform an operation on two columns and output it to a new txt file . file.txt 900.00000 1 1 1 500.00000 500.00000 100000.000 4 4 1.45257346E-07 899.10834 ... (4 Replies)
Discussion started by: shashi792
4 Replies

3. Homework & Coursework Questions

having massive trouble with 5 questions about egrep!

Hi all! I need help to do a few things with a .txt file using egrep. 1. I need to list all sequences where the vowel letters 'a, e, i, o, u' occur in that order, possibly separated by characters other than a, e, i, o, u; consisting of one or more complete words, possibly including punctuation. ... (1 Reply)
Discussion started by: dindiqotu
1 Replies

4. Shell Programming and Scripting

Massive Copy With Base Directory

I have a script that I am using to copy around 40-70k files to a NFS NAS. I have posted my code below in hopes that someone can help me figure out a faster way of achieving this. At the end of the script i need to have all the files in the list, copied over to the nas with source directory... (8 Replies)
Discussion started by: nitrobass24
8 Replies

5. Shell Programming and Scripting

Massive ftp

friends good morning FTP works perfect but I have a doubt if I want to transport 10 files, I imagine that I should not open 10 connections as I can transfer more than 1 file? ftp -n <<!EOF open caburga user ephfact ephfact cd /users/efactura/docONE/entrada bin mput EPH`date... (16 Replies)
Discussion started by: tricampeon81
16 Replies
dspam_merge(1)							       DSPAM							    dspam_merge(1)

NAME
dspam_merge - merge several users' metadata into a composite SYNOPSIS
dspam_merge [--profile=PROFILE] user1 user2 ... userN -o username DESCRIPTION
dspam_merge merges several users' metadata into a single user's dictionary. This tool is designed to create global users and seeded data. The hit sounds for each token and per-user totals are added together to produce a single composite dataset. After creating a composite user, dspam_clean should be run with the -p option to clean up extraneous data. NOTE : Merges may take a considerable amount of time. This could potentially increase the load on the server or even slow down the delivery of email. A merge should only be performed when the system is fairly acquiesce. OPTIONS
--profile=PROFILE Specify a storage profile from dspam.conf. The storage profile selected will be used for all database connectivity. See dspam.conf for more information. user1 user2 ... userN A list of users to merge together. -o username The target user which will be created (if necessary). This user will contain the composite generated by the merge. EXAMPLES
dspam_merge dick jane spot -o ralph Merges the metadata dictionaries of dick, jane, and spot into a single composite under the user ralph EXIT VALUE
0 Operation was successful. other Operation resulted in an error. COPYRIGHT
Copyright (C) 2002-2011 DSPAM Project All rights reserved. For more information, see http://dspam.sourceforge.net. SEE ALSO
dspam(1), dspam_admin(1), dspam_clean(1), dspam_crc(1), dspam_dump(1), dspam_logrotate(1), dspam_stats(1), dspam_train(1) DSPAM
Apr 17, 2010 dspam_merge(1)
All times are GMT -4. The time now is 04:01 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy