Sponsored Content
Top Forums UNIX for Advanced & Expert Users Solution for the Massive Comparison Operation Post 302428527 by raghav288 on Thursday 10th of June 2010 07:49:18 AM
Old 06-10-2010
Solution for the Massive Comparison Operation

Hi

We have 50 million records in mainframes DB2. We have a requirement to Record the Change Data Capture(CDC) records.

i.e New Records or Updated Records that were added into the DB2.
Unfortunately we dont have any column indicators to give the details of the changes made to the records.

So we decided to import the same into flat files into UNIX box everyday and then compare the previous day's file and get the changed rows using UNIX text processing functionalities.

The problem is everyday huge data changes happen approx 40 million

So can anyone give me a solution for better handling of the same or even the hardware requirement for UNIX Server for faster processing of the huge data comparison.

Thanks
 

5 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Looking for AWK Solution for column comparison in a single file

- I am looking for different kind of awk solution which I don't think is mentioned before in these forums. Number of rows in the file are fixed Their are two columns in file1.txt 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 I am looking for 3... (1 Reply)
Discussion started by: softwarekids23
1 Replies

2. Shell Programming and Scripting

Column operation : cosne and sine operation

I have a txt file with several columns and i want to peform an operation on two columns and output it to a new txt file . file.txt 900.00000 1 1 1 500.00000 500.00000 100000.000 4 4 1.45257346E-07 899.10834 ... (4 Replies)
Discussion started by: shashi792
4 Replies

3. Homework & Coursework Questions

having massive trouble with 5 questions about egrep!

Hi all! I need help to do a few things with a .txt file using egrep. 1. I need to list all sequences where the vowel letters 'a, e, i, o, u' occur in that order, possibly separated by characters other than a, e, i, o, u; consisting of one or more complete words, possibly including punctuation. ... (1 Reply)
Discussion started by: dindiqotu
1 Replies

4. Shell Programming and Scripting

Massive Copy With Base Directory

I have a script that I am using to copy around 40-70k files to a NFS NAS. I have posted my code below in hopes that someone can help me figure out a faster way of achieving this. At the end of the script i need to have all the files in the list, copied over to the nas with source directory... (8 Replies)
Discussion started by: nitrobass24
8 Replies

5. Shell Programming and Scripting

Massive ftp

friends good morning FTP works perfect but I have a doubt if I want to transport 10 files, I imagine that I should not open 10 connections as I can transfer more than 1 file? ftp -n <<!EOF open caburga user ephfact ephfact cd /users/efactura/docONE/entrada bin mput EPH`date... (16 Replies)
Discussion started by: tricampeon81
16 Replies
srec_stewie(5)							File Formats Manual						    srec_stewie(5)

NAME
srec_stewie - Stewie's binary file format DESCRIPTION
If you have a URL for documentation of this format, please let me know. Any resemblance to the Motorola S-Record is superficial, and extends only to the data records. The header records and termination records are completely different. None of the other Motorola S-Records record type are available. The Records All records start with an ASCII capital S character, value 0x53, followed by a type specifier byte. All records consist of binary bytes. The Header Record Each file starts with a fixed four byte header record. +-----+------+------+------+ |0x53 | 0x30 | 0x30 | 0x33 | +-----+------+------+------+ The Data Records Each data record consists of 5 fields. These are the type field, length field, address field, data field, and the checksum. The lines always start with a capital S character. +-----+------+---------------+---------+------+----------+ |0x53 | Type | Record Length | Address | Data | Checksum | +-----+------+---------------+---------+------+----------+ Type The type field is a one byte field that specifies whether the record has a two-byte address field (0x31), a three-byte address field (0x32) or a four-byte address field (0x33). The address is big-endian. Record Length The record length field is a one byte field that specifies the number of bytes in the record following this byte. Address This is a 2-, 3- or 4-byte address that specifies where the data in the record is to be loaded into memory. Data The data field contains the executable code, memory-loadable data or descriptive information to be transferred. Checksum The checksum is a one byte field that represents the least significant byte of the one's complement of the sum of the values repre- sented by the bytes making up the record's length, address, and data fields. The Termination Record Each file ends with a fixed two byte termination record. +-----+------+ |0x53 | 0x38 | +-----+------+ Size Multiplier In general, binary data will expand in sized by approximately 1.2 times when represented with this format. EXAMPLE
Here is an hex-dump example file. It contains the data "Hello, World[rq] to be loaded at address 0. 0000: 53 30 30 33 53 31 10 00 00 48 65 6C 6C 6F 2C 20 S003S1...Hello, 0010: 57 6F 72 6C 64 0A 9D 53 38 World..S8 COPYRIGHT
srec_cat version 1.58 Copyright (C) 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011 Peter Miller The srec_cat program comes with ABSOLUTELY NO WARRANTY; for details use the 'srec_cat -VERSion License' command. This is free software and you are welcome to redistribute it under certain conditions; for details use the 'srec_cat -VERSion License' command. AUTHOR
Peter Miller E-Mail: pmiller@opensource.org.au //* WWW: http://miller.emu.id.au/pmiller/ Reference Manual SRecord srec_stewie(5)
All times are GMT -4. The time now is 12:51 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy