Sponsored Content
Top Forums UNIX for Advanced & Expert Users Solution for the Massive Comparison Operation Post 302428527 by raghav288 on Thursday 10th of June 2010 07:49:18 AM
Old 06-10-2010
Solution for the Massive Comparison Operation

Hi

We have 50 million records in mainframes DB2. We have a requirement to Record the Change Data Capture(CDC) records.

i.e New Records or Updated Records that were added into the DB2.
Unfortunately we dont have any column indicators to give the details of the changes made to the records.

So we decided to import the same into flat files into UNIX box everyday and then compare the previous day's file and get the changed rows using UNIX text processing functionalities.

The problem is everyday huge data changes happen approx 40 million

So can anyone give me a solution for better handling of the same or even the hardware requirement for UNIX Server for faster processing of the huge data comparison.

Thanks
 

5 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Looking for AWK Solution for column comparison in a single file

- I am looking for different kind of awk solution which I don't think is mentioned before in these forums. Number of rows in the file are fixed Their are two columns in file1.txt 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 I am looking for 3... (1 Reply)
Discussion started by: softwarekids23
1 Replies

2. Shell Programming and Scripting

Column operation : cosne and sine operation

I have a txt file with several columns and i want to peform an operation on two columns and output it to a new txt file . file.txt 900.00000 1 1 1 500.00000 500.00000 100000.000 4 4 1.45257346E-07 899.10834 ... (4 Replies)
Discussion started by: shashi792
4 Replies

3. Homework & Coursework Questions

having massive trouble with 5 questions about egrep!

Hi all! I need help to do a few things with a .txt file using egrep. 1. I need to list all sequences where the vowel letters 'a, e, i, o, u' occur in that order, possibly separated by characters other than a, e, i, o, u; consisting of one or more complete words, possibly including punctuation. ... (1 Reply)
Discussion started by: dindiqotu
1 Replies

4. Shell Programming and Scripting

Massive Copy With Base Directory

I have a script that I am using to copy around 40-70k files to a NFS NAS. I have posted my code below in hopes that someone can help me figure out a faster way of achieving this. At the end of the script i need to have all the files in the list, copied over to the nas with source directory... (8 Replies)
Discussion started by: nitrobass24
8 Replies

5. Shell Programming and Scripting

Massive ftp

friends good morning FTP works perfect but I have a doubt if I want to transport 10 files, I imagine that I should not open 10 connections as I can transfer more than 1 file? ftp -n <<!EOF open caburga user ephfact ephfact cd /users/efactura/docONE/entrada bin mput EPH`date... (16 Replies)
Discussion started by: tricampeon81
16 Replies
srec_motorola(5)						File Formats Manual						  srec_motorola(5)

NAME
srec_motorola - Motorola S-Record hexadecimal file format DESCRIPTION
This format is also known as the Exorciser, Exormacs or Exormax format. Motorola's S-record format allows binary files to be uploaded and downloaded between two computer systems. This type of format is widely used when transferring programs and data between a computer system (such as a PC, Macintosh, or workstation) and an emulator or evaluation board for Motorola microcontrollers and microprocessors. The Lines Most S-Record file contain only S-Record lines (see the next section), which always start with a capital S character. Some systems gener- ate various "extensions[rq] which usually manifest as lines which start with something else. These "extension[rq] lines may or may not break other systems made by other vendors. Caveat emptor. The Fields The S-Record format consists of 5 fields. These are the type field, length field, address field, data field, and the checksum. The lines always start with a capital S character. +--+------+---------------+---------+------+----------+ |S | Type | Record Length | Address | Data | Checksum | +--+------+---------------+---------+------+----------+ Type The type field is a 1 character field that specifies whether the record is an S0, S1, S2, S3, S5, S6, S7, S8 or S9 field. Record Length The record length field is a 2 character (1 byte) field that specifies the number of character pairs (bytes) in the record, exclud- ing the type and record length fields. Address This is a 2-, 3- or 4-byte address that specifies where the data in the S-Record is to be loaded into memory. Data The data field contains the executable code, memory-loadable data or descriptive information to be transferred. Checksum The checksum is an 8-bit field that represents the least significant byte of the one's complement of the sum of the values repre- sented by the pairs of characters making up the record's length, address, and data fields. Record Types S0 This type of record is the header record for each block of S-Records. The data field may contain any descriptive information iden- tifying the following block of S-Records. (It is commonly "HDR[rq] on many systems.) The address field is normally zero. S1 A record containing data and the 2-byte address at which the data is to reside. S2 A record containing data and the 3-byte address at which the data is to reside. S3 A record containing data and the 4-byte address at which the data is to reside. S5 An optional record containing the number of S1, S2 and S3 records transmitted in a particular block. The count appears in the two- byte address field. There is no data field. This record is optional, you do not have to use it. Nobody knows if you can have more than one in a file; and if you do, nobody knows whether or not the line count resets after each one. The srec_cat command will only ever use one, provided the number of lines fits in 16 bits, otherwise it will use S6. S6 An optional record containing the number of S1, S2 and S3 records transmitted in a particular block. The count appears in the three-byte address field. There is no data field. This record is optional, you do not have to use it. Nobody knows if you can have more than one in a file; and if you do, nobody knows whether or not the line count resets after each one. Nobody knows what happens if you mix S5 and S6 records in a file. The srec_cat command will only ever use one, provided the number of lines fits in 24 bits. S7 A termination record for a block of S3 records. The address field may contain the 4-byte address of the instruction to which control is passed. There is no data field. S8 A termination record for a block of S2 records. The address field may optionally contain the 3-byte address of the instruction to which control is passed. There is no data field. S9 A termination record for a block of S1 records. The address field may optionally contain the 2-byte address of the instruction to which control is passed. If not specified, the first entry point specification encountered in the object module input will be used. There is no data field. Size Multiplier In general, binary data will expand in sized by approximately 2.4 times when represented with this format. EXAMPLE
Here is an example S-Record file. It contains the data "Hello, World[rq] to be loaded at address 0. S00600004844521B S110000048656C6C6F2C20576F726C640A9D S5030001FB S9030000FC COPYRIGHT
srec_cat version 1.58 Copyright (C) 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011 Peter Miller The srec_cat program comes with ABSOLUTELY NO WARRANTY; for details use the 'srec_cat -VERSion License' command. This is free software and you are welcome to redistribute it under certain conditions; for details use the 'srec_cat -VERSion License' command. AUTHOR
Peter Miller E-Mail: pmiller@opensource.org.au //* WWW: http://miller.emu.id.au/pmiller/ Reference Manual SRecord srec_motorola(5)
All times are GMT -4. The time now is 11:15 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy