Challenging Compare and validate question -- plus speed.


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Challenging Compare and validate question -- plus speed.
# 8  
Old 05-22-2006
Thank you very much...It did work perfectly...

I would like to give the script a .ksh extension since I have #!/usr/bin/ksh

Thanks again for all the help..Please advice...
# 9  
Old 05-22-2006
Code:
#!/usr/bin/ksh

nawk '
  FNR==NR{
     detail[$2]
     next
  }
  { 
    printf("Meta [%s] %s found in Detail-- %s\n",  $2, ($2 in detail) ? "" : "NOT",  ($2 in detail) ? "Successful" : "Failed")
  }' DetailFile.txt MetadataFile.txt

# 10  
Old 05-22-2006
Thanks again....I did follow your code and added it into the .ksh script.

PHP Code:
nawk '
  FNR==NR{
     detail[$2]
     next
  }
  {
    printf("Metadata Partner Name [%s] %s found in Detail File-- %s\n",  $2, ($2 in detail) ? "" : "NOT",  ($2 in detail) ? "Successful" : "Failed")
  }' 
${DETAIL_FILE} ${METADATA_FILE}

RC=$?

if [ 
$RC -ne 0 ]
then 
   
echo "*** Comparison Failed. Aborting Script... ***"
   
exit $RC
else
   echo 
"*** Comparison Completed ***"
   
echo "*** Partner Files compared Successfully ***"
fi 
It is not aborting though it did not find the name...
PHP Code:
        Detail File file2 to be compared found
Metadata Partner Name 
[ORBITZ]  found in Detail File-- Successful
Metadata Partner Name 
[AIRTRAN]  found in Detail File-- Successful
Metadata Partner Name 
[FRONTIER]  found in Detail File-- Successful
Metadata Partner Name 
[CAESAR]  found in Detail File-- Successful
Metadata Partner Name 
[BESTNOT found in Detail File-- Failed

*** Comparison Completed ***
*** 
Partner Files compared Successfully *** 
Please advice how to abort the flow if it failed to find.
# 11  
Old 05-22-2006
Code:
#!/usr/bin/ksh

nawk '
  FNR==NR{
     detail[$2]
     next
  }
  { 
     if ( $2 in detail)
        printf("Meta [%s] found in Detail-- Succefull\n",  $2)
     else {
        printf("Meta [%s] NOT found in Detail-- Failed\n",  $2)
        _ex=1
     }
  }
  END { exit(_ex)}' DetailFile.txt MetadataFile.txt

# 12  
Old 05-23-2006
Thank You vgersh99...

I have tested the script on a file that has 13 million records and it took 2.5 minutes.

Just another quick question...

Is there any way to enhance the script?

For example: The Metadata file has name 'AIRTRAN AIRWAYS' but in the Detail File it is
listed as 'AIRTRAN'. Can we make this a pass rather than failure?

When comparing the Metadata names with Detail record names, it should pass on these
conditions.

Can we create a control file like:

AIRTRAN: AIRTRAN AIRWAYS, AIRTRAN, AIRTRAN AIR
MIDWEST: MIDWEST AIRLINES, MIDWEST AIR

and look this up and pass the script....If it doesn't find anything related, the script should be aborted.

I am not sure how I can do this....If you have any idea, please let me know.
# 13  
Old 05-23-2006
Quote:
Originally Posted by madhunk
Thank You vgersh99...

I have tested the script on a file that has 13 million records and it took 2.5 minutes.

Just another quick question...

Is there any way to enhance the script?

For example: The Metadata file has name 'AIRTRAN AIRWAYS' but in the Detail File it is
listed as 'AIRTRAN'. Can we make this a pass rather than failure?

When comparing the Metadata names with Detail record names, it should pass on these
conditions.

Can we create a control file like:

AIRTRAN: AIRTRAN AIRWAYS, AIRTRAN, AIRTRAN AIR
MIDWEST: MIDWEST AIRLINES, MIDWEST AIR

and look this up and pass the script....If it doesn't find anything related, the script should be aborted.

I am not sure how I can do this....If you have any idea, please let me know.

The above works fine as long as the name in the METAfile starts with same name as it appears in the DETAIL file: 'AIRTRAN AIRWAYS' in metaFile; 'AIRTRAN' in detailFile.
# 14  
Old 05-23-2006
Thank You again vgersh99...

I did create two test files and it works if we have an example like that. It doesn't work if there is no space between AIRTRAN and AIRWAYS.
Example: AIRTRANAIRWAYS.

The files we get are really really bad. I am a little scared incase I get a Metadata file with AIRTRANAIRWAYS.

Below is the final script that I have...

Code:
#| check for correct number of parameters

if [ $# -ne 3 ]
then
   echo " "
   echo " Incorrect number of parameters entered..."
   echo " Correct usage: " $0 "<DIR> <DETAIL FILE> <METADATA FILE>"
   echo " "
   exit 1
fi

#-------------------------------------------------------------
#    Initialize variables
#-------------------------------------------------------------

DIR=$1
DETAIL_FILE=$2
METADATA_FILE=$3

#-------------------------------------------------------------
#    Check for the existence of the Detail and Metadata files
#-------------------------------------------------------------
cd ${DIR}

if [ -r ${DETAIL_FILE} ]; then
   echo "\tDetail File ${DETAIL_FILE} to be compared found"
else
   echo "\tError: Detail File ${DETAIL_FILE} was not found, Aborting!"
   echo " "
   exit 1
fi

if [ -r ${METADATA_FILE} ]; then
   echo "\tMetadata File ${METADATA_FILE} to be compared found"
else
   echo "\tError: Metadata File ${METADATA_FILE} was not found, Aborting!"
   echo " "
   exit 1
fi

#-------------------------------------------------------------
#    Compare both files for partner names
#-------------------------------------------------------------
time {
nawk '
  FNR==NR{
     detail[$2]
     next
  }
  { 
     if ( $2 in detail)
        printf("Metadata partner name [%s] found in Detail-- Successful\n",  $2)
     else {
        printf("Metadata partner name [%s] NOT found in Detail-- Failed\n",  $2)
        _ex=1
     }
  }
  END { exit(_ex)}' ${DETAIL_FILE} ${METADATA_FILE}  
}

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk to compare files and validate order of headers

The below awk verifies the count and order of each text file in the directory. The script does execute and produce output, however the order of the headers are not compared to key. The portion in bold is supposed to do that. If the order of the headers in each text file is the same as key, then... (0 Replies)
Discussion started by: cmccabe
0 Replies

2. Programming

Generic speed question (PHP vs.)

Hi, On a hosted linux environment which I have very little control over, I have a PHP script that takes in X number of floats, performs Y number of simple recursive arithmetic calculations, and produces some output for display to the user. When I first created the script, 'X' and 'Y' were... (4 Replies)
Discussion started by: MoreCowbell
4 Replies

3. Shell Programming and Scripting

Challenging scenario

Hi, My input file contains 1,2 2,4 3,6 4,9 9,10 My expected output is 1,10 2,10 3,6 4,1 9,10 (6 Replies)
Discussion started by: pandeesh
6 Replies

4. Shell Programming and Scripting

Another validate input Question.

I'm writing a bash shell script to 'help' me post to susepaste (I can NEVER remember the time options). Here's the code: #!/bin/bash ########## # # Project : personal script. # Started : Wed Aug 03, 2011 # Author : Habitual # Description : susepaste c-li script with user... (5 Replies)
Discussion started by: Habitual
5 Replies

5. Filesystems, Disks and Memory

data from blktrace: read speed V.S. write speed

I analysed disk performance with blktrace and get some data: read: 8,3 4 2141 2.882115217 3342 Q R 195732187 + 32 8,3 4 2142 2.882116411 3342 G R 195732187 + 32 8,3 4 2144 2.882117647 3342 I R 195732187 + 32 8,3 4 2145 ... (1 Reply)
Discussion started by: W.C.C
1 Replies

6. Shell Programming and Scripting

Need help with this challenging code....

Hello All, I am new to this forum, and the reason I came here is to seek solution from the experts. I have written following wrapper script, it was running fine from past couple of months, until last week. When one of the function in the script which suppose to login through ssh to the... (2 Replies)
Discussion started by: tajdar
2 Replies

7. Shell Programming and Scripting

Compare files question

Hi all, How do I compare contents of entire two files except for the first line is each of them? I am sure first lines from both my files are going to be different so I want to ignore them. Is there a easier way than creating temporary files by cutting out the first line and then comparing... (1 Reply)
Discussion started by: jakSun8
1 Replies

8. Shell Programming and Scripting

Challenging!! Help needed

Hi, I have a script xyz.ksh which accpets two parameters the format of first one is :X_TABLENAME_Y and second one is a digit. I can extract a table name from that parameter and store it in a variable var_tblnm, so if i pass a parameter X_TABLE1_Y the value in var_tblenm is "TABLE1" now i have... (1 Reply)
Discussion started by: hcdiss
1 Replies

9. Filesystems, Disks and Memory

dmidecode, RAM speed = "Current Speed: Unknown"

Hello, I have a Supermicro server with a P4SCI mother board running Debian Sarge 3.1. This is the "dmidecode" output related to RAM info: RAM speed information is incomplete.. "Current Speed: Unknown", is there anyway/soft to get the speed of installed RAM modules? thanks!! Regards :)... (0 Replies)
Discussion started by: Santi
0 Replies

10. UNIX for Advanced & Expert Users

Very Challenging Question! Need help bad!

I am in desperate need of an answer to this question. I have looked everywhere (even the man pages) and found very little. Solaris has the concept of "plumbing" a network interface. What does this mean? I would be really greatful to whoever could help me answer this question. I am so... (1 Reply)
Discussion started by: Sparticus007
1 Replies
Login or Register to Ask a Question