Sponsored Content
Top Forums Shell Programming and Scripting Faster Line by Line String/Date Comparison of 2 Files Post 302668305 by agentgrecko on Monday 9th of July 2012 05:52:53 AM
Old 07-09-2012
Faster Line by Line String/Date Comparison of 2 Files

Hello,

I was wondering if anyone knows a faster way to search and compare strings and dates from 2 files?
I'm currently using "for loop" but seems sluggish as i have to cycle through 10 directories with 10 files each containing thousands of lines.


Given:
Code:
-10 directories
-10 files (tab delimited)
-1 lookup file
-1 report file

Heres what i'm trying to achieve:

Code:
1. loop through the 10 dirs, with 10 files
2. for every line read from file
   a. get column 1 string and grep it from lookup file, write output to result file
   b. get column 3 date and store as vardate  
   c. for every line of result file from 2.a
      c.I get column 2 date as varstart
      c.II get column 3 date as varend
      c.III get column 7 as lastcol
      c.IV check if vardate from 2.b is between varstart and varend    
      c.V if vardate is between varstart and varend, write line from 2. + vartstart + varend  + lastcol to report file

Here is my straightforward solution so far, though its working fine, its not that elegant and fast:

Code:
for loop through the 10 dirs
do
   for loop through 10 files
   do
      while read LINE of file
       do
            grep column 1 from lookup file > result file
            vardate=column 3
            while read ROW of result file
            do
               varstart=column 2
               varend=column 3
               lastcol=column 7
               if [[ varstart -le vardate && varend -ge vardate ]]
               then
                  printf "LINE\tvarstart\tvarend\tlastcol\n" >> report file
               else
                  :
               fi
            done
       done
   done
done

I was trying to replace the if statement with the following (as i have learned through research, awk is much faster in line by line processing):

Code:
awk '{varstart=$2; varend=$3; lastcol=$7; getline;} varstart <= vardate && varend >= vardate {print l,varstart,varend,lastcol}' l=${LINE} vardate=column3 resultfile >> reportfile

But cant seem to get it to work properly. Plus, i was thinking if i can use awk as well on the parent while loop to be more efficient, but i have no idea anymore if its possible to use another awk, within an awk statement.

My goal is just to make this script work faster. Any suggestions or alternate approaches is well appreciated.

Thank you very much guys. Smilie

Last edited by agentgrecko; 07-09-2012 at 07:07 AM..
 

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

line by line file comparison

I have sifted through many past posts but have not found a good solution. I have a 2 files that hold a filename and its cksum value like this: FILE1 contents: 45435347589 filename1 FILE1 is from the directory structure of one host and FILE2 is generated from a rsh: host2 cksum command. I... (1 Reply)
Discussion started by: newbreed1
1 Replies

2. Homework & Coursework Questions

Date comparison with 'string date having slashes and time zone' in Bash only

1. The problem statement, all variables and given/known data: I have standard web server log file. It contains different columns (like IP address, request result code, request type etc) including a date column with the format . I have developed a log analysis command line utility that displays... (1 Reply)
Discussion started by: TariqYousaf
1 Replies

3. Shell Programming and Scripting

String comparison: All lines in file but each within line

Hi folks, I'm trying to do a bit of extra monitoring on who is sending mail through our server and as I'm sure you'll understand, the log files aren't exactly small. As a note, I'm working to sort out a solution to this myself, so I'll keep editing the post when I get a chance. Here an example... (2 Replies)
Discussion started by: beddo
2 Replies

4. Solaris

Line too long error Replace string with new line line character

I get a file which has all its content in a single row. The file contains xml data containing 3000 records, but all in a single row, making it difficult for Unix to Process the file. I decided to insert a new line character at all occurrences of a particular string in this file (say replacing... (4 Replies)
Discussion started by: ducati
4 Replies

5. Shell Programming and Scripting

how to read the contents of two files line by line and compare the line by line?

Hi All, I'm trying to figure out which are the trusted-ips and which are not using a script file.. I have a file named 'ip-list.txt' which contains some ip addresses and another file named 'trusted-ip-list.txt' which also contains some ip addresses. I want to read a line from... (4 Replies)
Discussion started by: mjavalkar
4 Replies

6. Shell Programming and Scripting

Need help in column comparison & adding extra line to files

Hi, I wanted to check whether the x,y,z coordinates of two files are equal or not. At times, when one file is converted to another suitable file extension , there are some chances that the data mismatch would happen during the conversion. In order to avoid the data misfit, i would like to... (6 Replies)
Discussion started by: b@l@ji
6 Replies

7. Shell Programming and Scripting

Comparison of fields then increment a counter reading line by line in a file

Hi, i have a scenario were i should compare a few fields from each line then increment a variable based on that. Example file 989878|8999|Y|0|Y|N|V 989878|8999|Y|0|N|N|V 989878|8999|Y|2344|Y|N|V i have 3 conditions to check and increment a variable on every line condition 1 if ( $3... (4 Replies)
Discussion started by: selvankj
4 Replies

8. Shell Programming and Scripting

Add Date string at end of line

HOWTO Append Date String (MMYYYY) to End of Line file A.txt B.txt Output: A_052014.txt B_052014.txt (9 Replies)
Discussion started by: satish1222
9 Replies

9. Shell Programming and Scripting

awk - 2 files comparison without for loop - multi-line issue

Greetings Experts, I need to handle the views created over monthly retention tables for which every new table in YYYYMMDD format, there is equivalent view created and the older table which might be dropped, the view over it has to be re-created over a dummy table so that it doesn't fail.... (2 Replies)
Discussion started by: chill3chee
2 Replies
All times are GMT -4. The time now is 02:49 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy