I used multiple array which could be avoided. but since your look-up file is not so-big, its worth trying.
I see you files are not at one place, those needs to be found.
I don't have a real system with me so can't test the performance on BIG files.
However, There could be surely other more efficient ways which might come-up soon.
Hi clx, thank you very much for taking your time with this.
I have as well developed similar awk codes to what you have. Posting it as it might help others as well:
What the code does is to join records from the file to the matching records from the lookup.
So columns 2/3 from the lookup is now column 9/10 of the joined file. And since all the dates that i needed are already in 1 file, it was easier for awk to compare them. (I took away the adding of the last column as well since it was already available upon joining the records)
This took away a big chunk of code from my script and was able to breeze through 30 million lines in around 10 minutes, which is already tolerable.
Again, thanks!
Last edited by agentgrecko; 07-10-2012 at 04:55 AM..
This User Gave Thanks to agentgrecko For This Post:
I have sifted through many past posts but have not found a good solution.
I have a 2 files that hold a filename and its cksum value like this:
FILE1 contents:
45435347589 filename1
FILE1 is from the directory structure of one host and FILE2 is generated from a rsh: host2 cksum command.
I... (1 Reply)
1. The problem statement, all variables and given/known data:
I have standard web server log file. It contains different columns (like IP address, request result code, request type etc) including a date column with the format .
I have developed a log analysis command line utility that displays... (1 Reply)
Hi folks,
I'm trying to do a bit of extra monitoring on who is sending mail through our server and as I'm sure you'll understand, the log files aren't exactly small.
As a note, I'm working to sort out a solution to this myself, so I'll keep editing the post when I get a chance.
Here an example... (2 Replies)
I get a file which has all its content in a single row.
The file contains xml data containing 3000 records, but all in a single row, making it difficult for Unix to Process the file.
I decided to insert a new line character at all occurrences of a particular string in this file (say replacing... (4 Replies)
Hi All,
I'm trying to figure out which are the trusted-ips and which are not using a script file.. I have a file named 'ip-list.txt' which contains some ip addresses and another file named 'trusted-ip-list.txt' which also contains some ip addresses. I want to read a line from... (4 Replies)
Hi,
I wanted to check whether the x,y,z coordinates of two files are equal or not. At times, when one file is converted to another suitable file extension , there are some chances that the data mismatch would happen during the conversion. In order to avoid the data misfit, i would like to... (6 Replies)
Hi,
i have a scenario were i should compare a few fields from each line then increment a variable based on that.
Example file
989878|8999|Y|0|Y|N|V
989878|8999|Y|0|N|N|V
989878|8999|Y|2344|Y|N|V
i have 3 conditions to check and increment a variable on every line
condition 1 if ( $3... (4 Replies)
Greetings Experts,
I need to handle the views created over monthly retention tables
for which every new table in YYYYMMDD format, there is
equivalent view created and the older table which might be
dropped, the view over it has to be re-created over a dummy
table so that it doesn't fail.... (2 Replies)