Thanks Cherry! I do not know perl so this is kind of out of scope for me but I heard perl is very fast. I have to learn that in future.
The below awk code runs fine for small number of records but now I'm running it on 200K records and it's taking lot of time, it's been 25 minutes and it wrote just 50 records in to the output file. Not sure how much more time it will take to complete the process.
Is there any thing wrong with the code that is making it to run long time?
Generally awk is very fast, right?
Actually this code is already availble in C++ and I'm trying to re-write in awk because of performance issues as awk is faster.
awk -f king.awk FS=, file1 FS='#KEY=' file2
---------- Post updated at 05:00 PM ---------- Previous update was at 11:37 AM ----------
My bad.. I used wrong data that has only 7 unique records and rest all of it is duplicate which will not happen in real world. So, I'm good.
For all 200K unique records and 1 duplicate for each record, it ran in ~3 mins.
Thanks for all your support!
Last edited by King Kalyan; 06-18-2009 at 12:43 PM..
I'm not sure why you changed the invocation:
TO
and do the 'index/substr' for each record/line in file2. It's definitely adding time to the execution.
Also if you take my last version - it should be a little faster as I don't rebuild the array if I just have 1 entry in it (probably the majority of your records in file2).
You could probably think of a different implementation that doesn't require rebuilding the array all together. This is left as an exercise for the OP
I changed the invocation because field <#KEY> can be anywhere in the file it's not always at second position. Sorry I did not mention that in my first post.
Yes I saw your last code and forgot to include that in my code, now I included it (skipping rebuilding array if there is only one entry).
After you told that, I thought of different implementation and here it is.. this is more faster...
I have files named with different prefixes. From each I want to extract the first line containing a specific string, and then print that line along with the prefix.
I've tried to do this with a while loop, but instead of printing the prefix I print the first line of the file twice.
Files:... (3 Replies)
Hi guys,
I have a text file named file1.txt that is formatted like this:
001 , ID , 20000
002 , Name , Brandon
003 , Phone_Number , 616-234-1999
004 , SSNumber , 234-23-234
005 , Model , Toyota
007 , Engine ,V8
008 , GPS , OFF
and I have file2.txt formatted like this:
... (2 Replies)
In the below bash a file is downloaded when the program is opened and then that file is searched based on user input and the result is written to a new file.
For example, the bash is opened and the download.txt is downloaded, the user then enters the id (NA04520). The id is used to search... (5 Replies)
Hi Forum.
I did some google search on what I'm trying to do but I cannot get my code to work correctly. I have 2 files which are very large and I want to read text from file1 and search in file2 - if present, keep the records.
I've tried fgrep -f file1 file2 but it is too slow.
File1:... (10 Replies)
Hello folks, I have another doozy. I have two files. The first file has four fields in it. These four fields map to different locations in my second file. What I want to do is read the master file (file 2 - 23 fields) and compare each line against each record in file 1. If I get a match in all four... (4 Replies)
Hi Frnds...
I have an input file name.txt and another file named as source.. name.txt is having only one column and source is having around 25 columns...i need to read from name.txt line by line and search it in source file and then save the result in results file..
I have a rough idea about the... (15 Replies)
Hi there, I have a text file with several colums separated by "|;#" I need to search the file extracting all columns starting with the value of "1" or "2" saving in a separate file just the first 7 columns of each row maching the criteria, with replacement of the saparators in the nearly created... (4 Replies)
Hi Experts.
I need to list the file and the filename comes from the file ListOfFile.txt.
Basicly I have a filename "ListOfFile.txt" and it contain
Example of ListOfFile.txt
/home/Dave/Program/Tran1.P
/home/Dave/Program/Tran2.P
/home/Dave/Program/Tran3.P
/home/Dave/Program/Tran4.P... (7 Replies)
Hi All,
Please help me in creating files through K-shell scripts.
I am having one file in this format.
OWNER.TABLE_NAME
OWNER.TABLE_NAME1
OWNER1.TABLE_NAME
OWNER1.TABLE_NAME1
I want to read the above file and create new file through k shell script.
The new file should looks like this.... (4 Replies)