Can you please help me in resolving the following problem?
My requirement is like this:
1) I have two files YESTERDAY_FILE and TODAY_FILE. Each one is having nearly two million data.
2) I need to check each record of TODAY_FILE in YESTERDAY_FILE. If exists we can skip that by treating as duplicate.
3) If does not exists, I need to check the primary key fields, say for example first 3 fields, if match is found i need check the data part, say for example 5th and 6th fields, if datapart is not matching, then i need to write the record in a new file, say OUTPUT_FILE, with a prefix of 'C' (Change). If datapart is matching skip the record.
4) If Primary Key fields match not found then i need to write the record with a prefix of A (Append).
4) After above process, i need to check each record of YESTERDAY_FILE in TODAY_FILE, if does not exists, i need to write the record with a prefix of D (Delete).
I developed the following logic which is taking too much time to execute...in one minute it is creating 100 records. Performance is too bad. Can any one of you please help me out.
My code is:
fields_to_compare, fields_to_skip and file_name are the parameters passed to the script. In the following case:
fields_to_compare=1 (primary Key fields: aaa, bbb etc)
fields_to_skip =3 (From 4th field i need to consider as data part)
file_name=today_file
I've noticed that for simple things, I can shell script something and it works fine. When things start getting complicated or there's a performance issue, I'll break out perl (or python if you like).
I'd take what you have and see if it could be done better in perl. I'm sure it'd be a lot faster and probably easier to write.
I won't even try to interpret your code, because it is too difficult to read without code tags, but with that in mind, have you considered using comm, diff, cmp, etc.? Unix apps such as these were designed for problems like this. Why reinvent the wheel?
I'm also in agreement with BOFH that if raw performance is your concern, perl (or even C) might be better.
If I could read your code, I'd probably have some more insight.
Look - tmarikle posted a nice little 5 line awk program that does most of what you want - print all the lines in file2 that do not exist in file1. A sort of "minus" in a result set sense.
This is a modifed version of it, change it as you want:
The key is the first three fields. Add more fields or just use $0 for the whole record.
I'm trying to remove duplicate data from an input file with unsorted data which is of size >50GB and write the unique records to a new file.
I'm trying and already tried out a variety of options posted in similar threads/forums. But no luck so far..
Any suggestions please ?
Thanks !! (9 Replies)
What do i need to do have the below perl program load 205 million record files into the hash. It currently works on smaller files, but not working on huge files. Any idea what i need to do to modify to make it work with huge files:
#!/usr/bin/perl
$ot1=$ARGV;
$ot2=$ARGV;
open(mfileot1,... (12 Replies)
I have 2 files; one file (say, details.txt) contains the details of employees and another file (say, emp.txt) has some selected employee names. I am extracting employee details from details.txt by using emp.txt and the corresponding code is:
while read line
do
emp_name=`echo $line`
grep -e... (7 Replies)
I have written a virtual HBA driver named "xmp_vhba". A scsi disk is attached on it. as shown below:
xmp_vhba, instance #0
disk, instance #11
But the performance became very bad when we read/write the scsi disk using the vdbench(a read/write io tool).
What is the reason? ... (7 Replies)
Hi all,
I have two large files and i want a field by field comparison for each record in it.
All fields are tab seperated.
file1:
Email SELVAKUMAR RAMACHANDRAN
Email SHILPA SAHU
Web NIYATI SONI
Web NIYATI SONI
Email VIINII DOSHI
Web RAJNISH KUMAR
Web ... (4 Replies)
There might be some problem with my server,
because every morning at 7, it's performance become bad with no DB extra deadlock.
But I just couldn't figure it out.
Please give me some advise, thanks a lot...
According to the CPU performace chart, Daily CPU loading Maximum: 42 %, Average:36%.
... (8 Replies)
Hi,
I have two files file A and File B. File A is a error file and File B is source file. In the error file. First line is the actual error and second line gives the information about the record (client ID) that throws error. I need to compare the first field (which doesnt start with '//') of... (11 Replies)
Hello guys!
I'm n00b in AIX and I'm sticked in a problem. (my English is poor enough, but I hope you can understand me :P). So.. I'm trying to connect to an AIX machine with putty, and .. 'using username xxx' appears after 2 sec (OK), but 'xxx@ip's password' appears after 1:15 min. After... (6 Replies)