Shell Programming and Scripting

Hi,

I have two files file A and File B. File A is a error file and File B is source file. In the error file. First line is the actual error and second line gives the information about the record (client ID) that throws error. I need to compare the first field (which doesnt start with '//') of file A with fifth field of file B. It field values in file A and file B matches i need to write it to output file as below.

File A
// 223 missing
223,Jan,ee,bla,bla

// data not found
254-11,Jan,ee,bla,bla

// data rejected
214-1,Jan,ee,bla,bla

File B
aaaa,bbbb,ccc,dddd,20054-11,fff,ggg...
aaaa,bbbb,ccc,dddd,254-11,fff,ggg...
aaaa,bbbb,ccc,dddd,2545456-1,fff,ggg...

output:
// data not found
254-11,Jan,ee,bla,bla

if First field of File A and Fifth field of File B (254-11) matches, then i need to write the records from file A (current line and the previous line) to a output file as above.

I could achieve it very easily using awk and grep with if loop. Problem is files are hugh. Nearly 1 million records are in both the files. script run for 3-4 hours. I would appreciate if some one could help me in giving good logic or better script which could complete the task in few minutes.

Note: File A and File B look exactly in the same format. Caution about the blanks in file A and Client ID fomat 000 or 000-0 or 000-00.

for compare:
comm file1 file2

for diffrence:
diff file1 file2

Hi Rahul,

Thank you for your reply. Command comm with the above files gives wrong output because i need to compare field 1 of file A and field 5 of file B and output the current and previous line from file A.

comm command will compare the files line by line. None of the lines will match except field 1 of file A and field 5 of file B.

Regards,
Mahesh k

By chance, I came on here with exactly the same problem.

I think that join may come in useful here:
> join -1 1 -2 5 -t, $fileA $fileB > $requiredFile

The thing is the files need to be sorted by the fields you plan to join... and I can't sort my files :-s.

Don't know if this is of any use.

Hi,

Thank you for your suggestion. In file A, Actual field is second line. Records seperated by ','. The first line is error or message where words are sepearated by space. so i cannot say first field because the fields are not proper.

Regards,
Mahesh K

can you extract the required identifiers into a different file with awk and/or grep?

yes buddy, i can do that. I have extracted the field 1 to file C and Field 5 to file D. I used comm command to compare and find the exact match. It just run for few seconds.

The output file is also quite big. I count number of lines in output file, put the while loop, get the ID from output file 1 by 1 and grep the File a and generate the exact output.

This is my problem. the above task run for 2-3 hours due to big loop. I dont know how to over come this problem to optimize my script.