Shell Programming and Scripting

Hi,

I have two files separated by bars at each line

File 1 :

Code:

A|4356|13456|23456
A|4356|2986|98732
A|8765|218|1432567

File 2:

Code:

B|12|13456|1234567
B|11|13456|123456789
B|33|2986|98732
B|11|2986|14578965
B|8765|218|147584

Common field is third field like 13456, 2986 and 218 in both the files

Desired output after merge:

Code:

A|4356|13456|23456
B|12|13456|1234567
B|11|13456|123456789
A|4356|2986|98732
B|33|2986|98732
B|11|2986|14578965
A|8765|218|1432567
B|8765|218|147584

I wrote a code but Its not giving me the desired result.

Code:

rm /dir/new_file.txt
touch /dir/new_file.txt

for i in $(cat /dir/file1.txt);
do

 echo $i >> /dir/new_file.txt

 k=`echo $i |awk -F '|' '{print $3}'`
 for j in $(cat /dir/file2.txt);
 do
 w=`echo $j | awk -F '|' '{print "$3"}'`
 
 if [ "$w" == "$k" ]
 then
 echo $j >> /dir/new_file.txt
 fi
 
 done
done

Script is throwing error at red line, also I am not sure it give me the desired result or not.

Code:

testing_script_for_file.ksh[10]: 13456:  not found.

Please help me in this. Thanks in advance.

Moderator's Comments:

Please use CODE tags

I can't find an error with that line, works for me. Looks like the extraction of 13456 works fine, but then it tries to execute/open it. On your machine, not on mine.

How about

Code:

sort -t\| -k3,3nr file1 file2
A|4356|13456|23456
B|11|13456|123456789
B|12|13456|1234567
A|4356|2986|98732
B|11|2986|14578965
B|33|2986|98732
A|8765|218|1432567
B|8765|218|147584

Thanks RudiC.

I want to do it using loop, could you please help me out in some other way because sorting is not successful every time as data is different each time.

On my machine my code is not working, please assist with some other code having the same functionality. I need it urgently.

Thank you.

RudiC gave you code that meets all of your stated requirements except that you don't like it. In what way does sort fail to meet your needs?

When we see statements like this after a seemingly good solution is proposed, it often means that the submitter is working on a homework assignment and is not allowed to use the tools proposed. Is this a homework assignment?

Are there other requirements that you haven't stated? If there are multiple lines for a given key in the first file, do you really want a copy of every line with that value in the second file duplicated in your new file for each occurrence of that value in the first file?

Is it important that your program run very slowly for medium sized files and run at a snail's pace for large files?

Is it important that your output file maintain the (possibly unsorted) order of lines in the 1st file and for all copies of lines from the 2nd file?

What OS are you using?

The people who read your posts and try to help you find solutions for your problems are volunteers. We are not on your payroll. This problem may be urgent for you, but you can't expect volunteers to ignore other things that they might want to do to satisfy your personal requirements. There are special forums that can be used for "Urgent" requests. In the regular forums requests that need to be responded to urgently are inappropriate. Is this another indication that you have a homework assignment that is due soon and you haven't figured out how to do it yet?

Hi Don,

Nice to hear these words from you. This sorting idea is not working for me as there are 12-14 files that i am trying to merge and each in a specific order. Its not an assignment. Also its my first post here so i don't know the proper functioning of the forum.

Now my code is working fine for me. I got error due to my OS.

Thanks everyone.

There is no magic here. You have a fairly large number of people who are intimately familiar with various aspects of various parts of Linux and UNIX systems. If you give these volunteers details about what you're trying to do, you can frequently get excellent tips on how to get an efficient solution to your problem. If you omit details about constraints on how the output is to be produced, we can all waste a lot of time wandering around paths that won't meet your needs.

Various systems provide various extensions to the standard utilities. If you always tell us what version of what OS you're using, we can avoid suggesting that you use extensions that are not available on your system.

Simply put, help us help you by giving us the information we would need to get the job done right on the system you will be using. Give us incomplete data, get one or more responses that meet all of your stated requirements, and then complain that it doesn't do something you never said was important in the first place, and the volunteers who wasted time trying to help you might not be interested in responding to your next request for help. This is just simple human nature.

Enough generalities...

If each of your input files are sorted in reverse numeric order on the key field as in your example (even if the key field is a different field in different files and has different field separators in different files), it would probably still be a lot faster to have an awk preprocessing step to create a single file with the key field, an added file number, and an added line number as fields from all of your input files; sort it by the key field, file #, and line #; and then use awk or sed to strip the file #, line #, and (if it had to be duplicated) the key fields added by the preprocessing step to just leave the desired sorted file as a result.

This could still be a lot faster than firing up a shell and awk for each line in each of the lines in each of all but the first of your 12 to 14 files multiplied by the number of lines in your 1st file (fork() and exec() are expensive operations).