Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Merge of two input file by search

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Merge of two input file by search
# 1  
Old 01-07-2012
Merge of two input file by search
Hi

i am running a issue with the way i handel open file in perl

i have the following input file <File1>

Code:
D33963|BNS Default Swap|-261564.923909249|
D24484|BNS Default Swap|-53356.6868058492|
D24485|BNS Default Swap|-21180.9904679111|
D33965|BNS Default Swap|154181.478745804|
D24486|BNS Default Swap|-47413.0013193052|
D33966|BNS Default Swap|-154181.478745804|
D24487|BNS Default Swap|-63253.9807711966|
D33968|BNS Default Swap|-160521.81007754|
D24489|BNS Default Swap|-10584.4665849774|
S85801|BNS Swap|451309.300774646|
D33969|BNS Default Swap|118166.419991555|

i would like too read the full file line by line, extract the first field (ex in the first case D33963)

search for that value in an other file <file2> that king of look like this
(it is a big file with me specific order)

Code:
:E00277,48089,,,Trading,FALSE,,CAISSE,19189,AA,CAD
:D24485,48085,,,Trading,FALSE,,CASSE,19139,AA,CAD
:D2448,48083,,,Trading,FALSE,,CAIE,19029,AA,CAD
:D33963,48082,,,Trading,FALSE,,CAISSE,19149,AA,CAD,
:E00286CAP,48082,,,Trading,FALSE,,CAISSE,19149,AA,CAD

then when i find the value in <file2> i would line the full line of <file1> and full line of file2 to be merge and writen in a output file

this is the code i come with, but for some raison it look like my <file2> is read only once, and start the second search from the point where it found the previous (so forgeting the begining of the file)




Code:
#!/bin/perl

open(F1,"<@ARGV[0]");
open(F2,"@ARGV[1]");
open (OUTFILE,">$ARGV[2]");

sub run () {
        while(<F1>)
        {
                chomp;
                $line = $_;
                @ARA_NAME = split(/\|/,$line);
                find_ara_name()

        }
close(F1);
close(F2);
}


sub find_ara_name() {
while (<F2>){
                chomp;
                $line1 = $_;
                        if ($line1 =~ m/:@ARA_NAME[0],/) {
                        print "found @ARA_NAME[0]\n";
                        $line1 =~ s/,/|/g;
                        print OUTFILE "$line$line1\n";
                        return;
                        }
                }
}



#main
run()

this is the output i would like for every record of <file1> if no match i will later write a other output file to track those

Code:
D33963|BNS Default Swap|-261564.923909249|:D33963|48082|||Trading|FALSE||CAISSE|19149|AA|CAD|

thanks for your help in advance
# 2  
Old 01-07-2012
Hi kykyboss,

One way:
Code:
$ cat file1
D33963|BNS Default Swap|-261564.923909249|
D24484|BNS Default Swap|-53356.6868058492|
D24485|BNS Default Swap|-21180.9904679111|
D33965|BNS Default Swap|154181.478745804|
D24486|BNS Default Swap|-47413.0013193052|
D33966|BNS Default Swap|-154181.478745804|
D24487|BNS Default Swap|-63253.9807711966|
D33968|BNS Default Swap|-160521.81007754|
D24489|BNS Default Swap|-10584.4665849774|
S85801|BNS Swap|451309.300774646|
D33969|BNS Default Swap|118166.419991555|
$ cat file2
:E00277,48089,,,Trading,FALSE,,CAISSE,19189,AA,CAD
:D24485,48085,,,Trading,FALSE,,CASSE,19139,AA,CAD
:D2448,48083,,,Trading,FALSE,,CAIE,19029,AA,CAD
:D33963,48082,,,Trading,FALSE,,CAISSE,19149,AA,CAD,
:E00286CAP,48082,,,Trading,FALSE,,CAISSE,19149,AA,CAD
$ cat script.pl
use warnings;
use strict;

die qq[Usage: perl $0 <file1> <file2>\n] unless @ARGV == 2;

open my $fh1, "<", shift @ARGV or die qq[Error: Cannot open input file\n];
open my $fh2, "<", shift @ARGV or die qq[Error: Cannot open input file\n];

my (%file1);

while ( my $line = <$fh1> ) {
        chomp $line;
        my ($field1, $rest) = split /\|/, $line, 2;
        $file1{ $field1 } = $line;
}

while ( my $line = <$fh2> ) {
        chomp $line;
        for ( keys %file1 ) {
                if ( index( $line, $_ ) > -1 ) {
                        printf qq[%s%s\n], $file1{ $_ }, $line =~ s/,/|/gr;
                        last;
                }
        }

}
$ perl script.pl file1 file2
D24485|BNS Default Swap|-21180.9904679111|:D24485|48085|||Trading|FALSE||CASSE|19139|AA|CAD
D33963|BNS Default Swap|-261564.923909249|:D33963|48082|||Trading|FALSE||CAISSE|19149|AA|CAD|

Regards,
Birei
This User Gave Thanks to birei For This Post:
kykyboss
# 3  
Old 01-07-2012
Hi Birei

I found the folowing code that work but it take a long time as i have to open and close the <file2> all the time, is there a optimise way of doing it, maybe using more array?

Code:
#!/bin/perl

open(F1,"<@ARGV[0]");
#open(F2,"<@ARGV[1]");
open (OUTFILE,">$ARGV[2]");

sub run () {
        while(<F1>)
        {
                chomp;
                $line = $_;
                @ARA_NAME = split(/\|/,$line);
                find_ara_name()

        }
close(F1);
}


sub find_ara_name() {
open(F2,"<@ARGV[1]");
   foreach (<F2>){
                chomp;
                $line1 = $_;
                        if ($line1 =~ m/:@ARA_NAME[0],/) {
                        $line1 =~ s/,/|/g;
                        print OUTFILE "$line$line1\n";
                        return;
                        }
                close(F2);
                }
}

#main
run()

---------- Post updated at 07:03 PM ---------- Previous update was at 06:55 PM ----------

Hi
tanks a lot, i receive a compilation error with your script

Bareword found where operator expected at ./retest.pl line 22, near "s/,/|/gr"
syntax error at ./retest.pl line 22, near "s/,/|/gr"
Execution of ./retest.pl aborted due to compilation errors
# 4  
Old 01-07-2012
Ok. It seems your 'perl' version doesn't support the 'r' switch in regex. Try this workaround with same result:
Code:
$ cat script.pl
use warnings;
use strict;

die qq[Usage: perl $0 <file1> <file2>\n] unless @ARGV == 2;

open my $fh1, "<", shift @ARGV or die qq[Error: Cannot open input file\n];
open my $fh2, "<", shift @ARGV or die qq[Error: Cannot open input file\n];

my (%file1);

while ( my $line = <$fh1> ) {
        chomp $line;
        my ($field1, $rest) = split /\|/, $line, 2;
        $file1{ $field1 } = $line;
}

while ( my $line = <$fh2> ) {
        chomp $line;
        for ( keys %file1 ) {
                if ( index( $line, $_ ) > -1 ) {
                        $line =~ tr/,/|/;
                        printf qq[%s%s\n], $file1{ $_ }, $line;
                        last;
                }
        }

}

Regards,
Birei
This User Gave Thanks to birei For This Post:
kykyboss
# 5  
Old 01-07-2012
hi

thanks again this time it work better, but the ouput is duplicated because i need to do the search for "
Code:
:D33963,

and not only D33963 (as they could duplicated) how can i modify this ?

---------- Post updated at 07:22 PM ---------- Previous update was at 07:20 PM ----------
Code:
E00284|BNS Default Swap|154181.478745804|:E00284|48082|||Trading|FALSE||CAISSE|19149|AA|CAD||||||||2919966|FALSE||||BNS|||E|CA|CA|OTH_CBR|Call|Asset Asian|8000000.00|408|OTC-EQUITY OPT-SLD-TRA-EX||Options|Derivatives|||48082||False||||Sell|||||||||||||||||||||||||||||
E00284|BNS Default Swap|154181.478745804|:E00284CAP|48082|||Trading|FALSE||CAISSE|19149|AA|CAD||||||||2919966|FALSE||||BNS|||E|CA|CA|OTH_CBR|Call|Asset Asian|11040000.00|408|OTC-EQUITY OPT-SLD-TRA-EX||Options|Derivatives|||48082||False||||Buy|||||||||||||||||||||||||||||
E00286|BNS Default Swap|-21180.9904679111|:E00286|48082|||Trading|FALSE||CAISSE|19149|AA|CAD||||||||2919966|FALSE||||BNS|||E|CA|CA|OTH_CBR|Call|Asset Asian|10000000.00|653|OTC-EQUITY OPT-SLD-TRA-EX||Options|Derivatives|||48082||False||||Sell|||||||||||||||||||||||||||||
E00286|BNS Default Swap|-21180.9904679111|:E00286CAP|48082|||Trading|FALSE||CAISSE|19149|AA|CAD||||||||2919966|FALSE||||BNS|||E|CA|CA|OTH_CBR|Call|Asset Asian|14500000.00|653|OTC-EQUITY OPT-SLD-TRA-EX||Options|Derivatives|||48082||False||||Buy|||||||||||||||||||||||||||||
E00287|BNS Default Swap|-261564.923909249|:E00287|48082|||Trading|FALSE||92767||Internal|CAD||||||||2920257|FALSE||||BNS|||E|CA|CA|OTH_CBR|Call|Asset Asian|1992000.00|647|||Options|Derivatives|||48082||False||||Sell|||||||||||||||||||||||||||||

---------- Post updated at 07:37 PM ---------- Previous update was at 07:22 PM ----------

ok cool i found it thnaks a lot

Code:
$file1{ ":$field1," } = $line;

---------- Post updated at 08:06 PM ---------- Previous update was at 07:37 PM ----------

one more question after i stop how can i create a second output that will copy recoord for <file1> that doesn't match in <File2> with your previous code
Last edited by zaxxon; 01-07-2012 at 08:44 PM.. Reason: code tags
# 6  
Old 01-08-2012
1.- Output is not duplicated, lines are different.
Code:
D24485|BNS Default Swap|-21180.9904679111|:D24485|48085|||Trading|FALSE||CASSE|19139|AA|CAD
D33963|BNS Default Swap|-261564.923909249|:D33963|48082|||Trading|FALSE||CAISSE|19149|AA|CAD|

2.- Do you want in file1 a search only for D33963?

3.- What would be that second output, can you provide an example?

Regards,
Birei
# 7  
Old 01-11-2012
No it is all good for the output, i just look like to see if is possible to create a second output that will copy record from <file1> that doesn't match in <File2> with your previous code
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Reducing input file size after pattern search

I have a very large file with millions of entries identified by @M. I am using the following script to "extract" entries based on specific strings/patterns: #!/bin/bash if ] then file=$1 else echo "Input_file passed as an argument $1 is NOT found." exit; fi MID=(NULL "string-1"... (10 Replies)
Discussion started by: Xterra
10 Replies

2. Shell Programming and Scripting

Search pattern in a file taking input from another file

Hi, Below is my requirement File1: svasjsdhvassdvasdhhgvasddhvasdhasdjhvasdjsahvasdjvdasjdvvsadjhv vdjvsdjasvdasdjbasdjbasdjhasbdasjhdbjheasbdasjdsajhbjasbjasbhddjb svfsdhgvfdshgvfsdhfvsdadhfvsajhvasjdhvsajhdvsadjvhasjhdvjhsadjahs File2: sdh hgv I need a command such that... (8 Replies)
Discussion started by: imrandec85
8 Replies

3. Linux

Search a template file and replace with input

Hi I have a CommonTemplateStop.template file . Inside the file i need to replace the variables DepName and CompInsName with the values(Trade and TradeIns) specified in the script. I have written the below .sh script in linux server which will read the .template file and has to replace the 2... (8 Replies)
Discussion started by: samrat dutta
8 Replies

4. Shell Programming and Scripting

UNIX Scripting help to input string and search a file to find

Hi Don, this is not homework question. I work for a Credit card company and my development goal this year is to learn Unix. I would love if others can help me get started, thanks. Hi everyone I am new to Unix and need help writing a script that can ask user for an input, then search that input... (2 Replies)
Discussion started by: 12ic11
2 Replies

5. Shell Programming and Scripting

UNIX Scripting help to input string and search a file to find

Hi everyone, I am new to Unix and need help writing a script that can ask user for an input, then search that input within a file I know will have to use the read and grep commands, anyone can give me somewhere to start would help Task: Write a script to display which volume pool a given... (1 Reply)
Discussion started by: 12ic11
1 Replies

6. Shell Programming and Scripting

Bash to search file based off user input then create new file

In the below bash a file is downloaded when the program is opened and then that file is searched based on user input and the result is written to a new file. For example, the bash is opened and the download.txt is downloaded, the user then enters the id (NA04520). The id is used to search... (5 Replies)
Discussion started by: cmccabe
5 Replies

7. Shell Programming and Scripting

Read input files and merge them in given order and write them to input one param or one file

Dear Friends, I am looking for a shell script to merge input files into one file .. here is my idea: 1st paramter would be outfile file (all input files content) read all input files and merge them to input param 1 ex: if I pass 6 file names to the script then 1st file name as output file... (4 Replies)
Discussion started by: hyd1234
4 Replies

8. Solaris

Keyword search input from a file

Hi, I have a file which got only one column and got some keywords. I have another file where the keywords used in the first file are repeated in the second file. Now I would like to know how many times each keyword from the first file is repeated in the second file. Request your help on... (1 Reply)
Discussion started by: pointers
1 Replies

9. Shell Programming and Scripting

Using an input (?) file to search another

I have a file (DCN.txt) that has about 35000 lines. It looks like: 10004470028 10005470984 10006470301 10007474812 .... I have several other files (a11.txt, a12.txt, a12_1.txt, a13.txt, etc. about 70, each 100 mb large) that have history records like so: LINE 10005470984 01/06/2010... (13 Replies)
Discussion started by: oriqin
13 Replies

10. Shell Programming and Scripting

search file, change existing value based on input (awk help)

I have a file (status.file) of the form: valueA 3450 valueB -20 valueC -340 valueD 48 I am tailing a data.file, and need to search and modify a value in status.file...the tail is: tail -f data.file | awk '{ print $3, ($NF - $(NF-1)) }' which will produce lines that look like this: ... (3 Replies)
Discussion started by: nortonloaf
3 Replies
Login or Register to Ask a Question