Search Pattern and combine into single file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Search Pattern and combine into single file
# 1  
Old 06-25-2013
Search Pattern and combine into single file

Hi Experts

Please help me out with the following thing:
2 files and want the output file: {No for using FOR loop because I got 22 million lines}
Tried that "It processes only 8000 records per hour"
I need a faster way out !!!
FileA:
Code:
9051
9052
9053
9054
9055
9056
9057
9058
9059

FileB:
Code:
9051;123
9052;456
9054;567
9057;789
9059;123

Output File:
Code:
9051;9051;123
9052;9052;456
9053;
9054;9054;567
9055;
9056;
9057;9057;789
9058;
9059;

Regards

Last edited by Franklin52; 06-25-2013 at 03:14 AM.. Reason: Please use code tags
# 2  
Old 06-25-2013
Try join. I can't tell for sure if this efficient enough (I don't have the huge data similar to the samples provided)
Code:
[user@host ~]$ join -t';' -a1 -o0 2.1 2.2 fileA fileB
9051;9051;123
9052;9052;456
9053;;
9054;9054;567
9055;;
9056;;
9057;9057;789
9058;;
9059;9059;123
[user@host ~]$

# 3  
Old 06-25-2013
Thanks Sir, but I am getting the following output:

Code:
9051;9051;123
9052;9052;456
9053;;
9054;;
9055;;
9056;;
9057;;
9058;;
9059;;

I tried the following command, but it only gives me matches pattern:
Code:
awk -F";" 'NR==FNR{A[$1]=$0;next}$1 in A{$0=$0","A[$1];print}' fileA fileB

Regards
Navkanwal

Last edited by radoulov; 06-25-2013 at 07:27 AM..
# 4  
Old 06-25-2013
Quote:
Originally Posted by navkanwal
2 files and want the output file: {No for using FOR loop because I got 22 million lines}
Tried that "It processes only 8000 records per hour"
I need a faster way out !!!
Quote:
Originally Posted by navkanwal
I tried the following command, but it only gives me matches pattern:
awk -F";" 'NR==FNR{A[$1]=$0;next}$1 in A{$0=$0","A[$1];print}' fileA fileB
If you load 22 millions lines in an array, it will definitely lead to performance issues. And here's your awk one-liner:
Code:
awk -F";" 'NR==FNR{A[$1]=$2;next} {if (A[$1]){$0=$0";"$1";"A[$1];print}else{print}}' fileB fileA

And, with the sample fileA and fileB you provided in post #1, using join I got the output which I've pasted in post #2. Not sure, why you're getting a different output. Could you please re-check.
# 5  
Old 06-25-2013
Thanks Sir.

The command is working fine and output was generated in less than 5 minutes.
Thanks for your support.

Regards
# 6  
Old 06-25-2013
Quote:
Originally Posted by navkanwal
The command is working fine and output was generated in less than 5 minutes.
Please mention which command worked fine.. awk or join? It'll be helpful to those who visit this thread at a later point.
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Search Pattern and Print lines in Single Column

Hi Experts I have small query where I request the into a single file Suppose: File1: {Unique entries} AA BB CC DD FileB: AA, 123 AA, 234 AA, 2345 CC, 123 CC, 5678 DD,123 BB, 7890 (5 Replies)
Discussion started by: navkanwal
5 Replies

2. Shell Programming and Scripting

How to use sed to search a particular pattern in a file backward after a pattern is matched.?

Hi, I have two files file1.txt and file2.txt. Please see the attachments. In file2.txt (which actually is a diff output between two versions of file1.txt.), I extract the pattern corresponding to 1172c1172. Now ,In file1.txt I have to search for this pattern 1172c1172 and if found, I have to... (9 Replies)
Discussion started by: saurabh kumar
9 Replies

3. Shell Programming and Scripting

Search for a pattern in a String file and count the occurance of each pattern

I am trying to search a file for a patterns ERR- in a file and return a count for each of the error reported Input file is a free flowing file without any format example of output ERR-00001=5 .... ERR-01010=10 ..... ERR-99999=10 (4 Replies)
Discussion started by: swayam123
4 Replies

4. Shell Programming and Scripting

Combine Multiple Files into Single One File One after other

I am trying to combine 4 .dat files into one single Output file Inputs are:- file123.dat, file256.dat, file378.dat & file490 Expected Output:- FileName=file1 {text from file1} EOF {blank line} FileName=file2 {text from file2} EOF {blank line} FileName=file3 {text from file3} EOF... (4 Replies)
Discussion started by: lancesunny
4 Replies

5. Shell Programming and Scripting

how to find first files in a directory and combine them as a single file?

i have below list of files in a directory. /root/admin/files/file1.txt /root/admin/files/file2.txt /root/admin/files/file3.txt /root/admin/files/pattern.txt /root/admin/files/server.txt i need combine the above text files in the below sequence, file1.txt, pattern.txt,server.txt =>... (8 Replies)
Discussion started by: vel4ever
8 Replies

6. Shell Programming and Scripting

Combine in single file

I have 2 files: phone.txt and mobile.txt ex. phone.txt MOBILENO|DISABLE_DATE 919687877754|9/1/2011| 919687877762|9/1/2011| 919687880573|9/1/2011| 919687882658|9/2/2011| Ex. mobile.txt MOBILENO |TIME 919687880573|2011-09-17 12:23:40| 919687882658|2011-10-10 21:15:33|... (4 Replies)
Discussion started by: khingx
4 Replies

7. Shell Programming and Scripting

awk pattern match and search in single statement

Hi All, I am trying to alter all lines between EXEC SQL and END-EXEC that have an INCLUDE in them. The following code search="INCLUDE " cp -f ${WORK}/$file.in ${WORK}/$file.wrk2 for item in `echo $search `; do > ${WORK}/$file.wrk1 awk -vITEM="$item" '{ if ( $0... (3 Replies)
Discussion started by: Bruble
3 Replies

8. Shell Programming and Scripting

Search a pattern in a file with contents in a single line

Hi all I am searching for a pattern in a file . The file content is in a single line.If am doing a grep or sed for the a particular pattern am getting whole file. I want the result in different lines. attaching the file for reference search pattern "/xxxxxx/hhhh/tttttttt/sss/" and... (4 Replies)
Discussion started by: sparks
4 Replies

9. Shell Programming and Scripting

Search file for pattern and grab some lines before pattern

I want to search a file for a string and then if the string is found I need the line that the string is on - but also the previous two lines from the file (that the pattern will not be found in) This is on solaris Can you help? (2 Replies)
Discussion started by: frustrated1
2 Replies
Login or Register to Ask a Question