Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Retrieving the relevant search from search file in the main file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Retrieving the relevant search from search file in the main file
# 1  
Old 07-31-2014
Retrieving the relevant search from search file in the main file
I have two files:
file 1:

Code:
hello.com    neo.com,japan.com,example.com
news.net    xyz.com, telecom.net, highlands.net, software.com
example2.com    earth.net, abc.gov.uk

file 2:
Code:
neo.com
example.com
abc.gov.uk

file 2 are the search keys to search in file 1 if any of the search key is found in file 1 it should return the row with the found search key else just the row of the file 1 like this:

Code:
hello.com    neo.com, example.com
news.net
example2.com    abc.gov.uk

I tried this:
Code:
awk 'NR==FNR{a[NR]=$0;next}      {l=$1;for(x in a)if($0~a[x]){l=$0;break}print l}' file2 file1

it gave me the result like this:
Code:
hello.com   neo.com,japan.com,example.com 
news.net 
example2.com    earth.net, abc.gov.uk

Any idea to fix this since ? file 1 is very huge like 2 GB what can be efficient way to write this ?
# 2  
Old 07-31-2014
Checking the entire array for the right key is kind of overkill, awk can do that automatically -- if you store it correctly. Use "hello.com" as the index, not the entire line, and ("hello.com" in A) will evaluate true or false depending.

You don't want to store the entire 2GB file in memory... I'm guessing file2 is the smaller one, let's store that.

Code:
awk 'NR==FNR { A[$1] ; next } $1 in A { print A[$1] }' file2 file1

If this doesn't work, your input data may not be quite what it looks like. Already I can see that it has inconsistent spacing all over.
# 3  
Old 07-31-2014
column 1 and column 2 are tab separated. The code you provided doesn't produce any output.
# 4  
Old 07-31-2014
Sorry, I made a mistake.

I should point out, you gave me example data which produces no output, though! None of the headers in file2 match file1.

Code:
awk 'NR==FNR { A[$1] ; next } $1 in A' file2 file1

# 5  
Old 07-31-2014
If you see file2 (search key) at file1 column2 you will see the match. I need to retrieve those.
# 6  
Old 07-31-2014
Note that neither the sample input nor the sample output you provided contained any tab characters. The following will work with fields separated by any combination of one or more commas, spaces, and tabs and produce output that separates the 1st two columns of the output with a tab and separates subsequent fields with a comma followed by a space.
Code:
awk -F '[ \t,]' '
FNR == NR {
	a[$1]
	next
}
{	o = $1
	c = 0
	for(i = 2; i <= NF; i++)
		if($i in a)
			o = o (c++ ? ", " : "\t") $i
	print o
}' file2 file1

With the sample input files you provided, it produces the output:
Code:
hello.com	neo.com, example.com
news.net
example2.com	abc.gov.uk

Note that if you want to try this on a Solaris/SunOS system, change awk to /usr/xpg4/bin/awk, /usr/xpg6/bin/awk, or nawk.
This User Gave Thanks to Don Cragun For This Post:
csim_mohan
# 7  
Old 07-31-2014
Quote:
Originally Posted by csim_mohan
If you see file2 (search key) at file1 column2 you will see the match.
I see what you mean now. You don't want to match the first column, but any of the columns after.

The inconsistent format of the data you posted becomes a problem again, then. Is it tab-separated, comma-separated, space-seperated, or -- as it seems here -- all three?
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Read in search strings from text file, search for string in second text file and output to CSV

Hi guys, I have a text file named file1.txt that is formatted like this: 001 , ID , 20000 002 , Name , Brandon 003 , Phone_Number , 616-234-1999 004 , SSNumber , 234-23-234 005 , Model , Toyota 007 , Engine ,V8 008 , GPS , OFF and I have file2.txt formatted like this: ... (2 Replies)
Discussion started by: An0mander
2 Replies

2. UNIX for Dummies Questions & Answers

Search file and print everything except multiple search terms

I'm trying to find a way to search a range of similar words in a file. I tried using sed but can't get it right:sed 's/\(ca01\)*//'It only removes "ca01" but leaves the rest of the word. I still want the rest of the information on the lines just not these specific words listed below. Any... (3 Replies)
Discussion started by: seekryts15
3 Replies

3. Shell Programming and Scripting

Nested search in a file and replace the inner search

Hi Team, I am new to unix, please help me in this. I have a file named properties. The content of the file is : ##Mobile props east.url=https://qa.east.corp.com/prop/end west.url=https://qa.west.corp.com/prop/end south.url=https://qa.south.corp.com/prop/end... (2 Replies)
Discussion started by: tolearn
2 Replies

4. Shell Programming and Scripting

Search string within a file and list common words from the line having the search string

Hi, Need your help for this scripting issue I have. I am not really good at this, so seeking your help. I have a file looking similar to this: Hello, i am human and name=ABCD. How are you? Hello, i am human and name=PQRS. I am good. Hello, i am human and name=ABCD. Good bye. Hello, i... (12 Replies)
Discussion started by: royzlife
12 Replies

5. Shell Programming and Scripting

Perl - use search keywords from array and search a file and print 3rd field when matched

Hi , I have been trying to write a perl script to do this job. But i am not able to achieve the desired result. Below is my code. my $current_value=12345; my @users=("bob","ben","tom","harry"); open DBLIST,"<","/var/tmp/DBinfo"; my @input = <DBLIST>; foreach (@users) { my... (11 Replies)
Discussion started by: chidori
11 Replies

6. Shell Programming and Scripting

Optimised way for search & replace a value on one line in a very huge file (File Size is 24 GB).

Hi Experts, I had to edit (a particular value) in header line of a very huge file so for that i wanted to search & replace a particular value on a file which was of 24 GB in Size. I managed to do it but it took long time to complete. Can anyone please tell me how can we do it in a optimised... (7 Replies)
Discussion started by: manishkomar007
7 Replies

7. Shell Programming and Scripting

sed help - search/copy from one file and search/paste to another

I am a newbie and would like some help with the following - Trying to search fileA for a string similar to - AS11000022010 30.4 31.7 43.7 53.8 60.5 71.1 75.2 74.7 66.9 56.6 42.7 32.5 53.3 I then want to replace that string with a string from fileB - ... (5 Replies)
Discussion started by: ncwxpanther
5 Replies

8. Shell Programming and Scripting

Need help in retrieving log from a UNIX file using the search patterns

Hi everyone, I am trying to retrieve certain log from a big file. The log size can be from 200 - 600 lines. I have 3 search patterns, out of which 2 (first and last lines) search patterns are common for all the transactions but 3rd search pattern (occurs in the middle of transaction) is... (5 Replies)
Discussion started by: msrayudu
5 Replies

9. Shell Programming and Scripting

Read a file and search a value in another file create third file using AWK

Hi, I have two files with the format shown below. I need to read first field(value before comma) from file 1 and search for a record in file 2 that has the same value in the field "KEY=" and write the complete record of file 2 with corresponding field 2 of the first file in to result file. ... (11 Replies)
Discussion started by: King Kalyan
11 Replies
Login or Register to Ask a Question