awk column comparison big file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting awk column comparison big file
# 1  
Old 03-03-2011
awk column comparison big file

Hi all,

I would like to compare a column in one file to a column in another file and when there is a match it prints the first column and the corresponding second column. Example

File1
ABA
ABC
ABE
ABF

File 2
ABA 123
ABB 124
ABD 125
ABC 126

So what I would like printed to a file is
ABA 123
ABC 126

The only thing is that in file 1 there are 8,000 columns while in file 2 there are 140,000 columns to search for. I have tried awk and grep-f but it doesn't work or is very slow?

Any quick solutions?

Thanks
# 2  
Old 03-03-2011
Here is one solution:
Code:
join file1 file2

Note that both files must be sorted.
# 3  
Old 03-03-2011
I would prefer not to sort them because I want to keep them in the order in which they are because it corresponds to a large excel file and join does not work for me
# 4  
Old 03-03-2011
Code:
awk 'NR==FNR{a[$0]=1}NR>FNR&&a[$1]' file1 file2

# 5  
Old 03-03-2011
awk 'NR==FNR{a[$0]=1}NR>FNR&&a[$1]' file1 file2
does not work
# 6  
Old 03-03-2011
how about this?
Code:
awk 'NR==FNR{a[$1]=1}NR>FNR&&a[$1]' file1 file2

# 7  
Old 03-03-2011
Code:
grep -f File1 File2

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Range Comparison Of Column Value in File1 with Other File

Hi, I have a file1 whose 17th column needs to be checked if it exists in between the values of column 2 & column 3 as mentioned in another file2. Output of the matched value to be put in separate file 3 & 4. File1: ... (10 Replies)
Discussion started by: siramitsharma
10 Replies

2. Shell Programming and Scripting

File comparison and proccessing using awk

Hi Guys, I am having two requirement in one of my scripts. please help out to find a fast solution using AWK (since there is lot of data to be processed) 1) First snippet - File1 has two columns and file2 has three columns If any value of column 1 of file1 matches with column 1... (4 Replies)
Discussion started by: stormfield
4 Replies

3. Shell Programming and Scripting

File comparison using awk

Hi All, i have two files file1 ,file 2 file 1 col1|col2|col3|col4|col5|col6|col7|col8 11346925|0|2009-09-20|9999-12-31|100|0 11346925|0|2009-09-20|9999-12-31|120|0 12954311|0|2009-09-11|9999-12-31|100|0 12954311|0|2009-07-23|2999-12-31|120|0 12954312|0|2009-09-11|9999-12-31|100|0... (9 Replies)
Discussion started by: mohanalakshmi
9 Replies

4. Shell Programming and Scripting

File comparison using awk

my files are as follows fileA sepearated by tab /t 00 lieferungen 00 attractiop 01 done 02 forness 03 rasp 04 alwaysisng 04 funny 05 done1 fileB funnymou120112 funnymou234470 mou3raspnhdhv rddfgmoudone1438748 so all those record which are greater than 3 and which are not... (6 Replies)
Discussion started by: rajniman
6 Replies

5. Shell Programming and Scripting

AWK - Parse a big file

INPUT SAMPLE Symmetrix ID : 000192601507 Masking View Name : TS00P22_13E_1 Last updated at : 05:10:18 AM on Tue Mar 22,2011 Initiator Group Name : 10000000c960b9cd Host Initiators { WWN : 10000000c960b9cd } Port Group Name :... (8 Replies)
Discussion started by: greycells
8 Replies

6. Shell Programming and Scripting

column value comparison in a file

Hi, Can any one help with my below requirement. i need to compare each line by line and in each line i have to compare some columns values with previous line column values in perl script. Can any one help me........! its very urgent. Thanks (3 Replies)
Discussion started by: jam_prasanna
3 Replies

7. Shell Programming and Scripting

Looking for AWK Solution for column comparison in a single file

- I am looking for different kind of awk solution which I don't think is mentioned before in these forums. Number of rows in the file are fixed Their are two columns in file1.txt 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 10 I am looking for 3... (1 Reply)
Discussion started by: softwarekids23
1 Replies

8. Shell Programming and Scripting

Inserting a column from one file into another big file

Hi I have two files, one is 1.6 GB. I would like to add one extra column of information to the large file at a specific location (after its 2nd column). For example: File 1 has two columns more than 1000 rows like this MM009987 1 File 2 looks like this MM00098 MM00076 3 4 2 4 2... (1 Reply)
Discussion started by: sogi
1 Replies

9. Shell Programming and Scripting

awk program for file comparison

Hello there, I'm trying to write an awk program in bash shell with the following three input files: File 1 1001 1 2 3 1002 4 5 6 1003 7 8 9 1004 10 11 12 File 2 1001 11 22 33 1002 44 55 66 1004 100 111 122 ... (4 Replies)
Discussion started by: kbirde
4 Replies

10. Shell Programming and Scripting

Big data file - sed/grep/awk?

Morning guys. Another day another question. :rolleyes: I am knocking up a script to pull some data from a file. The problem is the file is very big (up to 1 gig in size), so this solution: for results in `grep "^\ ... works, but takes ages (we're talking minutes) to run. The data is held... (8 Replies)
Discussion started by: dlam
8 Replies
Login or Register to Ask a Question