Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Lookup on large file based on a temp file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Lookup on large file based on a temp file
# 1  
Old 05-14-2012
Lookup on large file based on a temp file
hello guys

Please help me with the below issue

I have two files one base file another lookupfile

base file

Code:
 
abc-001
bcd-001
cde-001

Lookupfile

Code:
 
abc-001|11|12
abc-001|11|12
abc-001|11|12
bcd-001|11|12
bcd-001|11|12
cde-001|11|12
cde-001|11|12
cde-001|11|12
cde-001|11|12

based on the base file value (like abc-001,bcd-001) I have to lookup the lookupfile and output the count for each base value

o/p
Code:
 
abc-001|3
bcd-001|2
cde-001|4

I think taking the values in base file into a variable and grep that value in the lookup file will work but couldnt find the right code

Thanks a lot for your valuable time and help
# 2  
Old 05-14-2012
How about this?
Code:
awk -F"|" 'NR==FNR{a[$1]++;next}
a[$1]{print $1,a[$1]}' OFS="|" mainFile baseFile

This User Gave Thanks to pravin27 For This Post:
Pratik4891
# 3  
Old 05-14-2012
Thanks a lot Pravin

Need one more suggestion ,the value I am searching in lookup file is in column 4
So please let me know the modified code for this

Code:
 
Q|W|R|abc-001|11|12
Q|W|R|abc-001|11|12
Q|W|R|abc-001|11|12
Q|W|R|bcd-001|11|12
Q|W|R|bcd-001|11|12
Q|W|R|cde-001|11|12
Q|W|R|cde-001|11|12
Q|W|R|cde-001|11|12
Q|W|R|cde-001|11|12

Thanks again for your support
# 4  
Old 05-14-2012
Code:
# awk -F'|' 'FNR==NR{a[$1]}($4 in a){A[$4,x]++}END{for(i in A)print i FS A[i]}' base lookup
abc-001|3
bcd-001|2
cde-001|4

This User Gave Thanks to ygemici For This Post:
Pratik4891
# 5  
Old 05-14-2012
Thanks a lot ...working perfectlySmilie
# 6  
Old 05-15-2012
Lookup on large file based on a temp file
Hello ygemici ,

Thanks a lot for good post. Can you please explain the code as you used multidim array here. This will be use ful for us in future.

Code:
 
# awk -F'|' 'FNR==NR{a[$1]}($4 in a){A[$4,x]++}END{for(i in A)print i FS A[i]}' base lookup

Thanks
Krsnadasa
# 7  
Old 05-15-2012
i only use it (x or any) for the sequential print with index where in the (i in A)
you can read the usefull link Smilie
Multi-dimensional - The GNU Awk User's Guide

regards
ygemici
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Delete the records in file based on lookup file.

Hi I have two files one.txt and two.txt one.txt 123 324 456 235 456 two txt abc one 000 123 abc abc one 000 456 abc abc one 000 122 abc abc one 000 111 abc My question here is, the records which are present in one.txt has to deleted in second file two.txt my output result... (2 Replies)
Discussion started by: Ganesh L
2 Replies

2. Shell Programming and Scripting

Help with Splitting a Large XML file based on size AND tags

Hi All, This is my first post here. Hoping to share and gain knowledge from this great forum !!!! I've scanned this forum before posting my problem here, but I'm afraid I couldn't find any thread that addresses this exact problem. I'm trying to split a large XML file (with multiple tag... (7 Replies)
Discussion started by: Aviktheory11
7 Replies

3. Shell Programming and Scripting

Splitting large file and renaming based on field

I am trying to update an older program on a small cluster. It uses individual files to send jobs to each node. However the newer database comes as one large file, containing over 10,000 records. I therefore need to split this file. It looks like this: HMMER3/b NAME 1-cysPrx_C ACC ... (2 Replies)
Discussion started by: fozrun
2 Replies

4. Shell Programming and Scripting

Problem with splitting large file based on pattern

Hi Experts, I have to split huge file based on the pattern to create smaller files. The pattern which is expected in the file is: Master..... First... second.... second... third.. third... Master... First.. second... third... Master... First... second.. second.. second..... (2 Replies)
Discussion started by: saisanthi
2 Replies

5. Shell Programming and Scripting

Splitting large file into multiple files in unix based on pattern

I need to write a shell script for below scenario My input file has data in format: qwerty0101TWE 12345 01022005 01022005 datainala alanfernanded 26 qwerty0101mXZ 12349 01022005 06022008 datainalb johngalilo 28 qwerty0101TWE 12342 01022005 07022009 datainalc hitalbert 43 qwerty0101CFG 12345... (19 Replies)
Discussion started by: jimmy12
19 Replies

6. Shell Programming and Scripting

awk - splitting 1 large file into multiple based on same key records

Hello gurus, I am new to "awk" and trying to break a large file having 4 million records into several output files each having half million but at the same time I want to keep the similar key records in the same output file, not to exist accross the files. e.g. my data is like: Row_Num,... (6 Replies)
Discussion started by: kam66
6 Replies

7. Shell Programming and Scripting

Split large file based on last digit from a column

Hello, What's the best way to split a large into multiple files based on the last digit in the first column. input file: f 2738483300000x0y03772748378831x1y13478378358383x2y23743878383802x3y33787828282820x4y43748838383881x5y5 Desired Output: f0 3738483300000x0y03787828282820x4y4 f1... (9 Replies)
Discussion started by: alain.kazan
9 Replies

8. Shell Programming and Scripting

split large file based on field criteria

I have a file containing date/time sorted data of the form ... 2009/06/10,20:59:59.950,XAG/USD,Q,1,1115, 14.3025,100,1,1 2009/06/10,20:59:59.950,XAG/USD,Q,1,1116, 14.3026,125,1,1 2009/06/10,20:59:59.950,XAG/USD,R,0,0, , 0,0,0 2009/06/10,20:59:59.950,XAG/USD,R,1,0, 14.1910,100,1,1... (6 Replies)
Discussion started by: asriva
6 Replies

9. Shell Programming and Scripting

SED based on file lookup

Newb here trying to figure this one out. :confused: I am trying to create a SED (or some other idea) line that will replace the data field if the original text is seen in a separate text file. The lookup file would be line delimted. For example: sed 's/<if in file>/YES/' File structure:... (3 Replies)
Discussion started by: sdlennon
3 Replies

10. UNIX for Advanced & Expert Users

Clueless about how to lookup and reverse lookup IP addresses under a file!!.pls help

Write a quick shell snippet to find all of the IPV4 IP addresses in any and all of the files under /var/lib/output/*, ignoring whatever else may be in those files. Perform a reverse lookup on each, and format the output neatly, like "IP=192.168.0.1, ... (0 Replies)
Discussion started by: choco4202002
0 Replies
Login or Register to Ask a Question