Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Count number of occurences using awk

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Count number of occurences using awk
# 8  
Old 04-14-2013
Please try it and then let us know about problems, if any.
# 9  
Old 04-14-2013
This will work for the small files you presented above:
Code:
$ sort file2 | uniq -c | grep -f file1
      2 xx
      3 yy

On the other hand, grepping umpteen million lines with 11 million fixed strings will be seriously demanding, if doable at all.
# 10  
Old 04-15-2013
Quote:
Originally Posted by elixir_sinari
Please try it and then let us know about problems, if any.
Hey,

I think it didnt work for large files. Here are specifications

Code:
file1.txt
bash-3.00# cat file1.txt |wc -l
 17102666

more file1.txt
123advertise3
123advertise4
123advertise5
123advertiseb
123advertisec
123advertised
123advertisedebtconsolidation
123advertisee
123advertisef
123advertiseg
123advertiseh
123advertisehomaxproducts


Code:
file2.txt
bash-3.00#cat file2.txt | wc -l
 113842500


more file2.txt
123123apartment
123123attorney
123123auction
123123auto
123advertisedebtconsolidation
123advertiseb
123123automate
123123automatic
123123bank
123advertisedebtconsolidation
123advertiseb
123123banking
123123bankruptcy
123advertisedebtconsolidation
123123bargain
123123best
123123blog
123advertisedebtconsolidation
123123building

I ran below command as described above

Code:
bash-3.00# nawk 'FNR==NR{c[$1];next}$1 in c{++c[$1]}END{for(i in c) print i,c[i]}' file1.txt file2.txt

but I got only 36000 lines only as per below format. However, I wanted output like word: <number of occurances>

Code:
peaktablethomecsuchico
browsepropertyhomebase
clickflowershomedsn
worldwideflowerstravelagency
acepigb
acepigc
browsecompanytravelagent
liveearnhomedownpaymentassistance
acepigd
bargainsystemhomebvcure
acepige
acepigf
uniquecasinohomecycling
alternativeanyhomecanningrecipes
acepigj
annualsurveyhomedma


Can somebody help me here please for larger files?

---------- Post updated at 05:06 AM ---------- Previous update was at 12:27 AM ----------

Please help me.I am stuck.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Count and print the number of occurences

I have some files as shown below GLL ALM 654-656 654 656 SEM LYG 655-657 655 657 SEM LYG 655-657 655 657 ALM LEG 656-658 656 658 ALM LEG 656-658 656 658 ALM LEG 656-658 656 658 LEG LEG 658-660 658 660 LEG LEG 658-660 658 660 The value of GLL is... (5 Replies)
Discussion started by: arch
5 Replies

2. Shell Programming and Scripting

[solved]awk count occurences in time window

Input File Time, KeyStation 00:00:00,000;KS_1 00:00:01,000;KS_1 00:00:02,000;KS_1 00:00:03,000;KS_1 00:00:04,000;KS_1 00:00:05,000;KS_1 00:00:06,000;KS_1 00:01:01,000;KS_1 00:01:02,000;KS_1 00:01:03,000;KS_1 00:01:04,000;KS_1 00:01:05,000;KS_1 00:01:06,000;KS_1 01:00:01,000;KS_1... (0 Replies)
Discussion started by: Calypso
0 Replies

3. Shell Programming and Scripting

awk count occurences

line number:status, market, keystation 1,SENT,EBS,1 : 1 2,DONE,REU,1 : 1 3,SENT,EBS,2 : 1 4,DONE,EBS,1 : 0 5,SENT,EBS,2 : 0 6,SENT,EBS,2 : 0 7,SENT,EBS,2 : 0 8,SENT,EBS,1 : 1 for each status, market combination I want to keep a tally of active orders. i.e if an order is SENT, then +1, if... (8 Replies)
Discussion started by: Calypso
8 Replies

4. UNIX for Dummies Questions & Answers

Count pattern occurences

hi, I have a text..and i need to find a pattern in the text and count to the no of times the pattern occured. i have used grep command ..but the problem is , it shows the occurrences of the pattern but doesn't count no of times the pattern occuries. (5 Replies)
Discussion started by: nvnni
5 Replies

5. Shell Programming and Scripting

Awk to count occurences

Hi, i am in need of an awk script to accomplish the following: Input table looks like: Student1 arts Student2 science Student3 arts Student4 science Student5 science Student6 science Student7 science Student8 science Student9 science Student10 science Student11 science... (8 Replies)
Discussion started by: saint2006
8 Replies

6. Shell Programming and Scripting

to count the number of occurences of a column value

im trying to count the number of occurences of column 2 value(starting from KKK*) of the below file, file.txt using the code cat file.txt | awk ' BEGIN { print "Category Counts"} {FS=","} {NR > 2} { cats = cats + 1} END { for(c in cats) { print c, "=", cats} } ' but its returning as ... (6 Replies)
Discussion started by: michaelrozar17
6 Replies

7. Shell Programming and Scripting

Count number of occurences of a character in a field defined by the character in another field

Hello, I have a text file with n lines in the following format (9 column fields): Example: contig00012 149606 G C 49 68 60 18 c$cccccacccccccccc^c I need to count the number of lower-case and upper-case occurences in column 9, respectively, of the... (3 Replies)
Discussion started by: s052866
3 Replies

8. Shell Programming and Scripting

awk counting number of occurences

Hi, I am trying to count the max number of occurences of field1 in my apache log example: 10.0.0.1 field2 field3 10.0.0.2 filed2 field3 10.0.0.1 field2 field3 10.0.0.1 field2 field3 awk result to print out only the most occurence of field1 and number of occurence and field1 is... (3 Replies)
Discussion started by: phamp008
3 Replies

9. UNIX for Dummies Questions & Answers

Count number of occurences of a word

I want to count the number of occurences of say "200" in a file but that file also contains various stuff including dtaes like 2007 or smtg like 200.1 so count i am getting by doing grep -c "word" file is wrong Please help!!!!! (8 Replies)
Discussion started by: shikhakaul
8 Replies

10. Shell Programming and Scripting

How to count the number of occurences of this pattern?

Hi all, I have a pattern like this in a file: 123 4 56 789 234 5 67 789 121 3 56 789 222 4 65 789 321 6 90 100 478 8 40 789 243 7 80 789 How can I count the number of occurences of '789' (4th column) in this set...? Thanks for all your help! K (7 Replies)
Discussion started by: kripssmart
7 Replies
Login or Register to Ask a Question