Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Count common elements in a column

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Count common elements in a column
# 1  
Old 12-13-2013
Count common elements in a column
HI, I have a 3-column tab separated column (approx 1GB) in which I would like to count and output the frequency of all of the common elements in the 1st column.
For instance:
If my input was the following:

Code:
dot is-big 2
dot is-round 3
dot is-gray 4
cat is-big 3
hot in-summer 5

My desired output would be the following:
Code:
dot 3
cat 1
hot 1


How can I do this with `wc` or `grep`?
# 2  
Old 12-13-2013
Code:
$ cut -d' ' -f1 file | sort | uniq -c
      1 cat
      3 dot
      1 hot

Code:
$ awk '{A[$1]++}END{for(i in A)print i,A[i]}' file
hot 1
cat 1
dot 3

# 3  
Old 12-13-2013
Code:
awk '{ print $1}' <filename>| uniq -c

# 4  
Old 12-13-2013
How about awk?

Code:
awk '{a[$1]++} END {for (i in a) { print i OFS a[i] }}' file


EDIT: If you have a lot of different values then you might run into memory issues with that (which zozoo's solution should avoid).
# 5  
Old 12-13-2013
@zozoo :

You should sort before using uniq, in current example it's fine, but in some cases there will be chances of printing duplicates.
This User Gave Thanks to Akshay Hegde For This Post:
zozoo
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Add column and multiply its result to all elements of another column

Input file is as follows: 1 | 6 2 | 7 3 | 8 4 | 9 5 | 10 Output reuired (sum of the first column $1*$2) 1 | 6 | 90 2 | 7 | 105 3 | 8 | 120 4 |9 | 135 5 |10 | 150 Please enclose sample input, sample output, and code... (5 Replies)
Discussion started by: Sagar Singh
5 Replies

2. UNIX for Beginners Questions & Answers

Awk: count unique elements in a field and sum their occurence across the entire file

Hi, Sure it's an easy one, but it drives me insane. input ("|" separated): 1|A,B,C,A 2|A,D,D 3|A,B,B I would like to count the occurence of each capital letters in $2 across the entire file, knowing that duplicates in each record count as 1. I am trying to get this output... (5 Replies)
Discussion started by: beca123456
5 Replies

3. Shell Programming and Scripting

Sum column values based in common identifier in 1st column.

Hi, I have a table to be imported for R as matrix or data.frame but I first need to edit it because I've got several lines with the same identifier (1st column), so I want to sum the each column (2nd -nth) of each identifier (1st column) The input is for example, after sorted: K00001 1 1 4 3... (8 Replies)
Discussion started by: sargotrons
8 Replies

4. Shell Programming and Scripting

Matching column and search closest elements

Hi all I have a great challenge that I am not able to resolve. Briefly, I have a file like this: ID_1 chr1 100 - ID_2 chr2 300 + and another file like this: name_1 chr1 150 no - name_2 chr1 250 yes - name_3 chr2 350 yes + name_4 chr2 280 yes + Well, for each entry in file1 I would... (2 Replies)
Discussion started by: giuliangiuseppe
2 Replies

5. Shell Programming and Scripting

Getting the most common column with respect another

hi all, i want to get the most comon column w.r.t another this is my file Tom|london Tom|london Tom|Paris Adam|Madrid Adam|NY the Output to get me : Tom|london Adamn|Madrid ive tried (10 Replies)
Discussion started by: teefa
10 Replies

6. UNIX for Dummies Questions & Answers

Merging tables: identifiying common and unique elements

Hi all, I know how to merge two tables and to remove the duplicated lines based on a field (Column 2) . My next challenge is to be able to identify in a new column those common elements between table A & B, those elements in table A not present in table B and vice versa. A simple count would be... (6 Replies)
Discussion started by: lsantome
6 Replies

7. Shell Programming and Scripting

Count and merge using common column

I have the following records from multiple files. 415 A G 415 A G 415 A T 415 A . 415 A . 421 G A 421 G A,C 421 G A 421 G A 421 G A,C 421 G . 427 A C 427 A ... (3 Replies)
Discussion started by: empyrean
3 Replies

8. Shell Programming and Scripting

Filtering lines for column elements based on corresponding counts in another column

Hi, I have a file like this ACC 2 2 21 aaa AC 443 3 22 aaa GCT 76 1 33 xxx TCG 34 2 33 aaa ACGT 33 1 22 ggg TTC 99 3 44 wee CCA 33 2 33 ggg AAC 1 3 55 ddd TTG 10 1 22 ddd TTGC 98 3 22 ddd GCT 23 1 21 sds GTC 23 4 32 sds ACGT 32 2 33 vvv CGT 11 2 33 eee CCC 87 2 44... (1 Reply)
Discussion started by: polsum
1 Replies

9. UNIX for Dummies Questions & Answers

Average for repeated elements in a column

I have a file that looks like this 452 025_E3 8 025_E3 82 025_F5 135 025_F5 5 025_F5 23 025_G2 38 025_G2 71 025_G2 9 026_A12 81 026_A12 10 026_A12 some of the elements in column2 are repeated. I want an output file that will extract the... (1 Reply)
Discussion started by: FelipeAd
1 Replies

10. Shell Programming and Scripting

find common elements in 2 files (for loop)

Hi, i'm new here (and to scripting too). I was hoping for some help in comparing two files. i have a file called 'file1' with a list of names in the following format: adam jones paul higgins kelly lowe i also have another file which may contain some of the names but with a lot of... (4 Replies)
Discussion started by: ibking
4 Replies
Login or Register to Ask a Question