Looping through entire directory and count unique values


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Looping through entire directory and count unique values
# 1  
Old 07-23-2013
Looping through entire directory and count unique values

Hello,

I`m a complete newbie to coding, please help with this problem.

I have multiple files in a directory, I have to loop through the contents of each file and extract number of unique isoforms in that file. Each file is tab delimited and only the line with the first parent (column 3) needs to be read.

Sample file

Code:
1 Graph graph 260052556 260054526 . + . ID=HCG177897.2_FG022
1 Graph parent 260052556 260052696 . + . ID=HCG177897.2_FG022_1;Isoforms=HCG177897.2_FGT022,HCG177897.2_FGT023
1 Graph child 260052742 260054066 . + . ID=HCG177897.2_FG022_2;Parent=HCG177897.2_FG022_1;Isoforms=HCG177897.2_FGT022
1 Graph child 260054161 260054239 . + . ID=HCG177897.2_FG022_3;Parent=HCG177897.2_FG022_2;Isoforms=HCG177897.2_FGT022
1 Graph child 260054323 260054526 . + . ID=HCG177897.2_FG022_4;Parent=HCG177897.2_FG022_3;Isoforms=HCG177897.2_FGT022
1 Graph child 260054323 260054526 . + . ID=HCG177897.2_FG022_4;Parent=HCG177897.2_FG022_3;Isoforms=HCG177897.2_FGT022
1 Graph child 260054323 260054526 . + . ID=HCG177897.2_FG022_4;Parent=HCG177897.2_FG022_3;Isoforms=HCG177897.2_FGT023
1 Graph child 260054323 260054526 . + . ID=HCG177897.2_FG022_4;Parent=HCG177897.2_FG022_3;Isoforms=HCG177897.2_FGT023
1 Graph child 260054323 260054526 . + . ID=HCG177897.2_FG022_4;Parent=HCG177897.2_FG022_3;Isoforms=HCG177897.2_FGT022
1 Graph child 260054323 260054526 . + . ID=HCG177897.2_FG022_4;Parent=HCG177897.2_FG022_3;Isoforms=HCG177897.2_FGT022

In the above sample the only revelant line is with the word 'parent' in column 3.

Code:
1 Graph parent 260052556 260052696 . + . ID=HCG177897.2_FG022_1;Isoforms=HCG177897.2_FGT022,HCG177897.2_FGT023

Column 9 is a composite column, where subfields are separated by ';' but the last subfield doesnt end with ';' as in the above line.

The subfield Isoforms=HCG177897.2_FGT022,HCG177897.2_FGT023 needs to be extracted and the number of isoforms need to be counted.
In this case there are 2 isoforms, which can be also calculated by number of (commas+1). Here there are 2 isoforms, separated by a comma.

I should note that sometimes there may be irrelevant fields after Isoforms,

example

Code:
1 Graph parent 260052556 260052696 . + . ID=HCG177897.2_FG022_1;Isoforms=HCG177897.2_FGT022,HCG177897.2_FGT023
;code=TRUE;FGTLL

but the calculations remain the same.

So I want to count the number of isoforms in each file in the directory
and write output in the form

HCG177897.2_FG022.txt 2

where HCG177897.2_FG022.txt is the filename



Desired output for sample inputs (attached)

Code:
 
HCG177897.2_FG002.txt 1
HCG177897.2_FG022.txt 2
HCG186375.3_FG001.txt 4

# 2  
Old 07-23-2013
Try this

Code:
$ awk '$3 == "parent" {
    gsub(/.*Isoforms=/,//);
        gsub(/;.*/, "");
        print FILENAME " " split($0, dummy, ",")
}' *

HCG177897.2_FG002.txt 1
HCG177897.2_FG022.txt 2
HCG186375.3_FG001.txt 4

This User Gave Thanks to Chubler_XL For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Awk: count unique elements in a field and sum their occurence across the entire file

Hi, Sure it's an easy one, but it drives me insane. input ("|" separated): 1|A,B,C,A 2|A,D,D 3|A,B,B I would like to count the occurence of each capital letters in $2 across the entire file, knowing that duplicates in each record count as 1. I am trying to get this output... (5 Replies)
Discussion started by: beca123456
5 Replies

2. UNIX for Beginners Questions & Answers

How to find and get a file in an entire directory with an excluded directory specified?

How to get a file 'zlib.h' in an entire directory with an excluded directory specified lives under that starting directory by using find command, as it failed on: $ find . -name 'zlib.h' -a -ipath 'CHROME.TMP' -prune -o -print it'll just list entirely up (2 Replies)
Discussion started by: abdulbadii
2 Replies

3. Shell Programming and Scripting

Count number of unique values in each column of array

What is an efficient way of counting the number of unique values in a 400 column by 1000 row array and outputting the counts per column, assuming the unique values in the array are: A, B, C, D In other words the output should look like: Value COL1 COL2 COL3 A 50 51 52... (16 Replies)
Discussion started by: Geneanalyst
16 Replies

4. Shell Programming and Scripting

Print count of unique values

Hello experts, I am converting a number into its binary output as : read n echo "obase=2;$n" | bc I wish to count the maximum continuous occurrences of the digit 1. Example : 1. The binary equivalent of 5 = 101. Hence the output must be 1. 2. The binary... (3 Replies)
Discussion started by: H squared
3 Replies

5. Shell Programming and Scripting

Count occurrence of column one unique value having unique second column value

Hello Team, I need your help on the following: My input file a.txt is as below: 3330690|373846|108471 3330690|373846|108471 0640829|459725|100001 0640829|459725|100001 3330690|373847|108471 Here row 1 and row 2 of column 1 are identical but corresponding column 2 value are... (4 Replies)
Discussion started by: angshuman
4 Replies

6. Shell Programming and Scripting

Count Unique values from multiple lists of files

Looking for a little help here. I have 1000's of text files within a multiple folders. YYYY/ /MM /1000's Files Eg. 2014/01/1000 files 2014/02/1237 files 2014/03/1400 files There are folders for each year and each month, and within each monthly folder there are... (4 Replies)
Discussion started by: whegra
4 Replies

7. Shell Programming and Scripting

Count frequency of unique values in specific column

Hi, I have tab-deliminated data similar to the following: dot is-big 2 dot is-round 3 dot is-gray 4 cat is-big 3 hot in-summer 5 I want to count the frequency of each individual "unique" value in the 1st column. Thus, the desired output would be as follows: dot 3 cat 1 hot 1 is... (5 Replies)
Discussion started by: owwow14
5 Replies

8. Shell Programming and Scripting

Find and count unique date values in a file based on position

Hello, I need some sort of way to extract every date contained in a file, and count how many of those dates there are. Here are the specifics: The date format I'm looking for is mm/dd/yyyy I only need to look after line 45 in the file (that's where the data begins) The columns of... (2 Replies)
Discussion started by: ronan1219
2 Replies

9. Shell Programming and Scripting

List unique values and count instances in .csv file

I need to take the second column of a .csv file and count the number of instances of each unique value in that same second column. I'd like the output to be value,count sorted by most instances. Thanks for any guidance! Data example: 317476,317756,0 816063,318861,0 313123,319091,0... (4 Replies)
Discussion started by: batcho
4 Replies

10. Shell Programming and Scripting

How to count Unique Values from a file.

Hi I have the following info in a file - <Cell id="25D"/> <Cell id="26A"/> <Cell id="26B"/> <Cell id="26C"/> <Cell id="27A"/> <Cell id="27B"/> <Cell id="27C"/> <Cell id="28A"/> I would like to know how would you go about counting all... (4 Replies)
Discussion started by: Prega
4 Replies
Login or Register to Ask a Question