Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Unique count from flat file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Unique count from flat file
# 1  
Old 12-30-2011
Unique count from flat file
Hello Guys

I have a flat file with '|~|' delimited
When I use to record count using below command

Code:
awk -FS"[|~|]+" ' {print $colno}' filename | wc -l

the count is fine

But when I am trying to find the unique number of record the o/p is always 1

Code:
awk -FS"[|~|]+" ' {print $colno}' filename |sort|uniq|wc -l

Please let me know how to find the unique record count
# 2  
Old 12-30-2011
Field separator doesn't matter while finding the count of unique lines, does it?

Try:
Code:
sort filename | uniq | wc -l

# 3  
Old 12-30-2011
Thanks but I need to know the uniq record on particular column
# 4  
Old 12-30-2011
Please provide a sample input and expected output.
# 5  
Old 12-30-2011
I/P file
Code:
 
 
9961881|~|20111229|~|000000218311635|~|1015104|~|000192170510|~|1|~|1|~||~|1|~|3755593|~|3755593|~|218311635
9961881|~|20111229|~|000000218311636|~|1015104|~|000192170510|~|1|~|1|~||~|1|~|3755593|~|3755593|~|218311636
9961881|~|20111229|~|000000218312203|~|1014486|~|000192174061|~|1021|~|1|~||~|1|~|90875|~|90875|~|218312203
9961881|~|20111229|~|000000218312204|~|1014486|~|000192174061|~|1267|~|1|~||~|1|~|90875|~|90875|~|218312204
9961881|~|20111229|~|000000218478637|~|1023353|~|000192465057|~|253|~|1|~||~|1|~|3755593|~|3755593|~|218478637
9961881|~|20111229|~|000000218478639|~|1023353|~|000192465057|~|801|~|1|~||~|1|~|3755593|~|3755593|~|218478639
9961881|~|20111229|~|000000218478640|~|1023353|~|000192465057|~|802|~|1|~||~|1|~|3755593|~|3755593|~|218478640
9961881|~|20111229|~|000000218478641|~|1023353|~|000192465057|~|253|~|1|~||~|1|~|3755593|~|3755593|~|218478641
9961881|~|20111229|~|000000218478642|~|1023353|~|000192465057|~|801|~|1|~||~|1|~|3755593|~|3755593|~|218478642
9961881|~|20111229|~|000000218478643|~|1023353|~|000192465057|~|802|~|1|~||~|1|~|3755593|~|3755593|~|218478643

Need uniq record count number on 4th field

Code:
 
awk -FS"[|~|]+" ' {print $4}' test.dat|sort|uniq|wc -l

o/p 1 which should be 3

if I change the FS to | in source file the o/p is 3
# 6  
Old 12-30-2011
Code:
awk -F\~ '{print $4}' test.dat|sort|uniq|wc -l

This User Gave Thanks to Klashxx For This Post:
Pratik4891
# 7  
Old 12-30-2011
Its awk -F not awk -FS
Code:
$ awk -F"[|~|]+" '{print $4}' filename | sort | uniq | wc -l
3

This User Gave Thanks to balajesuri For This Post:
Pratik4891
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Awk: count unique elements in a field and sum their occurence across the entire file

Hi, Sure it's an easy one, but it drives me insane. input ("|" separated): 1|A,B,C,A 2|A,D,D 3|A,B,B I would like to count the occurence of each capital letters in $2 across the entire file, knowing that duplicates in each record count as 1. I am trying to get this output... (5 Replies)
Discussion started by: beca123456
5 Replies

2. UNIX for Beginners Questions & Answers

Count unique column

Hello, I am trying to count unique rows in my file based on 4 columns (2-5) and to output its frequency in a sixth column. My file is tab delimited My input file looks like this: Colum1 Colum2 Colum3 Colum4 Coulmn5 1.1 100 100 a b 1.1 100 100 a c 1.2 200 205 a d 1.3 300 301 a y 1.3 300... (6 Replies)
Discussion started by: nans
6 Replies

3. UNIX for Beginners Questions & Answers

Count unique words

Dear all, I would like to know how to list and count unique words in thousands number of text files. Please help me out thanks in advance (9 Replies)
Discussion started by: imranrasheedamu
9 Replies

4. Shell Programming and Scripting

Count occurrence of column one unique value having unique second column value

Hello Team, I need your help on the following: My input file a.txt is as below: 3330690|373846|108471 3330690|373846|108471 0640829|459725|100001 0640829|459725|100001 3330690|373847|108471 Here row 1 and row 2 of column 1 are identical but corresponding column 2 value are... (4 Replies)
Discussion started by: angshuman
4 Replies

5. Shell Programming and Scripting

Find and count unique date values in a file based on position

Hello, I need some sort of way to extract every date contained in a file, and count how many of those dates there are. Here are the specifics: The date format I'm looking for is mm/dd/yyyy I only need to look after line 45 in the file (that's where the data begins) The columns of... (2 Replies)
Discussion started by: ronan1219
2 Replies

6. Shell Programming and Scripting

List unique values and count instances in .csv file

I need to take the second column of a .csv file and count the number of instances of each unique value in that same second column. I'd like the output to be value,count sorted by most instances. Thanks for any guidance! Data example: 317476,317756,0 816063,318861,0 313123,319091,0... (4 Replies)
Discussion started by: batcho
4 Replies

7. UNIX for Advanced & Expert Users

Count number of unique patterns from a log file

Hello Everyone I need your help in fixing this issue., I have a log file which has data of users logging in to an application. I want to search for a particular pattern in the log ISSessionValidated=N If this key word is found , the above 8 lines will contain the name of the user who's... (12 Replies)
Discussion started by: xtechkid
12 Replies

8. Shell Programming and Scripting

How to count Unique Values from a file.

Hi I have the following info in a file - <Cell id="25D"/> <Cell id="26A"/> <Cell id="26B"/> <Cell id="26C"/> <Cell id="27A"/> <Cell id="27B"/> <Cell id="27C"/> <Cell id="28A"/> I would like to know how would you go about counting all... (4 Replies)
Discussion started by: Prega
4 Replies

9. Shell Programming and Scripting

count alphabets in a flat file and print

I have a text file with a huge dataset, and each row in that dataset has some data(65479 rows). In that file I need to find the number of times a-z & A-Z Appears. How Can I Initialise Array into an Array to parse the count also I need to parse a,A into a single array preferably. example of... (3 Replies)
Discussion started by: vmsenthil
3 Replies

10. Shell Programming and Scripting

Help with pipe count in a flat file!!!

Hi Friends, I have a flat file and it has 43 columns which means 42 pipes but some records have less number of pipes.Can anyone tell me a command to count the number of pipes in a record and redirect the count to some new file because the flat file has not less than 40,000 records. Thanks... (14 Replies)
Discussion started by: kumarsaravana_s
14 Replies
Login or Register to Ask a Question