How to search unique occurence in a file?


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers How to search unique occurence in a file?
# 1  
Old 10-05-2009
How to search unique occurence in a file?

Hi,

I have to search and count unique occurence of DE numbers in bold below in a file which has content like below.
Code:
Proc Tran F-BUY
Item Tkey Q5JV
Item Tsid JTIZ9
Item Tdat 20091001
Item Tset 20091001
Item Tbkr 5
Item Tshs 2
Item Tprc 897.0
Item Tcom 2000.0
Item Tcm1 20091001 05:28-e502157-DE29606619-1
Proc end
Proc Tran F-BUY
Item Tkey FSTCAN
Item Tsid JTIZ9
Item Tdat 20091001
Item Tset 20091001
Item Tbkr 5
Item Tshs 9
Item Tprc 897.0
Item Tcom 9000.0
Item Tcm1 20090930 15:45-e502157-DE29606620-1
Proc end
Proc Tran F-BUY
Item Tkey 148126
Item Tsid JTIZ9
Item Tdat 20091001
Item Tset 20091001
Item Tbkr 5
Item Tshs 5
Item Tprc 897.0
Item Tcom 5000.0
Item Tcm1 20091001 06:20-e502157-DE29606621-1
Proc end
Proc Tran F-BUY
Item Tkey 14813E
Item Tsid JTIZ9
Item Tdat 20091001
Item Tset 20091001
Item Tbkr 5
Item Tshs 2
Item Tprc 897.0
Item Tcom 2000.0
Item Tcm1 20091001 05:46-e502157-DE29606621-1
Proc end

Please help.

Thanks,
Akash

Last edited by pludi; 10-05-2009 at 07:49 AM.. Reason: code tags please...
# 2  
Old 10-05-2009
This should do:
Code:
perl -lne '$unique{$1}++ if /(DE\d+-\d)/; END{print "$_: $unique{$_}" foreach sort keys %unique;}' <yourfile>

Based on the file given it gives this output:
Code:
DE29606619-1: 1
DE29606620-1: 1
DE29606621-1: 2

# 3  
Old 10-05-2009
Thanks Pludi for your quick response.

Actually, I just need the distinct count of all DE's from the file, in the example the count is 3.

Is there any small unix script to get it?


Thanks in advance.
# 4  
Old 10-05-2009
Yes, there is (small variation of the first):
Code:
perl -lne '$unique{$1}++ if /(DE\d+-\d)/; END{print scalar keys %unique;}' <yourfile>

# 5  
Old 10-05-2009
It works. Thanks a lot. Smilie
# 6  
Old 10-05-2009
Quote:
Originally Posted by akash028
Thanks Pludi for your quick response.

Actually, I just need the distinct count of all DE's from the file, in the example the count is 3.

Is there any small unix script to get it?


Thanks in advance.
Another approach with awk:

Code:
awk -F- '$3 ~ /^DE/{a[$3FS$4]} END{for(i in a){{c++}}print c}' file

# 7  
Old 10-05-2009
Or:
(use gawk, nawk or /usr/xpg4/bin/awk on Solaris):

Code:
awk 'END { print c }
NF > 3 { _[substr($NF,15)]++ || ++c }
' infile

Some awk implementations (GNU awk, recent Bell Labs awk) support length(array) to return the number of keys:

Code:
awk 'END { print length(_) }
NF > 3 { _[substr($NF,15)] }
' infile


Last edited by radoulov; 10-05-2009 at 10:46 AM.. Reason: Perl code removed, Pludi already provided a better one.
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Awk: count unique elements in a field and sum their occurence across the entire file

Hi, Sure it's an easy one, but it drives me insane. input ("|" separated): 1|A,B,C,A 2|A,D,D 3|A,B,B I would like to count the occurence of each capital letters in $2 across the entire file, knowing that duplicates in each record count as 1. I am trying to get this output... (5 Replies)
Discussion started by: beca123456
5 Replies

2. Shell Programming and Scripting

Search only the first occurence in shell

Hello I have a configuration file and I want to extract a part of this configuration. Example of configuration: profile toto { bla bla blabla } conftest { toto { myarguements } I try to do a sed command: sed -n '/'toto' {$/,/^}/p' But the result is : profile toto { (6 Replies)
Discussion started by: maverick31
6 Replies

3. UNIX for Dummies Questions & Answers

To find the Nth Occurence of Search String

Hi guys, I like to find the Line number of Nth Occurence of a Search string in a file. If possible, if it will land the cursor to that particualar line will be great. Cheers!! (3 Replies)
Discussion started by: mac4rfree
3 Replies

4. UNIX for Dummies Questions & Answers

Search specific pattern in file and return number of occurence

Hi I want to search for a specific pattern in file Say ABC;HELLO_UNIX_WORLD;PQR ABC;HELLO_UNIX_WORLD_IS_NOT_ENOUGH;XYZ ABC;HELLO_UNIX_FORUM;LMN Pattern to search is : "HELLO_UNIX_*****" and not "HELLO_UNIX_***_***_" I mean after "HELLO_UNIX" there can only be one word.In this case... (2 Replies)
Discussion started by: dashing201
2 Replies

5. Shell Programming and Scripting

Search file for pattern2 starting from occurence of pattern1

Hi folks, I have a file which contains several occurences of 2 different patterns. I need to find out the line of first occurence of pattern2 starting after the position of first occurence of pattern1. example file: aaaa pattern2 bbbb pattern1 ccc pattern2 ddd pattern1 eee pattern2... (9 Replies)
Discussion started by: sameucho
9 Replies

6. Shell Programming and Scripting

How to identify the occurence of a pattern between a unique character?

hi, is it possible to find the number of occurences of a pattern between two paranthesis. for e.g i have a file as below. >>{ >>hi >>GoodMorning >>how are you? >>} >>is it good, >>tell me yes, if it is good In the above file, its clear the occurence of word "Good"... (17 Replies)
Discussion started by: divak
17 Replies

7. Shell Programming and Scripting

Perl : Search for next occurence of a word in an array

I have an array as follows: Space: ABC Name: def Age: 22 Type: new Name: fgh Age: 34 Type: old Space: XYZ Name: pqr Age: 44 Type: new : : How can I separate the array with elements starting from Space:ABC until Space: XYZ & put them in a different array & so on... (4 Replies)
Discussion started by: deo_kaustubh
4 Replies

8. Shell Programming and Scripting

search file between last occurence of 2 strings

I need to extract the last block of /== START OF SQLPLUS ==/ and /== END OF SQLPLUS ==/. The logifle is written to several times in a day using >> to append. I need a solution using grep/sed. logfile looks like this START OF LOGFILE /== START OF SQLPLUS ==/ ERROR /== END OF SQLPLUS... (5 Replies)
Discussion started by: hanton
5 Replies

9. Shell Programming and Scripting

Search for all the unique file extension

Greetings to All ... :b: I have one root folder containing different other folders within it. I need to get the list of all different types of file extensions residing in those folders. Could anyone help me providing some shell script? (1 Reply)
Discussion started by: riverside
1 Replies

10. UNIX for Dummies Questions & Answers

search& count for the occurence of a word

Greetings, I need to search and count all the occurences of a word in all the files in a directory. Any suggestions greatly appreciated. Thanks (1 Reply)
Discussion started by: skoppana
1 Replies
Login or Register to Ask a Question