Grouping and counting


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Grouping and counting
# 15  
Old 08-17-2016
One awk command might be faster than a grep | cut | sort | uniq command chain.
The FS or -F is an ERE, so one can put two optional " in it.
Code:
file="inputfile"
awk -F '"?[|]"?' '($4==1 && $5=="Y") { A[$NF]++ } END { for (i in A) print i, A[i] }' $file > $file.new &&
mv $file.new $file

awk produces a new file. If sucessful the mv command replaces the input file with it.

Last edited by MadeInGermany; 08-17-2016 at 03:22 PM.. Reason: colored
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Help with grouping and zipping

Hi can you please help with the below ? source file: Column1,Column2,Column3,Column4 abc,123,dir1/FXX/F19,1 abc,123,dir1/FXX/F20,1 abc,123,dir1/FXX/F23,2 abc,123,dir1/FXX/C25,2 abc,123,dir1/FXX/X25,2 abc,123,dir1/FXX/A23,3 abc,123,dir1/FXX/Z25,3 abc,123,dir1/FXX/Y25,4 I want to... (3 Replies)
Discussion started by: paul1234
3 Replies

2. Shell Programming and Scripting

Grouping and Calculating

Hi All, I want to read the input file and store the output in the Output file. I pasted the sample Input and Output file below. Help me with this. Input file ================================= ITEM1 AAAAA 1 ITEM1 BBBBB 1 ITEM1 CCCCC 1 ITEM2 AAAAA 5 ITEM2 CCCCC 4... (1 Reply)
Discussion started by: humaemo
1 Replies

3. Shell Programming and Scripting

Name grouping

awk 'FNR==NR {a; next} $NF in a' genes.txt refseq_exons.txt > output.txt I can not figure out how to group the same name in $4 together. Basically, all the SKI together in separate rows and all the TGFB2. Thank you :). chr1 2160133 2161174 SKI chr1 218518675 218520389 TGFB2... (1 Reply)
Discussion started by: cmccabe
1 Replies

4. Shell Programming and Scripting

UNIX grouping

Hi guys, I am a complete newbie to unix and have been tasked with creating a script to group the following data (file) by hourly slots so that I can count the transactions completed within the peak hour. I am not sure how to group data like this in unix. Can anyone please help? Here is an... (1 Reply)
Discussion started by: MrMidas
1 Replies

5. Shell Programming and Scripting

Grouping

Hi all, I am using following command: perl program.pl input.txt output.txt CUTOFF 3 > groups_3.txt containing program.pl, two files (input.txt, output.txt) and getting output in groups_3.txt: But, I wish to have 30 files corresponding to each CUTOFF ranging from 0 to 30 using the same... (1 Reply)
Discussion started by: bioinfo
1 Replies

6. UNIX for Dummies Questions & Answers

Grouping in grep

How do you do grouping in grep? Here's how I tried it at first: egrep 'qualit(y|ies)' /usr/share/dict/words -bash: syntax error near unexpected token `(' I'm using GNUgrep, and I found this on their site. grep regular expression syntax So I tried this: egrep 'qualit\(y\|ies\)'... (2 Replies)
Discussion started by: sudon't
2 Replies

7. Shell Programming and Scripting

Selective grouping

I have a text file in this format. Group: AAA Notes: IP : 11.11.11.11 #User xxxxxxxxx #Password aaaaaaaaaaaaaaaa Group: AAA Notes: IP : 11.11.11.22 #User yyyyyyyyyyyyy #Password bbbbbbbbbbbbb (8 Replies)
Discussion started by: anil510
8 Replies

8. UNIX for Advanced & Expert Users

grouping lines

Hi all, I have input lines like below: A;100;Paris;City;10;0;0 A;100;Paris;City;0;10;0 A;100;Paris;City in Europe;0;0;20 B;101;London;City;20;0;0 B;101;London;City;0;20;0 B;101;London;City in Europe;0;0;40 I need to group the above lines to: A;100;Paris;City in Europe;10;10;20... (4 Replies)
Discussion started by: andy2000
4 Replies

9. UNIX for Dummies Questions & Answers

Help with data grouping

Hi all, I have a set data as shown below, and i would like to eliminate the name that no children - boy and girl. What is the appropriate command can i use(other than grep)? Please assist... My input: name sex marital status children - boy children - girl ... (3 Replies)
Discussion started by: 793589
3 Replies

10. Shell Programming and Scripting

egrep and grouping

i am using the c shell on solaris. directories i'm working with: ls -1d DIV* DIV_dental/ DIV_ibc/ DIV_ifc/ DIV_index/ DIV_pharm/ DIV_sectionI/ DIV_sectionI-title/ DIV_sectionI-toc/ DIV_sectionII-title/ DIV_sectionII-toc/ DIV_standing/ DIV_standing-toc/ DIV_title/ DIV_vision/ (1 Reply)
Discussion started by: effigy
1 Replies
Login or Register to Ask a Question
DBI::ProfileData(3)					User Contributed Perl Documentation				       DBI::ProfileData(3)

NAME
DBI::ProfileData - manipulate DBI::ProfileDumper data dumps SYNOPSIS
The easiest way to use this module is through the dbiprof frontend (see dbiprof for details): dbiprof --number 15 --sort count This module can also be used to roll your own profile analysis: # load data from dbi.prof $prof = DBI::ProfileData->new(File => "dbi.prof"); # get a count of the records in the data set $count = $prof->count(); # sort by longest overall time $prof->sort(field => "longest"); # sort by longest overall time, least to greatest $prof->sort(field => "longest", reverse => 1); # exclude records with key2 eq 'disconnect' $prof->exclude(key2 => 'disconnect'); # exclude records with key1 matching /^UPDATE/i $prof->exclude(key1 => qr/^UPDATE/i); # remove all records except those where key1 matches /^SELECT/i $prof->match(key1 => qr/^SELECT/i); # produce a formatted report with the given number of items $report = $prof->report(number => 10); # clone the profile data set $clone = $prof->clone(); # get access to hash of header values $header = $prof->header(); # get access to sorted array of nodes $nodes = $prof->nodes(); # format a single node in the same style as report() $text = $prof->format($nodes->[0]); # get access to Data hash in DBI::Profile format $Data = $prof->Data(); DESCRIPTION
This module offers the ability to read, manipulate and format DBI::ProfileDumper profile data. Conceptually, a profile consists of a series of records, or nodes, each of each has a set of statistics and set of keys. Each record must have a unique set of keys, but there is no requirement that every record have the same number of keys. METHODS
The following methods are supported by DBI::ProfileData objects. $prof = DBI::ProfileData->new(File => "dbi.prof") $prof = DBI::ProfileData->new(Files => [ "dbi.prof.1", "dbi.prof.2" ]) Creates a a new DBI::ProfileData object. Takes either a single file through the File option or a list of Files in an array ref. If multiple files are specified then the header data from the first file is used. $copy = $prof->clone(); Clone a profile data set creating a new object. $header = $prof->header(); Returns a reference to a hash of header values. These are the key value pairs included in the header section of the DBI::ProfileDumper data format. For example: $header = { Path => '[ DBIprofile_Statement, DBIprofile_MethodName ]', Program => 't/42profile_data.t', }; Note that modifying this hash will modify the header data stored inside the profile object. $nodes = $prof->nodes() Returns a reference the sorted nodes array. Each element in the array is a single record in the data set. The first seven elements are the same as the elements provided by DBI::Profile. After that each key is in a separate element. For example: $nodes = [ [ 2, # 0, count 0.0312958955764771, # 1, total duration 0.000490069389343262, # 2, first duration 0.000176072120666504, # 3, shortest duration 0.00140702724456787, # 4, longest duration 1023115819.83019, # 5, time of first event 1023115819.86576, # 6, time of last event 'SELECT foo FROM bar' # 7, key1 'execute' # 8, key2 # 6+N, keyN ], # ... ]; Note that modifying this array will modify the node data stored inside the profile object. $count = $prof->count() Returns the number of items in the profile data set. $prof->sort(field => "field") $prof->sort(field => "field", reverse => 1) Sorts data by the given field. Available fields are: longest total count shortest The default sort is greatest to smallest, which is the opposite of the normal Perl meaning. This, however, matches the expected behav- ior of the dbiprof frontend. $count = $prof->exclude(key2 => "disconnect") $count = $prof->exclude(key2 => "disconnect", case_sensitive => 1) $count = $prof->exclude(key1 => qr/^SELECT/i) Removes records from the data set that match the given string or regular expression. This method modifies the data in a permanent fashion - use clone() first to maintain the original data after exclude(). Returns the number of nodes left in the profile data set. $count = $prof->match(key2 => "disconnect") $count = $prof->match(key2 => "disconnect", case_sensitive => 1) $count = $prof->match(key1 => qr/^SELECT/i) Removes records from the data set that do not match the given string or regular expression. This method modifies the data in a perma- nent fashion - use clone() first to maintain the original data after match(). Returns the number of nodes left in the profile data set. $Data = $prof->Data() Returns the same Data hash structure as seen in DBI::Profile. This structure is not sorted. The nodes() structure probably makes more sense for most analysis. $text = $prof->format($nodes->[0]) Formats a single node into a human-readable block of text. $text = $prof->report(number => 10) Produces a report with the given number of items. AUTHOR
Sam Tregar <sam@tregar.com> COPYRIGHT AND LICENSE
Copyright (C) 2002 Sam Tregar This program is free software; you can redistribute it and/or modify it under the same terms as Perl 5 itself. perl v5.8.0 2002-12-01 DBI::ProfileData(3)