Sponsored Content
Top Forums Shell Programming and Scripting Getting Sum, Count and Distinct Count of a file Post 302292439 by singhabhijit on Saturday 28th of February 2009 01:21:52 AM
Old 02-28-2009
Getting Sum, Count and Distinct Count of a file

Hi all this is a UNIX question.

I have a large flat file with millions of records.
col1|col2|col3
1|a|b
2|c|d
3|e|f
3|g|h
footer****

I am supposed to calculate the sum of col1 1+2+3+3=9, count of col1 1,2,3,3=4, and distinct count of col1 1,2,3=c3

I would like it if you avoid external commands like AWK. Also, can we do the same by creating a function?

Please bear in mind that the file is huge

Thanks in advance

Last edited by Franklin52; 02-28-2009 at 06:08 AM.. Reason: urls removed
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Count the distinct list of ids

Hello guys, I have a file in the following format(each line seperated by TAB): ========= Filename id Filename id1 Filename id Filename1 id7 Filename1 id7 Filename2 id1 Filename2 id1 Filename2 id3 Filename3 id2 Filename3 id4 Filename3 id4 Filename3 id6 ========= I would like to... (2 Replies)
Discussion started by: jingi1234
2 Replies

2. Shell Programming and Scripting

To count distinct fields in a row

I have . dat file which contains data in a specific format: 0 3 892 921 342 1 3 921 342 543 2 4 817 562 718 765 3 3 819 562 717 761 i need to compare each field in a row with another field of the same column but different row and cont the... (8 Replies)
Discussion started by: Abhik
8 Replies

3. Shell Programming and Scripting

sum divided by count

Dear friends, I'm stuck with the task below, I would be thankful for all your replies. INPUT : Date Price Volume 20110601 73052811.61 2845833 20110602 61489062.96 9909230 20110603 72790724.65 1108927 20110606 48299507.20 7435881 20110607 ... (5 Replies)
Discussion started by: hernand
5 Replies

4. Shell Programming and Scripting

awk and count sum ?

I have a input.txt file which have 3 fields separate by a comma place, os and timediff in seconds tampa,win7, 2575 tampa,win7, 157619 tampa,win7, 3352 dallas,vista,604799 greenbay,winxp, 14400 greenbay,win7 , 518400 san jose,winxp, 228121 san jose,winxp, 70853 san jose,winxp, 193514... (5 Replies)
Discussion started by: sabercats
5 Replies

5. UNIX for Dummies Questions & Answers

count number of distinct values in each column with awk

Hi ! input: A|B|C|D A|F|C|E A|B|I|C A|T|I|B As the title of the thread says, I would need to get: 1|3|2|4 I tried different variants of this command, but I don't manage to obtain what I need: gawk 'BEGIN{FS=OFS="|"}{for(i=1; i<=NF; i++) a++} END {for (b in a) print b}' input ... (2 Replies)
Discussion started by: beca123456
2 Replies

6. Shell Programming and Scripting

Script Shell: Count The sum of numbers in a file

Hi all; Here is my file: V1.3=4 V1.4=5 V1.1=3 V1.2=6 V1.3=6 Please, can you help me to write a script shell that counts the sum of values in my file (4+5+3+6+6) ? Thank you so much for help. Kind regards. (3 Replies)
Discussion started by: chercheur111
3 Replies

7. Shell Programming and Scripting

Help with Getting distinct record count from a .dat file using UNIX command

Hi, I have a .dat file with contents like the below: Input file ============SEQ NO-1: COLUMN1========== 9835619 7152815 ============SEQ NO-2: COLUMN2 ========== 7615348 7015548 9373086 ============SEQ NO-3: COLUMN3=========== 9373086 Expected Output: (I just... (1 Reply)
Discussion started by: MS06
1 Replies

8. Shell Programming and Scripting

Finding total distinct count from multiple csv files through UNIX script

Hi All , I have multiple pipe delimited csv files are present in a directory.I need to find out distinct count on a column on those files and need the total distinct count on all files. We can't merge all the files here as file size are huge in millions.I have tried in below way for each... (9 Replies)
Discussion started by: STCET22
9 Replies

9. UNIX for Beginners Questions & Answers

Awk: count unique elements in a field and sum their occurence across the entire file

Hi, Sure it's an easy one, but it drives me insane. input ("|" separated): 1|A,B,C,A 2|A,D,D 3|A,B,B I would like to count the occurence of each capital letters in $2 across the entire file, knowing that duplicates in each record count as 1. I am trying to get this output... (5 Replies)
Discussion started by: beca123456
5 Replies

10. UNIX for Beginners Questions & Answers

Copy columns from one file into another and get sum of column values and row count

I have a file abc.csv, from which I need column 24(PurchaseOrder_TotalCost) to get the sum_of_amounts with date and row count into another file say output.csv abc.csv- UTF-8,,,,,,,,,,,,,,,,,,,,,,,,, ... (6 Replies)
Discussion started by: Tahir_M
6 Replies
LaTeX::Encode(3pm)					User Contributed Perl Documentation					LaTeX::Encode(3pm)

NAME
LaTeX::Encode - encode characters for LaTeX formatting SYNOPSIS
use LaTeX::Encode; $latex = latex_encode($text, %options); VERSION
This manual page describes version 0.03 of the "LaTeX::Encode" module. DESCRIPTION
This module provides a function to encode text that is to be formatted with LaTeX. It encodes characters that are special to LaTeX or that are represented in LaTeX by LaTeX commands. The special characters are: "" (command character), "{" (open group), "}" (end group), "&" (table column separator), "#" (parameter specifier), "%" (comment character), "_" (subscript), "^" (superscript), "~" (non-breakable space), "$" (mathematics mode). Note that some of the LaTeX commands for characters are defined in the LaTeX "textcomp" package. If your text includes such characters, you will need to include the following lines in the preamble to your LaTeX document. usepackage[T1]{fontenc} usepackage{textcomp} The function is useful for encoding data that is interpolated into LaTeX document templates, say with "Template::Plugin::Latex" (shameless plug!). SUBROUTINES
/METHODS "latex_encode($text, %options)" Encodes the specified text such that it is suitable for processing with LaTeX. The behaviour of the filter is modified by the options: "except" Lists the characters that should be excluded from encoding. By default no special characters are excluded, but it may be useful to specify "except = "\{}"" to allow the input string to contain LaTeX commands such as "this is \textbf{bold} text" (the doubled backslashes in the strings represent Perl escapes, and will be evaluated to single backslashes). "iquotes" If true then single or double quotes around words will be changed to LaTeX single or double quotes; double quotes around a phrase will be converted to "``" and "''" and single quotes to "`" and "'". This is sometimes called "intelligent quotes" "use_textcomp" By default the "latex_encode" filter will encode characters with the encodings provided by the "textcomp" LaTeX package (for example the Pounds Sterling symbol is encoded as "\textsterling{}"). Setting "use_textcomp = 0" turns off these encodings. NOT YET IMPLEMENTED EXAMPLES
The following snippet shows how data from a database can be encoded and inserted into a LaTeX table, the source of which is generated with "LaTeX::Table". my $sth = $dbh->prepare('select col1, col2, col3 from table where $expr'); $sth->execute; while (my $href = $sth->fetchrow_hashref) { my @row; foreach my $col (qw(col1 col2 col3)) { push(@row, latex_encode($href->{$col})); } push @data, @row; } my $headings = [ [ 'Col1', 'Col2', 'Col3' ] ]; my $table = LaTeX::Table->new( { caption => 'My caption', label => 'table:caption', type => 'xtab', header => $header, data => @data } ); my $table_text = $table->generate_string; Now $table_text can be interpolated into a LaTeX document template. DIAGNOSTICS
None. You could probably break the "latex_encode" function by passing it an array reference as the options, but there are no checks for that. CONFIGURATION AND ENVIRONMENT
Not applicable. DEPENDENCIES
The "HTML::Entities" and "Pod::LaTeX" modules were used for building the encoding table in "LaTeX::Encode::EncodingTable", but this is not rebuilt at installation time. The "LaTeX::Driver" module is used for formatting the character encodings reference document. INCOMPATIBILITIES
None known. BUGS AND LIMITATIONS
Not all LaTeX special characters are included in the encoding tables (more may be added when I track down the definitions). The "use_textcomp" option is not implemented. AUTHOR
Andrew Ford <a.ford@ford-mason.co.uk> LICENSE AND COPYRIGHT
Copyright (C) 2007 Andrew Ford. All Rights Reserved. This module is free software; you can redistribute it and/or modify it under the same terms as Perl itself. This software is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. SEE ALSO
Template::Plugin::Latex perl v5.10.0 2007-10-02 LaTeX::Encode(3pm)
All times are GMT -4. The time now is 03:05 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy