Sponsored Content
Top Forums Shell Programming and Scripting Getting Sum, Count and Distinct Count of a file Post 302292439 by singhabhijit on Saturday 28th of February 2009 01:21:52 AM
Old 02-28-2009
Getting Sum, Count and Distinct Count of a file

Hi all this is a UNIX question.

I have a large flat file with millions of records.
col1|col2|col3
1|a|b
2|c|d
3|e|f
3|g|h
footer****

I am supposed to calculate the sum of col1 1+2+3+3=9, count of col1 1,2,3,3=4, and distinct count of col1 1,2,3=c3

I would like it if you avoid external commands like AWK. Also, can we do the same by creating a function?

Please bear in mind that the file is huge

Thanks in advance

Last edited by Franklin52; 02-28-2009 at 06:08 AM.. Reason: urls removed
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Count the distinct list of ids

Hello guys, I have a file in the following format(each line seperated by TAB): ========= Filename id Filename id1 Filename id Filename1 id7 Filename1 id7 Filename2 id1 Filename2 id1 Filename2 id3 Filename3 id2 Filename3 id4 Filename3 id4 Filename3 id6 ========= I would like to... (2 Replies)
Discussion started by: jingi1234
2 Replies

2. Shell Programming and Scripting

To count distinct fields in a row

I have . dat file which contains data in a specific format: 0 3 892 921 342 1 3 921 342 543 2 4 817 562 718 765 3 3 819 562 717 761 i need to compare each field in a row with another field of the same column but different row and cont the... (8 Replies)
Discussion started by: Abhik
8 Replies

3. Shell Programming and Scripting

sum divided by count

Dear friends, I'm stuck with the task below, I would be thankful for all your replies. INPUT : Date Price Volume 20110601 73052811.61 2845833 20110602 61489062.96 9909230 20110603 72790724.65 1108927 20110606 48299507.20 7435881 20110607 ... (5 Replies)
Discussion started by: hernand
5 Replies

4. Shell Programming and Scripting

awk and count sum ?

I have a input.txt file which have 3 fields separate by a comma place, os and timediff in seconds tampa,win7, 2575 tampa,win7, 157619 tampa,win7, 3352 dallas,vista,604799 greenbay,winxp, 14400 greenbay,win7 , 518400 san jose,winxp, 228121 san jose,winxp, 70853 san jose,winxp, 193514... (5 Replies)
Discussion started by: sabercats
5 Replies

5. UNIX for Dummies Questions & Answers

count number of distinct values in each column with awk

Hi ! input: A|B|C|D A|F|C|E A|B|I|C A|T|I|B As the title of the thread says, I would need to get: 1|3|2|4 I tried different variants of this command, but I don't manage to obtain what I need: gawk 'BEGIN{FS=OFS="|"}{for(i=1; i<=NF; i++) a++} END {for (b in a) print b}' input ... (2 Replies)
Discussion started by: beca123456
2 Replies

6. Shell Programming and Scripting

Script Shell: Count The sum of numbers in a file

Hi all; Here is my file: V1.3=4 V1.4=5 V1.1=3 V1.2=6 V1.3=6 Please, can you help me to write a script shell that counts the sum of values in my file (4+5+3+6+6) ? Thank you so much for help. Kind regards. (3 Replies)
Discussion started by: chercheur111
3 Replies

7. Shell Programming and Scripting

Help with Getting distinct record count from a .dat file using UNIX command

Hi, I have a .dat file with contents like the below: Input file ============SEQ NO-1: COLUMN1========== 9835619 7152815 ============SEQ NO-2: COLUMN2 ========== 7615348 7015548 9373086 ============SEQ NO-3: COLUMN3=========== 9373086 Expected Output: (I just... (1 Reply)
Discussion started by: MS06
1 Replies

8. Shell Programming and Scripting

Finding total distinct count from multiple csv files through UNIX script

Hi All , I have multiple pipe delimited csv files are present in a directory.I need to find out distinct count on a column on those files and need the total distinct count on all files. We can't merge all the files here as file size are huge in millions.I have tried in below way for each... (9 Replies)
Discussion started by: STCET22
9 Replies

9. UNIX for Beginners Questions & Answers

Awk: count unique elements in a field and sum their occurence across the entire file

Hi, Sure it's an easy one, but it drives me insane. input ("|" separated): 1|A,B,C,A 2|A,D,D 3|A,B,B I would like to count the occurence of each capital letters in $2 across the entire file, knowing that duplicates in each record count as 1. I am trying to get this output... (5 Replies)
Discussion started by: beca123456
5 Replies

10. UNIX for Beginners Questions & Answers

Copy columns from one file into another and get sum of column values and row count

I have a file abc.csv, from which I need column 24(PurchaseOrder_TotalCost) to get the sum_of_amounts with date and row count into another file say output.csv abc.csv- UTF-8,,,,,,,,,,,,,,,,,,,,,,,,, ... (6 Replies)
Discussion started by: Tahir_M
6 Replies
MAXDB_STMT_BIND_RESULT(3)						 1						 MAXDB_STMT_BIND_RESULT(3)

maxdb_stmt_bind_result - Binds variables to a prepared statement for result storage

       Procedural style

SYNOPSIS
bool maxdb_stmt_bind_result (resource $stmt, mixed &$var1, [mixed &$...]) DESCRIPTION
Object oriented style bool maxdb_stmt::bind_result (mixed &$var1, [mixed &$...]) maxdb_stmt_bind_result(3) is used to associate (bind) columns in the result set to variables. When maxdb_stmt_fetch(3) is called to fetch data, the MaxDB client/server protocol places the data for the bound columns into the specified variables $var1, .... Note Note that all columns must be bound prior to calling maxdb_stmt_fetch(3). Depending on column types bound variables can silently change to the corresponding PHP type. A column can be bound or rebound at any time, even after a result set has been partially retrieved. The new binding takes effect the next time maxdb_stmt_fetch(3) is called. RETURN VALUES
Returns TRUE on success or FALSE on failure. EXAMPLES
Example #1 Object oriented style <?php $maxdb = new maxdb("localhost", "MONA", "RED", "DEMODB"); if (maxdb_connect_errno()) { printf("Connect failed: %s ", maxdb_connect_error()); exit(); } /* prepare statement */ if ($stmt = $maxdb->prepare("SELECT zip, name FROM hotel.city ORDER BY name")) { $stmt->execute(); /* bind variables to prepared statement */ $stmt->bind_result($col1, $col2); /* fetch values */ while ($stmt->fetch()) { printf("%s %s ", $col1, $col2); } /* close statement */ $stmt->close(); } /* close connection */ $maxdb->close(); ?> Example #2 Procedural style <?php $link = maxdb_connect("localhost", "MONA", "RED", "DEMODB"); /* check connection */ if (!$link) { printf("Connect failed: %s ", maxdb_connect_error()); exit(); } /* prepare statement */ if ($stmt = maxdb_prepare($link, "SELECT zip, name FROM hotel.city ORDER BY name")) { maxdb_stmt_execute($stmt); /* bind variables to prepared statement */ maxdb_stmt_bind_result($stmt, $col1, $col2); /* fetch values */ while (maxdb_stmt_fetch($stmt)) { printf("%s %s ", $col1, $col2); } /* close statement */ maxdb_stmt_close($stmt); } /* close connection */ maxdb_close($link); ?> The above example will output something similar to: 12203 Albany 60601 Chicago 60615 Chicago 45211 Cincinnati 33575 Clearwater 75243 Dallas 32018 Daytona Beach 33441 Deerfield Beach 48226 Detroit 90029 Hollywood 92714 Irvine 90804 Long Beach 11788 Long Island 90018 Los Angeles 70112 New Orleans 10019 New York 10580 New York 92262 Palm Springs 97213 Portland 60018 Rosemont 95054 Santa Clara 20903 Silver Spring 20005 Washington 20019 Washington 20037 Washington SEE ALSO
maxdb_stmt_bind_param(3), maxdb_stmt_execute(3), maxdb_stmt_fetch(3), maxdb_prepare(3), maxdb_stmt_prepare(3), maxdb_stmt_init(3), maxdb_stmt_errno(3), maxdb_stmt_error(3). PHP Documentation Group MAXDB_STMT_BIND_RESULT(3)
All times are GMT -4. The time now is 09:51 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy