Sponsored Content
Top Forums UNIX for Dummies Questions & Answers Unique values in a row sum the next column in UNIX Post 302907987 by Don Cragun on Wednesday 2nd of July 2014 10:32:48 PM
Old 07-02-2014
Since you didn't say anything about whether or not your input file was sorted, the earlier suggestions had to make the assumption that lines in your 20,000,000 line file were in random order. Therefore, all of the key values and the sums of the corresponding 2nd fields had to be kept in memory until the entire file had been read. Then the totals could be printed for each of the different keys present in the file. The error messages you got say that awk ran out of memory trying to accumulate all of the data.

Now that we know that all of the lines with a given key are adjacent in your input file, the later scripts could print a sum for each key as soon as a new key was found. Very little memory is required to do that and your script runs much faster because it needs less system resources to get the job done.
These 2 Users Gave Thanks to Don Cragun For This Post:
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Concatenating column values with unique id into single row

Hi, I have a table in Db2 with data say id_1 phase1 id_1 phase2 id_1 phase3 id_2 phase1 id_2 phase2 I need to concatenate the values like id_1 phase1,phase2,phase3 id_2 phase1,phase2 I tried recursive query but in vain as the length of string to be concatenated in quite long. ... (17 Replies)
Discussion started by: jsaravana
17 Replies

2. Shell Programming and Scripting

print unique values of a column and sum up the corresponding values in next column

Hi All, I have a file which is having 3 columns as (string string integer) a b 1 x y 2 p k 5 y y 4 ..... ..... Question: I want get the unique value of column 2 in a sorted way(on column 2) and the sum of the 3rd column of the corresponding rows. e.g the above file should return the... (6 Replies)
Discussion started by: amigarus
6 Replies

3. Shell Programming and Scripting

Sum of values coming in a row

Hi, my requirement is to sum values in a row. eg: input is: sum,value1,value2,value3,.....,value N Required Output: sum,<summation of N values> Please help me... (5 Replies)
Discussion started by: MrGopal666
5 Replies

4. Shell Programming and Scripting

Print unique names in each row of a specific column using awk

Is it possible to remove redundant names in the 4th column? input cqWE 100 200 singapore;singapore AZO 300 400 brazil;america;germany;ireland;germany .... .... output cqWE 100 200 singapore AZO 300 400 brazil;america;germany;ireland (4 Replies)
Discussion started by: quincyjones
4 Replies

5. UNIX for Dummies Questions & Answers

awk to sum column field from duplicate row/lines

Hello, I am new to Linux environment , I working on Linux script which should send auto email based on the specific condition from log file. Below is the sample log file Name m/c usage abc xxx 10 abc xxx 20 abc xxx 5 xyz ... (6 Replies)
Discussion started by: asjaiswal
6 Replies

6. Shell Programming and Scripting

Sum specified values (columns) per row

Hello out there, file.txt: comp51820_c1_seq1 42 N 0:0:0:0:0:0 1:0:0:0:0:0 0:0:0:0:0:0 3:0:0:0:0:0 0:0:0:0:0:0 comp51820_c1_seq1 43 N 0:0:0:0:0:0 0:1:0:0:0:0 0:0:0:0:0:0 0:3:0:0:0:0 0:0:0:0:0:0 comp51820_c1_seq1 44 N 0:0:4:0:3:1 0:0:1:9:0:0 10:0:0:0:0:0 0:3:3:2:2:6 2:2:2:5:60:3... (16 Replies)
Discussion started by: pathunkathunk
16 Replies

7. Shell Programming and Scripting

Sum column values based in common identifier in 1st column.

Hi, I have a table to be imported for R as matrix or data.frame but I first need to edit it because I've got several lines with the same identifier (1st column), so I want to sum the each column (2nd -nth) of each identifier (1st column) The input is for example, after sorted: K00001 1 1 4 3... (8 Replies)
Discussion started by: sargotrons
8 Replies

8. UNIX for Dummies Questions & Answers

Match sum of values in each column with the corresponding column value present in trailer record

Hi All, I have a requirement where I need to find sum of values from column D through O present in a CSV file and check whether the sum of each Individual column matches with the value present for that corresponding column present in the trailer record. For example, let's assume for column D... (9 Replies)
Discussion started by: tpk
9 Replies

9. UNIX for Beginners Questions & Answers

Sum the values in the column using date column

I have a file which need to be summed up using date column. I/P: 2017/01/01 a 10 2017/01/01 b 20 2017/01/01 c 40 2017/01/01 a 60 2017/01/01 b 50 2017/01/01 c 40 2017/01/01 a 20 2017/01/01 b 30 2017/01/01 c 40 2017/02/01 a 10 2017/02/01 b 20 2017/02/01 c 30 2017/02/01 a 10... (6 Replies)
Discussion started by: Booo
6 Replies

10. UNIX for Beginners Questions & Answers

Copy columns from one file into another and get sum of column values and row count

I have a file abc.csv, from which I need column 24(PurchaseOrder_TotalCost) to get the sum_of_amounts with date and row count into another file say output.csv abc.csv- UTF-8,,,,,,,,,,,,,,,,,,,,,,,,, ... (6 Replies)
Discussion started by: Tahir_M
6 Replies
pthread_key_create(3T)													    pthread_key_create(3T)

NAME
pthread_key_create(), pthread_key_delete() - create or delete a thread-specific data key SYNOPSIS
PARAMETERS
key This is either a pointer to the location where the new key value will to be returned (create function) or the thread-spe- cific data key to be deleted (delete function). destructor Function to be called to destroy a data value associated with key when the thread terminates. DESCRIPTION
creates a unique thread-specific data key. The key may be used by threads within the process to maintain thread-specific data. The same key is used by all threads, but each thread has its own thread-specific value associated with key. For each thread, the value associated with key persists for the life of the thread. By default, the system allows a process to create up to number of thread-specific data keys. If the process needs more keys, the environ- ment variable can set the number of keys up to a maximum of 16384. If a value outside the range of and 16384 is specified, or if the envi- ronment variable is not set at all, the default value is considered. When a new thread-specific data key is created, each thread will ini- tially have the value NULL associated with the new key. Each time a thread is created, the new thread will have the value NULL for each thread-specific data key that has been created in the process. A thread may use to change the value associated with a thread-specific data key. Note: is an opaque data type. When a thread terminates, it may have non-NULL values associated with some or all of its thread-specific data keys. Typically, these val- ues will be pointers to dynamically allocated memory. If this memory is not released when the thread terminates, memory leaks in the process occur. An optional function may be provided at key creation time to destroy the thread-specific data of a terminating thread. When a thread terminates, the thread-specific data values associated with the thread will be examined. For each key that has a non-NULL thread-specific data value and a destructor function, the destructor function will be called with the thread-specific data value as its sole argument. The order in which destructor functions are called is unspecified. Once all the destructor functions have been called, the thread-specific data values for the terminating thread are examined again. If there are still non-NULL values in which the associated keys have destructor functions, the process of calling destructor functions is repeated. If after iterations of this loop there are still some non-NULL values with associated destructors, the system may stop calling the destructors or continue calling the destructors until there are no non-NULL values. Note: This may result in an infinite loop. If a destructor function is not desired for key, the value NULL may be passed in the destructor parameter. The function deletes a thread-specific data key. The key must have been previously created by The thread-specific data values associated with key are not required to be NULL when this function is called. Using key after it has been deleted results in undefined behavior. If a destructor function is associated with key, it will not be invoked by the function. Once key has been deleted, any function that was associated with key is not called when a thread exits. It is the responsibility of the application to free any application storage for each of the threads using key. The function can be called from a destructor function. RETURN VALUE
If successful, and return zero. Otherwise, an error number is returned to indicate the error (the variable is not set). ERRORS
If any of the following occur, the function returns the corresponding error number: The value specified by key is invalid. The necessary resources to create another thread-specific data key are not available, or the total number of keys per process has exceeded or an invalid value was specified with the environment variable There is insufficient memory available in which to create key. For each of the following conditions, if the condition is detected, the function returns the corresponding error number: The value specified by key is invalid. AUTHOR
and were derived from the IEEE POSIX P1003.1c standard. SEE ALSO
pthread_getspecific(3T), pthread_setspecific(3T). STANDARDS CONFORMANCE
Pthread Library pthread_key_create(3T)
All times are GMT -4. The time now is 09:09 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy