07-02-2014
Since you didn't say anything about whether or not your input file was sorted, the earlier suggestions had to make the assumption that lines in your 20,000,000 line file were in random order. Therefore, all of the key values and the sums of the corresponding 2nd fields had to be kept in memory until the entire file had been read. Then the totals could be printed for each of the different keys present in the file. The error messages you got say that awk ran out of memory trying to accumulate all of the data.
Now that we know that all of the lines with a given key are adjacent in your input file, the later scripts could print a sum for each key as soon as a new key was found. Very little memory is required to do that and your script runs much faster because it needs less system resources to get the job done.
These 2 Users Gave Thanks to Don Cragun For This Post:
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Hi,
I have a table in Db2 with data say
id_1 phase1
id_1 phase2
id_1 phase3
id_2 phase1
id_2 phase2
I need to concatenate the values like
id_1 phase1,phase2,phase3
id_2 phase1,phase2
I tried recursive query but in vain as the length of string to be concatenated in quite long. ... (17 Replies)
Discussion started by: jsaravana
17 Replies
2. Shell Programming and Scripting
Hi All,
I have a file which is having 3 columns as (string string integer)
a b 1
x y 2
p k 5
y y 4
.....
.....
Question:
I want get the unique value of column 2 in a sorted way(on column 2) and the sum of the 3rd column of the corresponding rows. e.g the above file should return the... (6 Replies)
Discussion started by: amigarus
6 Replies
3. Shell Programming and Scripting
Hi, my requirement is to sum values in a row.
eg:
input is: sum,value1,value2,value3,.....,value N
Required Output: sum,<summation of N values>
Please help me... (5 Replies)
Discussion started by: MrGopal666
5 Replies
4. Shell Programming and Scripting
Is it possible to remove redundant names in the 4th column?
input
cqWE 100 200 singapore;singapore
AZO 300 400 brazil;america;germany;ireland;germany
....
....
output
cqWE 100 200 singapore
AZO 300 400 brazil;america;germany;ireland (4 Replies)
Discussion started by: quincyjones
4 Replies
5. UNIX for Dummies Questions & Answers
Hello,
I am new to Linux environment , I working on Linux script which should send auto email based on the specific condition from log file. Below is the sample log file
Name m/c usage
abc xxx 10
abc xxx 20
abc xxx 5
xyz ... (6 Replies)
Discussion started by: asjaiswal
6 Replies
6. Shell Programming and Scripting
Hello out there,
file.txt:
comp51820_c1_seq1 42 N 0:0:0:0:0:0 1:0:0:0:0:0 0:0:0:0:0:0 3:0:0:0:0:0 0:0:0:0:0:0
comp51820_c1_seq1 43 N 0:0:0:0:0:0 0:1:0:0:0:0 0:0:0:0:0:0 0:3:0:0:0:0 0:0:0:0:0:0
comp51820_c1_seq1 44 N 0:0:4:0:3:1 0:0:1:9:0:0 10:0:0:0:0:0 0:3:3:2:2:6 2:2:2:5:60:3... (16 Replies)
Discussion started by: pathunkathunk
16 Replies
7. Shell Programming and Scripting
Hi,
I have a table to be imported for R as matrix or data.frame but I first need to edit it because I've got several lines with the same identifier (1st column), so I want to sum the each column (2nd -nth) of each identifier (1st column)
The input is for example, after sorted:
K00001 1 1 4 3... (8 Replies)
Discussion started by: sargotrons
8 Replies
8. UNIX for Dummies Questions & Answers
Hi All,
I have a requirement where I need to find sum of values from column D through O present in a CSV file and check whether the sum of each Individual column matches with the value present for that corresponding column present in the trailer record.
For example, let's assume for column D... (9 Replies)
Discussion started by: tpk
9 Replies
9. UNIX for Beginners Questions & Answers
I have a file which need to be summed up using date column.
I/P:
2017/01/01 a 10
2017/01/01 b 20
2017/01/01 c 40
2017/01/01 a 60
2017/01/01 b 50
2017/01/01 c 40
2017/01/01 a 20
2017/01/01 b 30
2017/01/01 c 40
2017/02/01 a 10
2017/02/01 b 20
2017/02/01 c 30
2017/02/01 a 10... (6 Replies)
Discussion started by: Booo
6 Replies
10. UNIX for Beginners Questions & Answers
I have a file abc.csv, from which I need column 24(PurchaseOrder_TotalCost) to get the sum_of_amounts with date and row count into another file say output.csv
abc.csv-
UTF-8,,,,,,,,,,,,,,,,,,,,,,,,,
... (6 Replies)
Discussion started by: Tahir_M
6 Replies
LEARN ABOUT SUNOS
pthread_key_create
pthread_key_create(3C) pthread_key_create(3C)
NAME
pthread_key_create - create thread-specific data key
SYNOPSIS
cc -mt [ flag... ] file... -lpthread [ library... ]
#include <pthread.h>
int pthread_key_create(pthread_key_t *key, void (*destructor, void*));
This function creates a thread-specific data key visible to all threads in the process. Key values provided by pthread_key_create() are
opaque objects used to locate thread-specific data. Although the same key value may be used by different threads, the values bound to the
key by pthread_setspecific() are maintained on a per-thread basis and persist for the life of the calling thread.
Upon key creation, the value NULL is associated with the new key in all active threads. Upon thread creation, the value NULL is associ-
ated with all defined keys in the new thread.
An optional destructor function may be associated with each key value. At thread exit, if a key value has a non-NULL destructor pointer,
and the thread has a non-NULL value associated with that key, the function pointed to is called with the current associated value as its
sole argument. Destructors can be called in any order.
If, after all the destructors have been called for all keys with non-NULL values, there are still some keys with non-NULL values, the
process will be repeated. If, after at least PTHREAD_DESTRUCTOR_ITERATIONS iterations of destructor calls for outstanding non-NULL values,
there are still some keys with non-NULL values, the process is continued, even though this might result in an infinite loop.
If successful, the pthread_key_create() function stores the newly created key value at *key and returns 0. Otherwise, an error number is
returned to indicate the error.
The pthread_key_create() function will fail if:
EAGAIN The system lacked the necessary resources to create another thread-specific data key, or the system-imposed limit on the
total number of keys per process PTHREAD_KEYS_MAX has been exceeded.
ENOMEM Insufficient memory exists to create the key.
The pthread_key_create() function will not return an error code of EINTR.
See attributes(5) for descriptions of the following attributes:
+-----------------------------+-----------------------------+
| ATTRIBUTE TYPE | ATTRIBUTE VALUE |
+-----------------------------+-----------------------------+
|Interface Stability |Standard |
+-----------------------------+-----------------------------+
|MT-Level |MT-Safe |
+-----------------------------+-----------------------------+
pthread_getspecific(3C), pthread_setspecific(3C), pthread_key_delete(3C), attributes(5), standards(5)
23 Mar 2005 pthread_key_create(3C)