07-02-2014
Since you didn't say anything about whether or not your input file was sorted, the earlier suggestions had to make the assumption that lines in your 20,000,000 line file were in random order. Therefore, all of the key values and the sums of the corresponding 2nd fields had to be kept in memory until the entire file had been read. Then the totals could be printed for each of the different keys present in the file. The error messages you got say that awk ran out of memory trying to accumulate all of the data.
Now that we know that all of the lines with a given key are adjacent in your input file, the later scripts could print a sum for each key as soon as a new key was found. Very little memory is required to do that and your script runs much faster because it needs less system resources to get the job done.
These 2 Users Gave Thanks to Don Cragun For This Post:
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Hi,
I have a table in Db2 with data say
id_1 phase1
id_1 phase2
id_1 phase3
id_2 phase1
id_2 phase2
I need to concatenate the values like
id_1 phase1,phase2,phase3
id_2 phase1,phase2
I tried recursive query but in vain as the length of string to be concatenated in quite long. ... (17 Replies)
Discussion started by: jsaravana
17 Replies
2. Shell Programming and Scripting
Hi All,
I have a file which is having 3 columns as (string string integer)
a b 1
x y 2
p k 5
y y 4
.....
.....
Question:
I want get the unique value of column 2 in a sorted way(on column 2) and the sum of the 3rd column of the corresponding rows. e.g the above file should return the... (6 Replies)
Discussion started by: amigarus
6 Replies
3. Shell Programming and Scripting
Hi, my requirement is to sum values in a row.
eg:
input is: sum,value1,value2,value3,.....,value N
Required Output: sum,<summation of N values>
Please help me... (5 Replies)
Discussion started by: MrGopal666
5 Replies
4. Shell Programming and Scripting
Is it possible to remove redundant names in the 4th column?
input
cqWE 100 200 singapore;singapore
AZO 300 400 brazil;america;germany;ireland;germany
....
....
output
cqWE 100 200 singapore
AZO 300 400 brazil;america;germany;ireland (4 Replies)
Discussion started by: quincyjones
4 Replies
5. UNIX for Dummies Questions & Answers
Hello,
I am new to Linux environment , I working on Linux script which should send auto email based on the specific condition from log file. Below is the sample log file
Name m/c usage
abc xxx 10
abc xxx 20
abc xxx 5
xyz ... (6 Replies)
Discussion started by: asjaiswal
6 Replies
6. Shell Programming and Scripting
Hello out there,
file.txt:
comp51820_c1_seq1 42 N 0:0:0:0:0:0 1:0:0:0:0:0 0:0:0:0:0:0 3:0:0:0:0:0 0:0:0:0:0:0
comp51820_c1_seq1 43 N 0:0:0:0:0:0 0:1:0:0:0:0 0:0:0:0:0:0 0:3:0:0:0:0 0:0:0:0:0:0
comp51820_c1_seq1 44 N 0:0:4:0:3:1 0:0:1:9:0:0 10:0:0:0:0:0 0:3:3:2:2:6 2:2:2:5:60:3... (16 Replies)
Discussion started by: pathunkathunk
16 Replies
7. Shell Programming and Scripting
Hi,
I have a table to be imported for R as matrix or data.frame but I first need to edit it because I've got several lines with the same identifier (1st column), so I want to sum the each column (2nd -nth) of each identifier (1st column)
The input is for example, after sorted:
K00001 1 1 4 3... (8 Replies)
Discussion started by: sargotrons
8 Replies
8. UNIX for Dummies Questions & Answers
Hi All,
I have a requirement where I need to find sum of values from column D through O present in a CSV file and check whether the sum of each Individual column matches with the value present for that corresponding column present in the trailer record.
For example, let's assume for column D... (9 Replies)
Discussion started by: tpk
9 Replies
9. UNIX for Beginners Questions & Answers
I have a file which need to be summed up using date column.
I/P:
2017/01/01 a 10
2017/01/01 b 20
2017/01/01 c 40
2017/01/01 a 60
2017/01/01 b 50
2017/01/01 c 40
2017/01/01 a 20
2017/01/01 b 30
2017/01/01 c 40
2017/02/01 a 10
2017/02/01 b 20
2017/02/01 c 30
2017/02/01 a 10... (6 Replies)
Discussion started by: Booo
6 Replies
10. UNIX for Beginners Questions & Answers
I have a file abc.csv, from which I need column 24(PurchaseOrder_TotalCost) to get the sum_of_amounts with date and row count into another file say output.csv
abc.csv-
UTF-8,,,,,,,,,,,,,,,,,,,,,,,,,
... (6 Replies)
Discussion started by: Tahir_M
6 Replies
LEARN ABOUT HPUX
pthread_key_delete
pthread_key_create(3T) pthread_key_create(3T)
NAME
pthread_key_create(), pthread_key_delete() - create or delete a thread-specific data key
SYNOPSIS
PARAMETERS
key This is either a pointer to the location where the new key value will to be returned (create function) or the thread-spe-
cific data key to be deleted (delete function).
destructor
Function to be called to destroy a data value associated with key when the thread terminates.
DESCRIPTION
creates a unique thread-specific data key. The key may be used by threads within the process to maintain thread-specific data. The same
key is used by all threads, but each thread has its own thread-specific value associated with key. For each thread, the value associated
with key persists for the life of the thread.
By default, the system allows a process to create up to number of thread-specific data keys. If the process needs more keys, the environ-
ment variable can set the number of keys up to a maximum of 16384. If a value outside the range of and 16384 is specified, or if the envi-
ronment variable is not set at all, the default value is considered. When a new thread-specific data key is created, each thread will ini-
tially have the value NULL associated with the new key. Each time a thread is created, the new thread will have the value NULL for each
thread-specific data key that has been created in the process. A thread may use to change the value associated with a thread-specific data
key. Note: is an opaque data type.
When a thread terminates, it may have non-NULL values associated with some or all of its thread-specific data keys. Typically, these val-
ues will be pointers to dynamically allocated memory. If this memory is not released when the thread terminates, memory leaks in the
process occur. An optional function may be provided at key creation time to destroy the thread-specific data of a terminating thread.
When a thread terminates, the thread-specific data values associated with the thread will be examined. For each key that has a non-NULL
thread-specific data value and a destructor function, the destructor function will be called with the thread-specific data value as its
sole argument. The order in which destructor functions are called is unspecified.
Once all the destructor functions have been called, the thread-specific data values for the terminating thread are examined again. If
there are still non-NULL values in which the associated keys have destructor functions, the process of calling destructor functions is
repeated. If after iterations of this loop there are still some non-NULL values with associated destructors, the system may stop calling
the destructors or continue calling the destructors until there are no non-NULL values. Note: This may result in an infinite loop.
If a destructor function is not desired for key, the value NULL may be passed in the destructor parameter.
The function deletes a thread-specific data key. The key must have been previously created by The thread-specific data values associated
with key are not required to be NULL when this function is called. Using key after it has been deleted results in undefined behavior.
If a destructor function is associated with key, it will not be invoked by the function. Once key has been deleted, any function that was
associated with key is not called when a thread exits. It is the responsibility of the application to free any application storage for
each of the threads using key.
The function can be called from a destructor function.
RETURN VALUE
If successful, and return zero. Otherwise, an error number is returned to indicate the error (the variable is not set).
ERRORS
If any of the following occur, the function returns the corresponding error number:
The value specified by
key is invalid.
The necessary resources to create another thread-specific
data key are not available, or the total number of keys per process has exceeded or an invalid value was specified
with the environment variable
There is insufficient memory available in which to create
key.
For each of the following conditions, if the condition is detected, the function returns the corresponding error number:
The value specified by
key is invalid.
AUTHOR
and were derived from the IEEE POSIX P1003.1c standard.
SEE ALSO
pthread_getspecific(3T), pthread_setspecific(3T).
STANDARDS CONFORMANCE
Pthread Library pthread_key_create(3T)