I try to avoid loops in awk to speed things up. If you have gnu awk, you can do this:
Code:
awk '{a+=$1} END {print a}' RS=" |\n" file?
1779
While there may be combinations of AWK implementation and operating system on which your suggestion is faster, I compared it against its predecessor on two combinations and yours was slower everytime.
For each of the following results, the best of 5 runs was chosen.
Cygwin/GAWK 4.1.0:
Code:
$ time gawk '{for(i=1;i<=NF;i++)t+=$i} END {print t}' data
500000500000
real 0m1.359s
user 0m1.327s
sys 0m0.015s
$ time gawk '{a+=$1} END {print a}' RS=' |\t|\n' data
500000500000
real 0m2.797s
user 0m2.796s
sys 0m0.030s
Linux/MAWK 1.3.3:
Code:
$ time mawk '{for(i=1;i<=NF;i++)t+=$i} END {print t}' data
5e+11
real 0m0.753s
user 0m0.640s
sys 0m0.032s
$ time mawk '{a+=$1} END {print a}' RS=' |\t|\n' data
5e+11
real 0m1.346s
user 0m1.268s
sys 0m0.012s
In my opinion, unless there is a confirmed performance issue and unless the AWK implementation is known, unqualified AWK optimization tips are usually a bad idea (doubly so when advising a novice who is more likely to blindly internalize the advice).
Different awk implementations, and even different versions of the same implementation, implement differing sets of optimization strategies. One example I ran into recently: gawk lazily recomputes $0. As you probably know, POSIX requires recomputing $0 whenever a field is modified. gawk will not perform that recomputation until $0 is referenced (if at all). That optimization in effect:
Code:
$ time gawk '{for (i=1;i<=NF;i++) $i=""}' data
real 0m0.594s
user 0m0.593s
sys 0m0.030s
$ time mawk '{for (i=1;i<=NF;i++) $i=""}' data
real 0m1.039s
user 0m0.900s
sys 0m0.060s
Even though it is MAWK who has the speedy reputation, this version of GAWK is much faster because it doesn't recompute $0 after each $i="" (since $0 is never referenced after a field modification, it is never recomputed).
Regards,
Alister
These 2 Users Gave Thanks to alister For This Post:
This was very interesting, and an eye opener. I have never tested this, just thought i many be solver to run ting in loop. This prove it many be wrong.
Thanks for taking time to test.
I need help with this assignment. I'm very new to using UNIX/LINUX, and my only previous experience with programing anything is using python.
We are writing scripts using vim, and this one I'm stumped on.
"Write a shell script that finds and display the sum of even positive integers from 0 to... (5 Replies)
I want to count the number of lines, I need this result be a number, and sum the last numeric column, I had done to make this one at time, but I need to make this for a crontab, so, it has to be an script, here is my lines:
It counts the number of lines:
egrep -i String file_name_201611* |... (5 Replies)
Hi
i have to calculate some numbers, column by column.
Herfore i used a for-loop..
for i in {4..26};do awk -F"," '{x'$i'+=$'$i'}END{print '$i'"\t" x'$i'}' file.tmp;done
----- printout -----
4 660905240
5 71205272
6 8.26169e+07
7 8.85961e+07
8 8.60936e+07
9 7.42238e+07
10 5.6051e+07... (7 Replies)
Hi,
I want to count the number of occurrences of numbers from a file of 6,000,000 lines. Because its too large, I decided to split the counts up in multiple files. So I have files of the counts of 5,000 lines. Now I want to add up the counts of all those files.
The "counts file" looks like... (9 Replies)
I basically have a file where I had to do a bunch of greps to get a list of numbers
example: a file called numbers.txt
10000
10000
superman
10000
batman
10000
10000
grep '100' * |
10000
10000
10000
10000
10000 (2 Replies)
cat *.out |grep "<some text>" | awk '{print $6}'
For ex,This will reutrn me
11111
22222
is it possible to add these two numbers in the above given command itself?I can write this to a file and find the sum.
But I prefer to this calculation in the above given line itself.
Any... (3 Replies)
I have 11 directories with around 200 files in each. In each directory the files are labeled out.0 through out.201 . Each file has around 118 numbers in a single column. I need to sum the files in each directory so each directory will have a resultant vector that is 118 numbers long. I then... (5 Replies)
hello im looking for short way to sum numbers from stdout the way i found to do it is to long for me i wander if there is shorter way to do it
ok it 2 stage action
this will make the list of number in to file sum.txt
grep -c include *.c | awk '{l=split($0,a,":");print a;}' > sum.txt
this... (1 Reply)
Hi,
i want to sum all nubers in one column. Example:
12.23
11
23.01
3544.01
I'm trying to do this in awk, but it doesn't work properly.
Seems like awk is summing only integers, for example:
12
11
23
3544
It cuts off numbers after dot.
I used this command:
akw /text/ file.txt |nawk... (1 Reply)
10-10-2013
prefer loops and statements...More clear for me..