Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Sum value from selected lines script (awk,perl)

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Sum value from selected lines script (awk,perl)
# 8  
Old 10-12-2009
First of all THANK YOU VERY MUCH to durden_tyler and to Scrutinizer.
I need some time to elaborate(and study) your examples, and in some
days (if you want) I' ll came back with some other questions (but I realise
I need to REALLY study some manuals on awk/Perl) to obtain the
data filtering objects on which I'm working now.
Thanks again.


Quote:
Originally Posted by Scrutinizer
Using awk:

Case1:
Code:
awk '{ sum1[int($1/500)]+=$2 } END { for (i in sum1) print "Sum1 "sum1[i] } ' infile

Case2:
Code:
awk 'BEGIN{
       min=1
     }
     { time[NR]=$1
       val[NR]=sum2[NR]=$2
       i=min
       while (time[NR]-time[i]>=500)
         i++
       min=i
       for (i=min;i<NR;i++)
         sum2[NR]+=val[i]
     }
     END {
       for (i in sum2)
         print "Sum2: "sum2[i]
     }' infile

Case1+2 combined:
Code:
awk 'BEGIN{
       min=1
     }
     { sum1[int($1/500)]+=$2
       time[NR]=$1
       val[NR]=sum2[NR]=$2
       i=min
       while (time[NR]-time[i]>=500)
         i++
       min=i
       for (i=min;i<NR;i++)
         sum2[NR]+=val[i]
     }
     END {
       for (i in sum1)
         print "Sum1 "sum1[i]
       print ""
       for (i in sum2)
         print "Sum2: "sum2[i]
     }' infile

Original testset:
Code:
Sum1 17
Sum1 205
Sum1 96

Sum2: 4
Sum2: 17
Sum2: 218
Sum2: 279
Sum2: 301

Additional testset:
Code:
Sum1 10
Sum1 18

Sum2: 1
Sum2: 3
Sum2: 6
Sum2: 10
Sum2: 15
Sum2: 20
Sum2: 25



---------- Post updated 10-12-09 at 04:41 PM ---------- Previous update was 10-11-09 at 09:01 PM ----------

---------- Post updated at 08:04 PM ---------- Previous update was at 08:02 PM ----------

$ gunzip -c ES_05_10_Oct2009.gz |sed -e 's/BEST /BEST/g' -e 's/[:.]//g'|awk '{ sum1[int($1/500)]+=$4 } END { for (i in sum1) print "i="i"Su
m1 "sum1[i] }'|tee outfile.txt
awk: (FILENAME=- FNR=4094608) fatal: format_tree: obuf: can't allocate 512 bytes of memory (Cannot allocate memory)

Is there a way to do the sum step by step avoiding an array structure?
(problem with very big data structures)
Thanks

Paolo

Quote:
Originally Posted by Scrutinizer
Using awk:

Case1:
[code]awk '{ sum1[int($1/500)]+=$2 } END { for (i in sum1) print "Sum1 "sum1[i] } ' infile
[COLOR="#738fbf"]

---------- Post updated at 08:01 PM ---------- Previous update was at 04:41 PM ----------
# 9  
Old 10-12-2009
Hi paolfili,

Sure:

Code:
awk '{ if ( int($1/500) > curtime ) {
         if ( NR>1 ) {  print "Sum1: "cursum }
         curtime=int($1/500)
         cursum=0
       }
       cursum+=$2
     }
     END { print "Sum1: "cursum
     }' infile

one line:
Code:
awk '{t=int($1/500); if(t>rt) {if (NR>1) {print "Sum: "s}; rt=t; s=0} s+=$2} END {print "Sum: "s}' infile

Last edited by Scrutinizer; 10-12-2009 at 07:03 PM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Shell script count lines and sum numbers from multiple files

I want to count the number of lines, I need this result be a number, and sum the last numeric column, I had done to make this one at time, but I need to make this for a crontab, so, it has to be an script, here is my lines: It counts the number of lines: egrep -i String file_name_201611* |... (5 Replies)
Discussion started by: Elly
5 Replies

2. UNIX for Dummies Questions & Answers

awk to sum column field from duplicate row/lines

Hello, I am new to Linux environment , I working on Linux script which should send auto email based on the specific condition from log file. Below is the sample log file Name m/c usage abc xxx 10 abc xxx 20 abc xxx 5 xyz ... (6 Replies)
Discussion started by: asjaiswal
6 Replies

3. Shell Programming and Scripting

Summing over specific lines and replacing the lines with the sum using sed, awk

Hi friends, This is sed & awk type question. I have a text file which has numbers spread all over the file. I want to sum the series of numbers whenever i find it and produce an output file with the sum. For example ###start of input text file #### abc def ghi 1 2 3 4 kjld random... (3 Replies)
Discussion started by: kaaliakahn
3 Replies

4. Shell Programming and Scripting

AWK script - extracting min and max values from selected lines

Hi guys! I'm new to scripting and I need to write a script in awk. Here is example of file on which I'm working ATOM 4688 HG1 PRO A 322 18.080 59.680 137.020 1.00 0.00 ATOM 4689 HG2 PRO A 322 18.850 61.220 137.010 1.00 0.00 ATOM 4690 CD ... (18 Replies)
Discussion started by: grincz
18 Replies

5. Shell Programming and Scripting

awk script for getting the selected records from a file.

Hello, I have attached one file named file.txt . I have to create a file using the awk script with the records in which 38th position is P and not V . ex it should have 00501 HOLTSVILLE NYP00501 and it should not include 00501 I R S SERVICE CENTER ... (3 Replies)
Discussion started by: sonam273
3 Replies

6. Shell Programming and Scripting

trying to print selected fields of selected lines by AWK

I am trying to print 1st, 2nd, 13th and 14th fields of a file of line numbers from 29 to 10029. I dont know how to put this in one code. Currently I am removing the selected lines by awk 'NR==29,NR==10029' File1 > File2 and then doing awk '{print $1, $2, $13, $14}' File2 > File3 Can... (3 Replies)
Discussion started by: ananyob
3 Replies

7. Shell Programming and Scripting

Perl script to find particular field and sum it

Hi, I have a file with format a b c d e 1 1 2 2 2 1 2 2 2 3 1 1 1 1 2 1 1 1 1 4 1 1 1 1 6 in column e i want to find all similar fields ( with perl script )and sum it how many are there for instance in format above. 2 - 2 times 4 - 1 time 6 - 1 time what i use is ... (14 Replies)
Discussion started by: Learnerabc
14 Replies

8. Shell Programming and Scripting

shell script(Preferably awk or sed) to print selected number of columns from each row

Hi Experts, The question may look very silly by seeing the title, but please have a look at it clearly. I have a text file where the first 5 columns in each row were supposed to be attributes of a sample(like sample name, number, status etc) and the next 25 columns are parameters on which... (3 Replies)
Discussion started by: ks_reddy
3 Replies

9. Shell Programming and Scripting

Sum of all lines in file without roundup with awk

Hi, I have a file and I want to sum all the numbers in it. Example of the file: 0.6714359 -3842.59553830551 I used your forum (https://www.unix.com/shell-programming-scripting/74293-how-get-sum-all-lines-file.html) and found a script, what worked for me: awk '{a+=$0}END{print a}'... (8 Replies)
Discussion started by: mario8eren
8 Replies

10. UNIX for Dummies Questions & Answers

extracting selected few lines through perl

How can I extract few lines(like 10 to 15, top 10 and last 10) from a file using perl. I do it with sed, head and tail in unix scripting. I am new to perl. Appreciate your help. (2 Replies)
Discussion started by: paruthiveeran
2 Replies
Login or Register to Ask a Question