summing numbers in files


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting summing numbers in files
# 8  
Old 07-31-2008
Tools can you use the bc command?

Perhaps try modifying my previous script...

Code:
counter=0
while read zf
   do
   counter=$((counter+1))
# may want to adjust this
   val1=$(echo "$zf")
   val2=$(cat file2 | head -"$counter" | tail -1)
#   val3=$((val1+val2))
   val3=$(echo $val1 + $val2 | bc)
   echo "$val3" >>file3
done <file1

I am doing something a little different in the line making val3.
# 9  
Old 07-31-2008
joey, it doesn't like the syntax for the while loop. I added paren's but it gave the same error. if i am missing something obvious you will need to help me out because Im still very new at this
# 10  
Old 07-31-2008
The Joeyg's script is for ksh or bash shell.
If you run it under csh, you will get errors.
Add a shebang to she script :
Code:
#!/usr/bin/ksh
counter=0
while read zf
   do
   counter=$((counter+1))
# may want to adjust this
   val1=$(echo "$zf")
   val2=$(cat file2 | head -"$counter" | tail -1)
#   val3=$((val1+val2))
   val3=$(echo $val1 + $val2 | bc)
   echo "$val3" >>file3
done <file1

Jean-Pierre.
# 11  
Old 07-31-2008
Hammer & Screwdriver yep... I think aigles is correct

Last night I was just free-typing off the top of my head, and forgot to include that line. And yes, I normally code for bash.

Code:
> cat file1
12.50
19.123
127
256.5
10.1234
> cat file2
45
609.12
19.456
105.101
56.65
> add_em
> cat file3
57.50
628.243
146.456
361.601
66.7734

Code:
> cat add_em
#! /usr/bin/bash

#needed to clear out the results file first
rm file3 2>/dev/null

counter=0
while read zf
   do
   counter=$((counter+1))
# may want to adjust this
   val1=$(echo "$zf")
   val2=$(cat file2 | head -"$counter" | tail -1)
   val3=$(echo $val1 + $val2 | bc)
   echo "$val3" >>file3
done <file1

# 12  
Old 07-31-2008
I already have a version of this program written in Fortran 90, but since it is opening and reading from 40401 files (each about 77kb) it ends up taking between 1.5 hrs and 3.5 hrs depending on the server load at that time. By writing it in csh I was hoping to speed up the process. While joeyg's method works well it seems like it will take about the same amount of time. The other methods were significantly faster, is there anyway to get those methods to output the full results or speed up the code supplied by joeyg?
# 13  
Old 07-31-2008
Quote:
Originally Posted by pattywac
... There is a total of 5192 numbers in each file and I want to add them row by row... ie. first row of file 1 + first row of file 2 = first row of output. Eventually I will be summing 40401 of these files together ....
Quote:
Originally Posted by pattywac
... both cut off the numbers after the first number after the decimal point(some go to 13 decimal places). I need most of these numbers in full, is this an easy fix?
If you have Time, RAM and CPU you can try this script:
Code:
#!/bin/bash
declare -a ARRAY            
for i in $(ls file*)                 # for files from file00001 to file40401
do
   arr_tmp=($(tr '\n' ' ' < $i))     # push file to array
   cnt=0
   while [ $cnt -le 5192 ] ;
      do
        ARRAY[$cnt]=${ARRAY[$cnt]:-0}"+"${arr_tmp[$cnt]}
        ((cnt++))
      done
done
for ((i=0; i<${#ARRAY[@]};i++))
do
        echo "scale=13;${ARRAY[$i]}" | bc >> output
done

# 14  
Old 07-31-2008
The following script paste and sum recursivelyfiles by group of 5 (GroupingFactor).
Work files are created in /tmp (WorkFilePrefix).
Code:
GroupingFactor=5
WorkFilePrefix=/tmp/work.$$

sum() {
   local    _inputFileList=$1
   local -i _level=$((${2:-0} + 1))

   local _workFilePrefix=${WorkFilePrefix}.${_level}
   local _outFileCount=0
   local _outFileList=${_workFilePrefix}.filelist
   local _fileList _outFile

   > ${_outFileList}
   xargs -a $_inputFileList -n 5 |
   while read _fileList
   do
      _outFile=${_workFilePrefix}.$((++_outFileCount))
      paste $_fileList |
      awk '{ sum=0 ; for(i=1; i<=NF; i++) sum += $i ; print sum}' > $_outFile
      echo $_outFile >> $_outFileList
      (( _level > 1 )) && rm -f $_fileList
   done

   if (( $(wc -l < ${_outFileList}) > 1 ))
   then
      sum $_outFileList $_level
   else
      cat $(<${_outFileList})
   fi
   rm -f ${_workFilePrefix}.*
}

if [ -z "$1" ]
then
   in=${WorkFilePrefix}.0.filelist
   cat - >$in
   sum $in
   rm -f $in
else
   sum $1
fi

The input files (50 files with same content) :
Code:
> ls pattywac_datas/
file.1   file.13  file.17  file.20  file.24  file.28  file.31  file.35  file.39  file.42  file.46  file.5   file.8
file.10  file.14  file.18  file.21  file.25  file.29  file.32  file.36  file.4   file.43  file.47  file.50  file.9
file.11  file.15  file.19  file.22  file.26  file.3   file.33  file.37  file.40  file.44  file.48  file.6
file.12  file.16  file.2   file.23  file.27  file.30  file.34  file.38  file.41  file.45  file.49  file.7
> cat pattywac_datas/file.1
1
2
3
4
5
6
7
8
9
10
> ls -l /tmp
total 0
>

Summing files:
Code:
> ls pattywac_datas/* | pattywac.sh
50
100
150
200
250
300
350
400
450
500
> ls -l /tmp
total 0
>

or
Code:
> pattywac.sh ./filelist
50
100
150
200
250
300
350
400
450
500
>

Jean-Pierre.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Summing up values of rows of numbers

data file contains failed=24 error=23 error=163 failed=36 error=903 i need to get a total count of each value above. i'm looking for the most efficient method to do this as the datafile i provided is just a sample. the actual data can be several hundred thousands of lines. so from... (3 Replies)
Discussion started by: SkySmart
3 Replies

2. Shell Programming and Scripting

Sum of numbers in three or more files

I have files : cat file1 15 88 44 667 33 4cat file2 445 66 77 3 56 (12 Replies)
Discussion started by: Natalie
12 Replies

3. Shell Programming and Scripting

How to take a Average of numbers from different files?

Hi, I have 3 to 4 different files, from that i need to take a Average of numbers from a particular column. here i have to take 4th column, that should present in diff. file. File 1: Col1 col2 col3 col4 1 11 sa 12.00 2 22 sb 134.59 3 33 sc 11.99 4 44 sd 12.44 Col1 col2 col3... (8 Replies)
Discussion started by: Shenbaga.d
8 Replies

4. Shell Programming and Scripting

summing from two different files

I have two files hhhh 3674.00 a 75 1535 183 2134 291 2452 442 2738 704 3048 a 1007 3549 1282 4413 1494 5001 1631 5217 1954 5610 a 2540 5832 3248 6080 3629 6264 4851 6600 7004 6985 ... (4 Replies)
Discussion started by: Indra2011
4 Replies

5. Shell Programming and Scripting

Multiply numbers from different files

Hi All, I have tried few things with this but it did not help much. I have some 200,000 files in a directory. There are two sets of files. 1. Files with extension .dat with file names like these (1.dat, 2.dat, 5.dat, 8.dat....200000.dat) 2. Another set of files with .txt extension and... (5 Replies)
Discussion started by: shoaibjameel123
5 Replies

6. Shell Programming and Scripting

Summing numbers after specific word

Hi all, Looking for suggestions on a better way to sum numbers in a key value pair formated file. What I have works but seems really clunky to me. Any suggestions would be greatly appreciated. cat test.txt | perl -ne 'm/(M=)(\d+\.?\d?\d?)/ && print "$2\n"' | awk '{ sum+=$1} END {printf... (7 Replies)
Discussion started by: cgol
7 Replies

7. Shell Programming and Scripting

Sum Numbers from different files

Hi All, I need to print the sum of numbers from different files. Input files: file1.out 10 20 30 file2.out 10 20 30 (5 Replies)
Discussion started by: saint2006
5 Replies

8. Shell Programming and Scripting

Reading several files and summing their content line-by-line

Hey, I am gettin a bit crazy with my script. I have several input datas with the same name (5.ill) in different folders (daysim_01, daysim_02, etc.). The 4. column of each of the data has to be summed with each other and then hass to be written in one new file. So file1: 1 1 0 1 2 1 1 2 ... (7 Replies)
Discussion started by: ergy1983
7 Replies

9. UNIX for Dummies Questions & Answers

To get unique numbers from two files

here i have two files: file 1 1 2 3 4 5 5 6 7 8 9 file 2 4 5 6 6 8 8 (6 Replies)
Discussion started by: i.scientist
6 Replies

10. Shell Programming and Scripting

altering numbers in files

I want to change a number in a file into number -1.. for instance file_input is fdisdlf_s35 fdjsk_s27 fsdf_s42 jkljllljkkl_s57 ... etc now i want the output to be fdisdlf_s34 fdjsk_s26 fdsf_s41 jkljllljkkl_s56 ... etc I was think of using "sed -e 's/2/1/g' -e 's/3/2/g' -e... (4 Replies)
Discussion started by: bigboizvince
4 Replies
Login or Register to Ask a Question