Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

any way to speed up calculations in bash script

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting any way to speed up calculations in bash script
# 1  
Old 01-15-2009
any way to speed up calculations in bash script
hi i have a script that is taking the difference of multiple columns in a file from a value from a single row..so far i have a loop to do that.. all the data is floating point..fin[i] has the difference between array1 and array2..array1 has 700 x 300= 210000 values and array2 has 700 values..

Code:
while [ $j -lt $outer ];do
b=${ARRAY2[j]}
i=0
while [ $i -lt $count1 ];do
a=${ARRAY1[k]}
fin[i]=`echo $a - $b | bc`
printf "%.1f," ${fin[i]} >> "$count".diff
((i++))
((k++))
done 
echo >> "$count".diff
((j++))
done

i am using bc to calculate the floating point subtraction. problem is i have about 600 columns and 300 rows..the script takes a long time to execute.. is there any way i can speed it up?
# 2  
Old 01-15-2009
Quote:
Originally Posted by npatwardhan
is there any way i can speed it up?

Don't call the external command 21,000 times. Pipe your calculations to a single instance of it.

And I would use awk rather than bc.

Code:
while [ $j -lt $outer ];do
  i=0
  while [ $i -lt $count1 ];do
    printf "%f\n" "${ARRAY1[k]} - ${ARRAY2[j]}"
    i=$(( $i + 1 ))
    k=$(( $k + 1 ))
  done 
  echo
  j=$(( $j + 1 ))
done | awk '/^$/ { print; next }
{ printf "%.1f," $1 - $2 }
'  > "$count.diff"

If you are processing the entire arrays:

Code:
for a in "${ARRAY1[@]}"
do
  printf "%f $a\n" ${ARRAY2[@]}
  echo
done | awk '/^$/ { print; next }
{ printf "%.1f,", $1 - $2 }
'

# 3  
Old 01-15-2009
thanks. i am using bc because the numbers are floating point.. i also tried:

Code:
while [ $j -lt $outer ];do
  i=0
  while [ $i -lt $count1 ];do
    printf "%f\n" "${ARRAY1[k]} - ${ARRAY2[j]}"
    i=$(( $i + 1 ))
    k=$(( $k + 1 ))
  done 
  echo
  j=$(( $j + 1 ))
done | awk '/^$/ { print; next }
{ printf "%.1f," $1 - $2 }
'  > "$count.diff"

and i got an error:
Code:
./lines1: line 77: printf: -0.5422 - : invalid number
./lines1: line 77: printf: -0.5422 - : invalid number
./lines1: line 77: printf: -0.5422 - : invalid number
./lines1: line 77: printf: -0.5422 - : invalid number
./lines1: line 77: printf: -0.5422 - : invalid number
./lines1: line 77: printf: -0.5422 - : invalid number
./lines1: line 77: printf: -0.5422 - : invalid number
./lines1: line 77: printf: -0.5422 - : invalid number
./lines1: line 77: printf: -0.5422 - : invalid number
./lines1: line 77: printf: -0.5422 - : invalid number

seemed like it was taking the difference but errored out..do i still need bc?
# 4  
Old 01-15-2009
Quote:
Originally Posted by npatwardhan
seemed like it was taking the difference but errored out..do i still need bc?
printf cannot subtract numbers for you, that's what bc is for.
# 5  
Old 01-15-2009
Quote:
Originally Posted by npatwardhan
thanks. i am using bc because the numbers are floating point.. i also tried:

Code:
while [ $j -lt $outer ];do
  i=0
  while [ $i -lt $count1 ];do
    printf "%f\n" "${ARRAY1[k]} - ${ARRAY2[j]}"

Code:
    printf "%f %f\n" "${ARRAY1[k]}" "${ARRAY2[j]}"

# 6  
Old 01-15-2009
that is just going to print the values.. i still need to take the difference.. is there any way i can round the data values in ARRAY1 and ARRAY2? this way i can get rid of the extra precision and speed up my calculations..
# 7  
Old 01-15-2009
Quote:
Originally Posted by npatwardhan
that is just going to print the values..

It prints the values which are piped to awk where the subtraction is performed.
Quote:
i still need to take the difference..

That's done in awk.
Quote:
is there any way i can round the data values in ARRAY1 and ARRAY2? this way i can get rid of the extra precision and speed up my calculations..

Changing the precision will not affect the speed.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Speed up extraction od tar.bz2 files using bash

The below bash will untar each tar.bz2 folder in the directory, then remove the tar.bz2. Each of the tar.bz2 folders ranges from 40-75GB and currently takes ~2 hours to extract. Is there a way to speed up the extraction process? I am using a xeon processor with 12 cores. Thank you :). ... (7 Replies)
Discussion started by: cmccabe
7 Replies

2. Shell Programming and Scripting

Speed up bash loop?

I am running the below bash loop on all the files of a specific type (highlighted in bold) in a directory. There are 4 awk commands that use the input files to search another and look for a match. The input files range from 27 - 259 and are a list of names. The file that is searched is... (11 Replies)
Discussion started by: cmccabe
11 Replies

3. Shell Programming and Scripting

Arithmetic calculations in bash file

I have 2 numbers xmin = 0.369000018 xmax = 0.569000006 and want to calculate (xmax- xmin) / 5.0 I have tried using $(( )) but is always giving an error (8 Replies)
Discussion started by: kristinu
8 Replies

4. Shell Programming and Scripting

Help with Arithmetic calculations in Shell script

Hi, I need a help with arithmetic calculations in my script. I have two variables: a=17; b=1712 I want to perform ($a/$b)*100 with two decimals in the result. I tried with following: res=$((100*a/b)) res=`echo "scale=2; $a / $b" | bc` But I am not getting the decimal values.... (4 Replies)
Discussion started by: karumudi7
4 Replies

5. Shell Programming and Scripting

Date Calculations using script!!

Hi all, Thanks in Advance , i am very new to programming part in script i think using some caluations+ sed command only we can do this process in script. for exampl: i have file in that one line is like this using sed i can replace the date and all but my requirement is The... (3 Replies)
Discussion started by: anishkumarv
3 Replies

6. Shell Programming and Scripting

bash script range calculations

Hi, I have data in the following form: AB001 10 AB002 9 AB003 9 etc AB200 5 What I need to do is sum up the second value according to groups of the first, i.e. AB001 to AB030 the total being X, AB031 to AB050 the total being Y etc (there are 5 AB ranges of different sizes). I'm sure... (3 Replies)
Discussion started by: chrissycc
3 Replies

7. Filesystems, Disks and Memory

data from blktrace: read speed V.S. write speed

I analysed disk performance with blktrace and get some data: read: 8,3 4 2141 2.882115217 3342 Q R 195732187 + 32 8,3 4 2142 2.882116411 3342 G R 195732187 + 32 8,3 4 2144 2.882117647 3342 I R 195732187 + 32 8,3 4 2145 ... (1 Reply)
Discussion started by: W.C.C
1 Replies

8. Shell Programming and Scripting

calculations in bash

HI i have following problem, i need to use split command to split files each should be cca 700 lines but i dont know how to inplement it in the scripts becasuse each time the origin file will be various size , any body got any idea cheers (2 Replies)
Discussion started by: kvok
2 Replies

9. Shell Programming and Scripting

Sed or awk script to remove text / or perform calculations from large CSV files

I have a large CSV files (e.g. 2 million records) and am hoping to do one of two things. I have been trying to use awk and sed but am a newbie and can't figure out how to get it to work. Any help you could offer would be greatly appreciated - I'm stuck trying to remove the colon and wildcards in... (6 Replies)
Discussion started by: metronomadic
6 Replies

10. Shell Programming and Scripting

Non-integer calculations in bash

I'm new at scripting but I thought I was getting pretty good at it. I've hit a snag. I try to use expr to compute a fraction say: expr 3 / 4, and I'm getting zero. I guess it's just truncating to the integer, in this case 0, but I need the decimal 0.75. What can I do to compute this value in... (2 Replies)
Discussion started by: jeriryan87
2 Replies
Login or Register to Ask a Question