calculate diff in csv


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting calculate diff in csv
# 1  
Old 02-11-2009
calculate diff in csv

I wrote a func to calculcate integers difference in csv.

getSegLatency() {
latencies=0$1
E2E=0`echo $2 | sed 's/\.000000//g'`
integer segLatency=0
set -A arr `echo $latencies | sed 's/\.000000//g' | sed 's/,/ /g'`
res=${arr[0]}
integer i=1
while ((i < ${#arr[*]})); do
((segLatency=${arr[$i]} - ${arr[$i - 1]}));
res="$res,$segLatency"
(( i = i + 1));
done
((segLatency=$E2E - ${arr[$i - 1]}));
res="$res,$segLatency"
echo $res
}

in which the number of elements in $1 may vary
e.g. when i call like:
getSegLatency 1.000000,4.000000,6.000000,7.000000 11.000000
i'll get:
3,2,1,4

It works well but the performance is slow.

Is there any way to polish the script to run faster?
# 2  
Old 02-11-2009
Code:
$ perl -ne '@times=split /,/; for ($i=1;$i<$#times;$i++){print $times[$i]-$times[$i-1],",";}print $times[$i]-$times[$i-1]; print "\n";'

Reads from stdin or a file.
# 3  
Old 02-11-2009
Quote:
Originally Posted by csmklee
I wrote a func to calculcate integers difference in csv.

Please put code inside [code] tags.
Quote:
Code:
getSegLatency() {
latencies=0$1
E2E=0`echo $2 | sed 's/\.000000//g'`


External commands are slow; there's no need for sed:

Code:
E2E=0${2%.*}

Quote:
Code:
integer segLatency=0


There is no standard command integer (and it's unnecessary).
Quote:
Code:
set -A arr `echo $latencies | sed 's/\.000000//g' | sed 's/,/ /g'`


No need for sed.

If you are using bash or ksh93, you can do that with:

Code:
IFS=,
arr=( $latencies )
IFS=$' \t\n'

Quote:
Code:
res=${arr[0]}
integer i=1
while ((i < ${#arr[*]})); do


The standard syntax is:

Code:
while [ $i -lt ${#arr[*]} ]; do

Quote:
Code:
((segLatency=${arr[$i]} - ${arr[$i - 1]}));


The standard syntax is:

Code:
segLatency=$(( ${arr[$i]} - ${arr[$i - 1]} ))

Quote:
Code:
res="$res,$segLatency"
(( i = i + 1));


The standard syntax is:

Code:
i=$(( $i + 1 ))

Quote:
Code:
done
((segLatency=$E2E - ${arr[$i - 1]}));


See previous note.
Quote:
Code:
res="$res,$segLatency"
echo $res
}

in which the number of elements in $1 may vary
e.g. when i call like:
getSegLatency 1.000000,4.000000,6.000000,7.000000 11.000000
i'll get:
3,2,1,4

With your code and that input, I get:
Code:
01,3,2,1,2

Quote:

It works well but the performance is slow.

Is there any way to polish the script to run faster?

It seems reasonably snappy to me, despite the unnecessary use of sed

Code:
getSegLatency() {
  IFS=,
  set -- $*
  IFS=$' \t\n'
  result=$(
    while [ $# -ge 2 ]
    do
      printf "%d," $(( (${1%.*} - ${2%.*}) * -1 ))
      shift
   done )
   printf "%s\n" ${result%,}
}


Last edited by cfajohnson; 02-11-2009 at 05:20 PM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Calculate Time diff in milli milliseconds(Time format : HH:MM:SS,NNN)

Hi All, I have one file which contains time for request and response. I want to calculate time difference in milliseconds for each line. This file can contain 10K lines. Sample file with 4 lines. for first line. Request Time: 15:23:45,255 Response Time: 15:23:45,258 Time diff... (6 Replies)
Discussion started by: Raza Ali
6 Replies

2. Shell Programming and Scripting

How to calculate average of csv using shell scripting?

Hi, I need to calculate the average of the following values using shell scripitng. Can anyone please suggest a solution? ... (10 Replies)
Discussion started by: karan pratap si
10 Replies

3. Shell Programming and Scripting

How to calculate avg values of csv file using shell scripting .?

hi all i have a reporting work and i want it to be automated using shell scripting kindly let me know how can i make that possibe . eg data are :... (2 Replies)
Discussion started by: Avinash shaw
2 Replies

4. Shell Programming and Scripting

Compare two CSV files and put the difference in third file with line no,field no and diff value.

I am having two csv files i need to compare these files and the output file should have the information of the differences at the field level. For Example, File 1: A,B,C,D,E,F 1,2,3,4,5,6 File 2: A,C,B,D,E,F 1,2,4,5,5,6 out put file: (12 Replies)
Discussion started by: karingulanagara
12 Replies

5. Shell Programming and Scripting

Script for Diff b/w 2 CSV files

Hello friends, I have two CSV files.. I am looking to create a script which automatically 'diff' between those 2 csv files each month..Can anyone help me in creating the automatic script which will find the differences between 2 csv files. Appreciated your help. Thanks. (4 Replies)
Discussion started by: bobby1015
4 Replies

6. Shell Programming and Scripting

compare 2 CSV fields from same diff output

Attached is a file called diff.txt It is the output from this command: diff -y --suppress-common-lines --width=5000 1.txt 2.txt > diff.txt I have also attached 1.txt and 2.txt for your convenience. Both 1.txt and 2.txt contain one very long CSV string. File 1.txt is a CSV dump of... (0 Replies)
Discussion started by: gvolpini
0 Replies

7. Shell Programming and Scripting

serach diff filename in diff location using shell scripting

Hi, I am new to shell scripting. please help me to find out the solution. I need a script where we need to read the text file(consists of all file names) and get the file names one by one and append the date suffix for each file name as 'yyyymmdd' . Then search each file if exists... (1 Reply)
Discussion started by: Lucky123
1 Replies

8. Shell Programming and Scripting

Calculate average from CSV file using PERL script

Hi All I have this csv file and I need to calculate the average of FPS. FPS:27.7420, Interval:1314184238772 FPS:25.9798, Interval:1314184242646 FPS:27.4772, Interval:1314184246311 FPS:26.1623, Interval:1314184250159 FPS:26.4515, Interval:1314184253972 FPS:31.5896, Interval:1314184257163... (24 Replies)
Discussion started by: sayachop
24 Replies

9. Shell Programming and Scripting

.procmailrc and uudeview (put attachments from diff senders to diff folders)

Moderator, please, delete this topic (1 Reply)
Discussion started by: optik77
1 Replies

10. Shell Programming and Scripting

diff 2 files; output diff's to 3rd file

Hello, I want to compare two files. All records in file 2 that are not in file 1 should be output to file 3. For example: file 1 123 1234 123456 file 2 123 2345 23456 file 3 should have 2345 23456 I have looked at diff, bdiff, cmp, comm, diff3 without any luck! (2 Replies)
Discussion started by: blt123
2 Replies
Login or Register to Ask a Question