Taking the averages of columns with deletion of some lines


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Taking the averages of columns with deletion of some lines
# 1  
Old 05-12-2013
Taking the averages of columns with deletion of some lines

Hi,

I am in stage of post processing some of my results. I wanted to plot the data against the three axis x,y,z. The data file is quite complicated and i have to take the average of x, y,z over different steps of my test. A typical file look like below:

HTML Code:
Time taken:4s
No.of series : 3
Step:1
ID   X       Y      z
1    0.1    0.4   0.45
2    1.2   -1.2   0.25
3   -0.5   -1.2   0.26
3    0.8   -2.1   1.45
1    1.2   -8.2   0.25
2    0.25   1.2   0.25
Time taken:5s
No.of series : 3
Step:2
ID   X       Y      z
3    1.0    0.1    0.5
1   -2.0   -0.2    0.25
2   -0.4   -0.2   -0.60
2   -0.1   -1.1   -0.45
3    0.2   -0.2    0.25
1    1.1   -1.2    0.25
What i wanted to do?
From the above data, i wanted to get the average of values of X , Y , Z (column 2,3,4) along with Column 1 with title as their corresponding step number (i.e) Step.

The thing is that original file is so big that the number of IDs here shown as 3 is actually 600 and the steps are around 25000 which is shown here as two steps.I wanted to delete these following lines (which is displayed inbetween the steps ) while doing this automatically :

HTML Code:
Time taken:5s
No.of series : 3
Expected Output : I am in need of an output as a file, which exactly looks like this:

HTML Code:
Step:1 
ID    X         Y        z
1    0.65    -3.9    0.35
2    0.725    0      0.25
3    0.15    -1.65  0.855
Step:2
ID   X         Y      z
1    -0.45  -0.7     0.25
2   -0.25   -0.65   -0.525
3    0.6     -0.05    0.375

I would much thankfull for your immediate inputs on making a script for achieving the above result.
# 2  
Old 05-12-2013
Put this in "script.awk":
Code:
/^Step/
/^ID/
/^Time/ && p {
    for (i in c) {
      print i" "x[i]/c[i]" "y[i]/c[i]" "z[i]/c[i]
      c[i]=0
      x[i]=0
      y[i]=0
      z[i]=0
    }
}
/^[0-9]/ {
  c[$1]++
  x[$1]+=$2
  y[$1]+=$3
  z[$1]+=$4
  p=1
}
END {
  for (i in c) {
    print i" "x[i]/c[i]" "y[i]/c[i]" "z[i]/c[i]
  }
}

Then run:
Code:
awk -f script.awk datafile

This User Gave Thanks to bartus11 For This Post:
# 3  
Old 05-12-2013
hi thanks for your reply. But it works properly but my intention is NOT fulfilled. the script displays the average for x,y,z columns all over the file. But, what i need to find the Averages of X,Y,Z at particular steps separately and save in one file as shown in EXPECTED OUTPUT.

Here , when i run the script, it display only one set of X,Y,Z average values for both the steps combined. But, i wanted to get the averages at step 1 and a separate average at step2.
# 4  
Old 05-12-2013
Can you post the output that you are getting? I get something like this:
Code:
# awk -f script.awk datafile
Step:1
ID   X       Y      z
1 0.65 -3.9 0.35
2 0.725 0 0.25
3 0.15 -1.65 0.855
Step:2
ID   X       Y      z
1 -0.45 -0.7 0.25
2 -0.25 -0.65 -0.525
3 0.6 -0.05 0.375

This User Gave Thanks to bartus11 For This Post:
# 5  
Old 05-12-2013
Hi,
i am extremely sorry. I get the same output as you got for the data i pasted. But when i run for actual data , it takes all averages for all timesteps. The actual data which looks like this :

HTML Code:
SOURCE: TIMESTEP
1
SOURCE: NUMBER OF SERIES
594
ITEM: SHAPE CONSTRAIN pp pp pp
-10 30
-10 30
-10 30
ITEM: SERIES id type x y z 
300 1 -0.018638 -2.55234 -6.32752 
100 1 1.41254 -2.58375 -6.49904 
200 1 -2.83075 0.0568085 -6.31853 
100 1 -2.13227 -1.15906 -6.22779 
200 1 -0.72638 -1.26406 -6.59486 
300 1 0.0490859 0.005266 -6.90442 
SOURCE: TIMESTEP
2
SOURCE: NUMBER OF SERIES
594
ITEM: SHAPE CONSTRAIN pp pp pp
-10 30
-10 30
-10 30
ITEM: SERIES id type x y z 
100 1 -0.458893 -1.85236 -6.76382 
200 1 2.44809 -1.75353 -6.94784 
300 1 1.69774 -0.511574 -6.93282 
300 1 -2.26969 -5.05726 -4.42174 
200 1 0.629752 -5.01888 -4.87928 
100 1 2.07906 -4.80346 -4.90814  

Issue : When i run the script for this, it takes average of X,Y,Z for all the timestep. I would be really thankful, if I get separate X,Y,Z averages for corresponding TIMESTEPs.

And i do not want this get displayed in the output:

HTML Code:
SOURCE: NUMBER OF SERIES
594
ITEM: SHAPE CONSTRAIN pp pp pp
-10 30
-10 30
-10 30
# 6  
Old 05-12-2013
Try this:
Code:
/^SOURCE: TIMESTEP/ {
  if (p) {
    for (i in c) {
      print i" "x[i]/c[i]" "y[i]/c[i]" "z[i]/c[i]
      c[i]=0
      x[i]=0
      y[i]=0
      z[i]=0
    }
  }
  print
  getline
  print
}
/^[0-9]/ && NF==5 {
  c[$1]++
  x[$1]+=$3
  y[$1]+=$4
  z[$1]+=$5
  p=1
}
END {
  for (i in c) {
    print i" "x[i]/c[i]" "y[i]/c[i]" "z[i]/c[i]
  }
}

This User Gave Thanks to bartus11 For This Post:
# 7  
Old 05-12-2013
thank you so much. You have really helped me a lot. I would be grateful , if it is possible for you to explain shortly how it works.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Merging multiple lines to columns with awk, while inserting commas for missing lines

Hello all, I have a large csv file where there are four types of rows I need to merge into one row per person, where there is a column for each possible code / type of row, even if that code/row isn't there for that person. In the csv, a person may be listed from one to four times... (9 Replies)
Discussion started by: RalphNY
9 Replies

2. How to Post in the The UNIX and Linux Forums

Daily averages...

I have date file like below.. 1995 1 2 10 29 38.6706 -6.53823 41.9201 1995 1 2 10 29 -49.2477 -4.59733 17.2704 1995 1 2 10 29 -49.2369 -4.48045 8.61348 1995 1 3 8 48 -42.2643 ... (3 Replies)
Discussion started by: athithi
3 Replies

3. UNIX for Dummies Questions & Answers

Taking the average of two columns and printing it on a new column

Hi, I have a space delimited text file that looks like the following: Aa 100 200 Bb 300 100 Cc X 500 Dd 600 X Basically, I want to take the average of columns 2 and 3 and print it in column 4. However if there is an X in either column 2 or 3, I want to print the non-X value. Therefore... (11 Replies)
Discussion started by: evelibertine
11 Replies

4. Shell Programming and Scripting

Moving Averages SMA !

Hello, I am trying to create a awk script to calculate the simple moving average of two fields but I got only the result of the first field. Could you please help me to fix this awk shell script awk -F, -v points=5 ' { a = $2; b = $3; if(NR>=points) { total_a = 0; total_b... (1 Reply)
Discussion started by: csierra
1 Replies

5. Shell Programming and Scripting

Calculate Averages !

Hi, I have a file with more than 2,000 rows like this: 05/26/2011,1200,1500 I would like to create a awk shell script that calculate the price average of the second and third field each 5,10 and 20 rows or ask me for the values, starting at first row. Finally compare the average value... (1 Reply)
Discussion started by: csierra
1 Replies

6. Shell Programming and Scripting

Comparison and deletion of lines between two files

Hi i need an help. I have two files list1 and list2, both contains the server names i want to delete the servers in list2 which were also found in list1. for an eg list2 list1 oradg1 oradg4 oradg2 oradg2 oradg3 ... (5 Replies)
Discussion started by: sudharson
5 Replies

7. Shell Programming and Scripting

Taking 3 greped/awked lines of text

Say I used grep and awk for taking text out of a text file... I now /have 3 lines of text that I want to combine to make 1 line. What command could I use to do this? ... display line 1 twice then sed s/$line1/$line2/ ????... display line 2 twice then sed s/$line2/$line3/ (2 Replies)
Discussion started by: puttster
2 Replies

8. Shell Programming and Scripting

Operations on a file with Deletion , Modification and Insertion of lines

Hi All , Given a file , I need to delete , modify and insert lines matching certain patterns in that file using shell scripting. e.g. If a file FILE1 has following content : CUST.ABC.DEF = CUST.ABC.DEF * CUST.ABC.DEF PRINTF(CUST.ABC.DEF) CUST.ABC.DEF = mid(CUST.ABC.DEF,10.34)... (5 Replies)
Discussion started by: swapnil.nawale
5 Replies

9. UNIX for Dummies Questions & Answers

deletion of all lines ends with :

I have below file say temp1 BSCAJM1:HWJA10C BSCAJM1: BSCALW1: BSCALW1:GVND01B BSCALW1: BSCALW1: BSCBKNR:IJNMKTA BSCBKNR: BSCJOD1: BSCJOD1:JOD121B i want to delete all the lines ending with : and have below output BSCAJM1:HWJA10C BSCALW1:GVND01B BSCBKNR:IJNMKTA... (8 Replies)
Discussion started by: lalchand
8 Replies

10. Shell Programming and Scripting

Deletion of lines in a text file

Hi Everyone, Please help me with this. I have gone through many posts here but couldn't find what I wanted. I have a file with 79000+ lines and I want to delete lines in a pattern. I want to delete every 141st line in the file, starting from line 2000 till 50000. Please help guys. ... (8 Replies)
Discussion started by: max29583
8 Replies
Login or Register to Ask a Question