averaging specific column of multiple files


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting averaging specific column of multiple files
# 1  
Old 12-28-2011
averaging specific column of multiple files

Hi all, I'm needing help again on scripting. I have weekly files with 3 columns, and I need to do monthly averaging on the values on column 3, the file naming convention is as follows:
Code:
20000105.u- 2000:year 01:month 05:day
20000112.u
20000119.u
20000126.u
20000202.u
20020209.u

I need to do a loop that will read the month part of each filename and do the average of column 3 for all the files falling on the same month. the desired output will be something like:
Code:
200001.u containing 3 columns still:
cols.1 & 2: the values are the same for all files and so just need to append them to the final output
col.3: average of col. 3 from 20000105.u - 20000126.u

Suggestions are very much welcome and appreciated. thanks much.

On the side, I had another question partly unrelated to the ones on top. Is there a way to change the filename on a directed output automatically, like for instance in the pseudo-code below. thanks again
Code:
list="$(ls *.u)"
for i in $list; do
    command > "$i".out
done

directed output name: file.u.out
desired output name: file.out

thanks a lot,Smilie
# 2  
Old 12-28-2011
Code:
ksh
for i in *.u
do mv $i "${i%.u}.out"
done

This User Gave Thanks to ctsgnb For This Post:
# 3  
Old 12-28-2011
Quote:
Originally Posted by ida1215
On the side, I had another question partly unrelated to the ones on top. Is there a way to change the filename on a directed output automatically, like for instance in the pseudo-code below. thanks again
Code:
list="$(ls *.u)"
for i in $list; do
    command > "$i".out
done

directed output name: file.u.out
desired output name: file.out

thanks a lot,Smilie
Code:
  command > "${i%.*}.out"

--ahamed
This User Gave Thanks to ahamed101 For This Post:
# 4  
Old 12-28-2011
I might be being daft myself, but i don't quite understand your need. Do you
  1. have a set of files with many records and for each of them you want to get the average of column 3?
  2. have a set of files with a single record and you want to get the average between column three for a given pair of files?
If it's option a, I could write a cheap & nasty shell script, but I'm sure an awk would be far superior. If it is options b then it might be simpler as a script.

Code:
#/bin/ksh
read col1 col2 f1col3 rest < $1
read col1 col2 f2col3 rest < $2
((tot=$f1col3+$f2col3))
((avg=$tot/2))
echo "The average for column 3 in $1 & $2 is $avg

Of course, this doesn't handle the half and isn't very flexible. You could multiply the $avg by 2 and if it's different to $tot, then display the extra half.

Can you elaborate on your needs a bit better?





On the second point, if you have a files called filenames.u.out and you want to create filenames.out from them, you could try something like:-

Code:
#!/bin/ksh
for source in *.u.out
do
   target="${source%.u.out}.out"
   mv $source $target
done

The definition of target is a way to chop off the end of a string, to which I then appended .out again. I might have the wrong end of the stick again I suppose.




Robin
Liverpool/Blackburn
UK
# 5  
Old 12-28-2011
thanks a bunch. worked seamlessly,Smilie
# 6  
Old 12-28-2011
You are welcome. What worked for what problem?



Robin
# 7  
Old 12-28-2011
For the averaging part, do you provide an input? i.e. get me the average for this particular month or do you want to averga out for all the files present in a directory by month?

--ahamed
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

How do I extract specific column in multiple csv files?

file1: Name,Threshold,Curr Samples,Curr Error%,Curr ART GETHome,100,21601,0.00%,47 GETregistry,100,21592,0.00%,13 GEThomeLayout,100,30466,0.00%,17 file2: Name,Threshold,Curr Samples,Curr Error%,Curr ART GETHome,100,21601,0.00%,33 GETregistry,100,21592,0.00%,22... (6 Replies)
Discussion started by: Raghuram717
6 Replies

2. UNIX for Beginners Questions & Answers

How to copy a column of multiple files and paste into new excel file (next to column)?

I have data of an excel files as given below, file1 org1_1 1 1 2.5 100 org1_2 1 2 5.5 98 org1_3 1 3 7.2 88 file2 org2_1 1 1 2.5 100 org2_2 1 2 5.5 56 org2_3 1 3 7.2 70 I have multiple excel files as above shown. I have to copy column 1, column 4 and paste into a new excel file as... (26 Replies)
Discussion started by: dineshkumarsrk
26 Replies

3. Shell Programming and Scripting

Sum values of specific column in multiple files, considering ranges defined in another file

I have a file (let say file B) like this: File B: A1 3 5 A1 7 9 A2 2 5 A3 1 3 The first column defines a filename and the other two define a range in that specific file. In the same directory, I have also three more files (File A1, A2 and A3). Here is 10 sample lines... (3 Replies)
Discussion started by: Bastami
3 Replies

4. Shell Programming and Scripting

How to print multiple specific column after a specific word?

Hello.... Pls help me (and sorry my english) :) So I have a file (test.txt) with 1 long line.... for example: isgc jsfh udgf osff 8462 error iwzr 653 idchisfb isfbisfb sihfjfeb isfhsi gcz eifh How to print after the "error" word the 2nd 4th 5th and 7th word?? output well be: 653 isfbisfb... (2 Replies)
Discussion started by: marvinandco
2 Replies

5. Shell Programming and Scripting

Converting Single Column into Multiple rows, but with strings to specific tab column

Dear fellows, I need your help. I'm trying to write a script to convert a single column into multiple rows. But it need to recognize the beginning of the string and set it to its specific Column number. Each Line (loop) begins with digit (RANGE). At this moment it's kind of working, but it... (6 Replies)
Discussion started by: AK47
6 Replies

6. Shell Programming and Scripting

Loop for row-wise averaging of multiple files using awk

Hello all, I need to compute a row-wise average of files with a single column based on the pattern of the filenames. I really appreciate any help on this. it would just be very difficult to do them manually as the rows are mounting to 100,000 lines. the filenames are as below with convention as... (2 Replies)
Discussion started by: ida1215
2 Replies

7. Shell Programming and Scripting

Averaging 3 files with multiple rows

Hi, I am trying to average the values from 3 files with the same format. They are very large files so I will describe the file and show some it of. Basically the file has 83 columns (with nearly 7000 rows). The first three columns are the same for each file while the remaining 80 are values that... (1 Reply)
Discussion started by: kylle345
1 Replies

8. Shell Programming and Scripting

How to averaging column based on first column values

Hello I have file that consist of 2 columns of millions of entries timestamp and throughput I want to find the average (throughput ) for each equal timestamp before change it to proper format e.g : i want to average 2 coloumnd fot all 1308154800 values in column 1 and then print... (4 Replies)
Discussion started by: aadel
4 Replies

9. Shell Programming and Scripting

Averaging multiple columns

Hello, I am trying to average multiple columns simultaneously while skipping the first column. I am using this awk line to average one column awk '{sum+=$3} END { print "Average = ",sum/NR}' But I want to be able to do it for multiple columns while skipping the first column. There... (4 Replies)
Discussion started by: gisele_l
4 Replies

10. Shell Programming and Scripting

averaging column values with awk

Hello. Im just starting to learn awk so hang in there with me...I have a large text file formatted as such everything is in a single column ID001 value 1 value 2 value....n ID002 value 1 value 2 value... n I want to be able to calculate the average for values for each ID from the... (18 Replies)
Discussion started by: johnmillsbro
18 Replies
Login or Register to Ask a Question