Processing Multiple Files


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Processing Multiple Files
# 1  
Old 11-30-2008
Processing Multiple Files

Hello Everyone,

I am new to scripting and confused with how to do this efficiently. I am trying to use AWK to do this.

I have a lot of files in a folder which has the data of my throughput measurements in two columns i.e. Serial # and Throughput. like this
177.994 847.9
178.996 858.0
179.996 847.8
180.992 851.2
.
.
.
I want to do the following with these files.

1. Separate the second column, so I only have the Throughput column.
2. Calculate the Max/Min/Average and Standard Deviation of that Column.
3. Save the Min, Max, Average, SD in a separate file with the FileName from which they were calculated

The file names are like this

p1-x20-y0-w
p1-x20-y0-wo
p1-x20-y10-w
p1-x20-y10-wo
p1-x20-y20-w
p1-x20-y20-wo
.
.
.
values with 'y' change from 0 to 100 in steps of 10
values with 'x' change from 20 to 40 in steps of 10

I have to submit a project report and tons to data to process before that. Can anybody help me with that?

Thanks,
Shahjehan
# 2  
Old 12-01-2008
Which part are you having trouble with? Show us your code so far.

Processing a large number of files is no problem, you can just use a shell for loop around it like:

Code:
for f in p1*
do
    awk 'your awk script' $f > $f.awked
done

Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Processing multiple files (environment setting)

Hello I posted on here a while ago about processing multiple files within a script. See original post below: I have a program cfxfrwb which is designed to remove headers from reports files. The cfxfrwb is located in the following directory /u01/efin/v40/live/bin I run the program against... (8 Replies)
Discussion started by: captainrhodes
8 Replies

2. Shell Programming and Scripting

Processing multiple files

Hello I have a program cfxfrwb which is designed to remove headers from reports files. The cfxfrwb is located in the following directory /u01/efin/v40/live/bin I run the program against a single report file in the temp directory and it does it's job../cfxfrwb... (2 Replies)
Discussion started by: captainrhodes
2 Replies

3. Shell Programming and Scripting

Passing multiple files to awk for processing in bash script

Hi, I'm using awk command in bash script. I'm able to pass multiple files to awk for processing.The code i can use is as below(sample code) #!/bin/bash awk -F "," 'BEGIN { ... ... ... }' file1 file2 file3 In the above code i'm passing the file names manually and it is fine till my... (7 Replies)
Discussion started by: shree11
7 Replies

4. Shell Programming and Scripting

Processing multiple files awk

hai i need my single awk script to act on 4 trace files of ns2 and to calculate througput and it should print result from each trace file in a single trace file. i tried with the following code but it doesnt work awk -f awkscript inputfile1 inputfile2 inputfile3 inputfile4>outputfile ... (4 Replies)
Discussion started by: sarathyy
4 Replies

5. Shell Programming and Scripting

multiple groups of files processing

I have five directories, dir1 to dir5 for each directory, I have all same number-named folders. There are four types of folders, {1..10}, {20..30}, { 40..50}, {60..70} Now for each types of folder, I will do the same thing, here is the code for i in {1..5} do cd dir$i mkdir temp1 for... (5 Replies)
Discussion started by: ksgreen
5 Replies

6. UNIX for Dummies Questions & Answers

single output of awk script processing multiple files

Helllo UNIX Forum :) Since I am posting on this board, yes, I am new to UNIX! I read a copy of "UNIX made easy" from 1990, which felt like a making a "computer-science time jump" backwards ;) So, basically I have some sort of understanding what the basic concept is. Problem Description:... (6 Replies)
Discussion started by: Kasimir
6 Replies

7. UNIX for Dummies Questions & Answers

Script Problem-Processing multiple scripts

Hi All, I have four scripts to run. My 1st script will make script2 and script3 to run. I am setting a cron job for this script1 to run continuously. This script1 will check for 2 text files and based on the existance of those text files it will initiate the script2 and script3. Now my doubt... (2 Replies)
Discussion started by: RSC1985
2 Replies

8. UNIX for Dummies Questions & Answers

Multiple excel files processing on unix

Hi all, I am faced with a rather unusual problem regarding interaction between NT and UNIX. I am using an ETL (Extract-Transform-Load) tool on unix that has the capability to read .xls files. So, when I FTP an excel (.xls) file from a windows server to unix and attempt to read it with this... (3 Replies)
Discussion started by: ucode_2482
3 Replies

9. UNIX for Dummies Questions & Answers

multiple file processing

I need to process a directory which will have a different amount of files in it from time to time. This is an error directory, I need to process each file indvidually in to one log file, then display the file to the user. So I would like to display the file name as well in the log file. example... (4 Replies)
Discussion started by: jagannatha
4 Replies
Login or Register to Ask a Question