Extract numbers from text file work out average


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Extract numbers from text file work out average
# 8  
Old 04-22-2010
Quote:
Originally Posted by scottn
$NF is the value of the last field
Smilie
# 9  
Old 04-22-2010
Got it, the following did it:

Code:
#!/bin/sh
#echo "please enter the name of the file you wish to analyse:"
#read filetoint
cd logs/db
ls -l logs/db
cat stats.txt
awk '{sum+=$3}END{print sum/NR}' stats.txt

amazing! cheers Smilie
# 10  
Old 04-22-2010
FWIW using Scottn's solutions on Ubuntu (using mawk) I get:

Code:
$ awk '!(T+= $NF) END { print T/NR }' infile
awk: line 1: syntax error at or near END

$ awk '{T+= $NF} END { print T/NR }' infile
1517.5

# 11  
Old 04-22-2010
Quote:
Originally Posted by Scrutinizer
On Ubuntu (using mawk) I get:

Code:
$ awk '!(T+= $NF) END { print T/NR }' infile
awk: line 1: syntax error at or near END

$ awk '{T+= $NF} END { print T/NR }' infile
1517.5

Glad I changed it then Smilie
# 12  
Old 04-23-2010
Code:
#!/bin/sh
#echo "please enter the name of the file you wish to analyse:"
#read filetoint
cd logs/db
ls -l logs/db
cat stats.txt
awk '{sum+=$3}END{print sum/NR}' stats.txt

I'd like to expand this to look at all text files (which have the same format) in the directory and run the awk alg. against them - I've tried a for loop but what do I replace the stats.txt part of the awk with?

---------- Post updated at 10:13 PM ---------- Previous update was at 09:56 AM ----------

need some advice on this script, hope someone can help:

Code:
#!/bin/sh

directory=test/logs/etc

cd $directory

FILES="*.txt"

for f in "$FILES"

do

    echo "stats at `date`:"
    echo "slowest response time from db was `cut -f9 -d" " $f | sort -n | tail -1` ms"
    echo "fastest response time from db was `cut -f9 -d" " $f | sort -n | head -1` ms"
    echo "average response time from db was `awk '{sum+=$9}END{print sum/NR}' $f` ms"

#every hour, output to file in format:
#DATE TIME - MIN - MAX - AVG

done

what I now need it to do is write to a file every hour in the following format:

DATE TIME - MIN - MAX - AVG

I also want it to write to a new file each day is that possible?
# 13  
Old 04-23-2010
Quote:
Originally Posted by rich@ardz
...I'd like to expand this to look at all text files (which have the same format) in the directory and run the awk alg. against them - I've tried a for loop but what do I replace the stats.txt part of the awk with?
...
With the value of the iterator variable of your for loop.

Code:
$ 
$ # list all text file in current directory
$ ls -1 *.txt
file1.txt
file2.txt
file3.txt
file4.txt
$ 
$ # show their contents
$ perl -lne 'print $.==1 ? "\n== $ARGV ==\n$_" : $_; close ARGV if eof' *.txt

== file1.txt ==
abcd e544f222e0jk55 1503
abcd e544f222e0jk56 1504
abcd e544f222e0jk57 1505

== file2.txt ==
abcd e544f222e0jk58 1506
abcd e544f222e0jk59 1507
abcd e544f222e0jk60 1508

== file3.txt ==
abcd e544f222e0jk61 1509
abcd e544f222e0jk62 1510
abcd e544f222e0jk63 1511

== file4.txt ==
abcd e544f222e0jk64 1512
abcd e544f222e0jk65 1513
abcd e544f222e0jk66 1514
abcd e544f222e0jk67 1515
$ 
$ # display the contents of the shell script that loops through these
$ # text files and calculates the average of the 3rd field for each
$ # of them
$ 
$ cat -n script1.sh
     1    #!/bin/bash
     2    for f in *.txt; do
     3      avg=$(awk '{sum += $3} END {print sum/NR}' $f)
     4      echo "Average for file: $f = $avg"
     5    done
$ 
$ # run the shell script
$ . script1.sh
Average for file: file1.txt = 1504
Average for file: file2.txt = 1507
Average for file: file3.txt = 1510
Average for file: file4.txt = 1513.5
$ 
$

tyler_durden
# 14  
Old 08-04-2010
Code:
awk '{T+= $NF} END { print T/NR }' text2

remove the ! and try it out.

Last edited by Scott; 08-04-2010 at 08:34 AM.. Reason: Code tags
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Average result of text file

Hi all I have created a script to allow me to gain a number of file times and how long they take to execute. I have been asked to get the average time for the log. The log will update every 5 minutes and never be empty. 141 152 157 161 168 169 171 179 202 207 229 651 666 714... (3 Replies)
Discussion started by: simpsa27
3 Replies

2. Shell Programming and Scripting

Extract numbers from file name-how to ?

Hello friends,I am new to Unix programming. how do I achieve the following in Unix shell script (I am running ksh on AIX) extract the number from name of file? My file format is like "LongFileName-1234.020614-221030.txt" now I want to extract value which is between (-) hyphen and (.) dot... (4 Replies)
Discussion started by: f150
4 Replies

3. UNIX for Dummies Questions & Answers

Extract Numbers from a log file

Hi, I am trying to grep/extract the number list from this log file, can I get some help on this. I can grep the word 'href' to see the numbers, but it is resulting with the complete line. Content of my file: <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN"> <html> <head>... (4 Replies)
Discussion started by: Sajjadmehdi
4 Replies

4. Shell Programming and Scripting

Color prompt with file numbers does not work anymore

I have used this color prompt on my servers for long time, in file ~\.bashrc Black="\" Dark="\" Blue="\" LBlue="\" Green="\" LGreen="\" Cyan="\" LCyan="\" Red="\" LRed="\" Purple="\" LPurple="\" Brown="\" Yellow="\" LGray="\" White="\" Reset="\" PS1="$Yellow\u@\h $LBlue\w... (4 Replies)
Discussion started by: Jotne
4 Replies

5. Shell Programming and Scripting

Need to extract only decimal numbers for a glob of text

If you have a look at this thread, you'll see that users have been posting the output a script which are numbers that range from 2 to 5 decimal places. If I dump this entire thread to txt file, how can I: 1) Delete everything except for numbers of the following formats (where 'x' is a digit and... (5 Replies)
Discussion started by: graysky
5 Replies

6. UNIX for Dummies Questions & Answers

Extracting lines from a text file based on another text file with line numbers

Hi, I am trying to extract lines from a text file given a text file containing line numbers to be extracted from the first file. How do I go about doing this? Thanks! (1 Reply)
Discussion started by: evelibertine
1 Replies

7. UNIX Desktop Questions & Answers

Calculate average for rows in a text file

Dear Gurus, I have tab-delimited text files with matrix containing values. The first column is a identifier and other columns have the corresponding values. I would like to calculate the average value (total number/number of entries) for all entries from 2nd column to the last column in row... (3 Replies)
Discussion started by: Unilearn
3 Replies

8. Shell Programming and Scripting

Need help please with Grep/Sed command to extract text and numbers from a file

Hello All, I need to extract lines from a file that contains ALPHANUMERIC and the length of Alphanumeric is set to 16. I have pasted the sample of the lines from the text file that I have created. My problem is that sometimes 16 appears in other part of the line. I'm only interested to... (14 Replies)
Discussion started by: mnassiri
14 Replies

9. UNIX for Dummies Questions & Answers

Extract numbers from .txt file

I need to extract all the p-value numbers and the rho numbers from a .txt file and write them as coma separated values in a new file. Ideally I would get two files in the end, one for p- values and one for rho. Any suggestions? I appreciate your help!!! The .txt file looks essentially like this... (5 Replies)
Discussion started by: eggali
5 Replies

10. Shell Programming and Scripting

BASH. Need to extract some numbers and take the average

Hey all, I ran some simulations, of which the output is 100s of files. I've used grep to extract the vital information needed from the files. This has made my task somewhat easier. But I still need to perform some mathematical calculations (average and geometrical average) on the results of the... (9 Replies)
Discussion started by: slackjack
9 Replies
Login or Register to Ask a Question