Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Computations on a text file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Computations on a text file
# 1  
Old 12-14-2011
Computations on a text file
hey,

I have text file which has some data for some device Id's. i want to perform AVG, MAX,2nd MAX on these device ids. The .txt file looks like below. please help me in finding the computations per Device id. My output file should contatin with DeviceID,Avg, max and 2nd max of the device ID.

***********************************************
bash-3.00$ cat temp.txt
[281207], (Mid) 100581038,(Rid) 200394032, (date) [1323664218], 2011.12.12-04.30.18, (Val) 0.0d
[281208], (Mid) 100581038,(Rid) 200394032, (date) [1323665119], 2011.12.12-04.45.19, (Val) 0.0d
[281209], (Mid) 100581038,(Rid) 200394034, (date) [1323662419], 2011.12.12-04.00.19, (Val) 0.0d
[281210], (Mid) 100581038,(Rid) 200394034, (date) [1323663318], 2011.12.12-04.15.18, (Val) 0.0d
[281211], (Mid) 100581038,(Rid) 200394034, (date) [1323664218], 2011.12.12-04.30.18, (Val) 0.0d
[281212], (Mid) 100581038,(Rid) 200394034, (date) [1323665119], 2011.12.12-04.45.19, (Val) 0.0d
[281213], (Mid) 100581038,(Rid) 200394035, (date) [1323662419], 2011.12.12-04.00.19, (Val) 0.0d
[281214], (Mid) 100581038,(Rid) 200394035, (date) [1323663318], 2011.12.12-04.15.18, (Val) 0.0d
[281215], (Mid) 100581038,(Rid) 200394035, (date) [1323664218], 2011.12.12-04.30.18, (Val) 0.0d
[281216], (Mid) 100581038,(Rid) 200394035, (date) [1323665119], 2011.12.12-04.45.19, (Val) 0.0d
[282093], (Mid) 100581049,(Rid) 200393955, (date) [1323662416], 2011.12.12-04.00.16, (Val) 0.0d
[282094], (Mid) 100581049,(Rid) 200393955, (date) [1323663317], 2011.12.12-04.15.17, (Val) 0.0d
[282095], (Mid) 100581049,(Rid) 200393955, (date) [1323664216], 2011.12.12-04.30.16, (Val) 0.0d
[282096], (Mid) 100581049,(Rid) 200393955, (date) [1323665115], 2011.12.12-04.45.15, (Val) 0.0d
[279065], (Mid) 100580981,(Rid) 200393809, (date) [1323666018], 2011.12.12-05.00.18, (Val) 0.0d
[279066], (Mid) 100580981,(Rid) 200393809, (date) [1323666918], 2011.12.12-05.15.18, (Val) 0.0d
[279067], (Mid) 100580981,(Rid) 200393809, (date) [1323667818], 2011.12.12-05.30.18, (Val) 0.0d
[279068], (Mid) 100580981,(Rid) 200393809, (date) [1323668718], 2011.12.12-05.45.18, (Val) 0.0d
[279215], (Mid) 100580982,(Rid) 200393809, (date) [1323667818], 2011.12.12-05.30.18, (Val) 0.0d
[279216], (Mid) 100580982,(Rid) 200393809, (date) [1323668718], 2011.12.12-05.45.18, (Val) 0.0d
[279217], (Mid) 100580982,(Rid) 200393815, (date) [1323666018], 2011.12.12-05.00.18, (Val) 0.0d
[279218], (Mid) 100580982,(Rid) 200393815, (date) [1323666918], 2011.12.12-05.15.18, (Val) 0.0d
[279219], (Mid) 100580982,(Rid) 200393815, (date) [1323667818], 2011.12.12-05.30.18, (Val) 0.0d
[279220], (Mid) 100580982,(Rid) 200393815, (date) [1323668718], 2011.12.12-05.45.18, (Val) 0.0d
[279221], (Mid) 100580982,(Rid) 200393826, (date) [1323666018], 2011.12.12-05.00.18, (Val) 0.0d
[279222], (Mid) 100580982,(Rid) 200393826, (date) [1323666918], 2011.12.12-05.15.18, (Val) 0.0d
[279223], (Mid) 100580982,(Rid) 200393826, (date) [1323667818], 2011.12.12-05.30.18, (Val) 0.0d
[279224], (Mid) 100580982,(Rid) 200393826, (date) [1323668718], 2011.12.12-05.45.18, (Val) 0.0d
[279225], (Mid) 100580982,(Rid) 200393828, (date) [1323666018], 2011.12.12-05.00.18, (Val) 0.0d
[279226], (Mid) 100580982,(Rid) 200393828, (date) [1323666918], 2011.12.12-05.15.18, (Val) 0.0d
[279227], (Mid) 100580982,(Rid) 200393828, (date) [1323667818], 2011.12.12-05.30.18, (Val) 0.0d
[279228], (Mid) 100580982,(Rid) 200393828, (date) [1323668718], 2011.12.12-05.45.18, (Val) 0.0d
[279229], (Mid) 100580982,(Rid) 200393852, (date) [1323666018], 2011.12.12-05.00.18, (Val) 0.0d
[279230], (Mid) 100580982,(Rid) 200393852, (date) [1323666918], 2011.12.12-05.15.18, (Val) 0.0d
[279231], (Mid) 100580982,(Rid) 200393852, (date) [1323667818], 2011.12.12-05.30.18, (Val) 0.0d
[279232], (Mid) 100580982,(Rid) 200393852, (date) [1323668718], 2011.12.12-05.45.18, (Val) 0.0d
[279233], (Mid) 100580982,(Rid) 200393855, (date) [1323666018], 2011.12.12-05.00.18, (Val) 0.0d
[279354], (Mid) 100580982,(Rid) 200394082, (date) [1323666918], 2011.12.12-05.15.18, (Val) 0.0d
[279355], (Mid) 100580982,(Rid) 200394082, (date) [1323667818], 2011.12.12-05.30.18, (Val) 0.0d
[279356], (Mid) 100580982,(Rid) 200394082, (date) [1323668718], 2011.12.12-05.45.18, (Val) 0.0d
[279230], (Mid) 100580978,(Rid) 200393852, (date) [1323666918], 2011.12.12-05.15.18, (Val) 0.0d
[279231], (Mid) 100580978,(Rid) 200393852, (date) [1323667818], 2011.12.12-05.30.18, (Val) 0.0d
[279232], (Mid) 100580978,(Rid) 200393852, (date) [1323668718], 2011.12.12-05.45.18, (Val) 0.0d
[279233], (Mid) 100580989,(Rid) 200393855, (date) [1323666018], 2011.12.12-05.00.18, (Val) 0.0d
[279354], (Mid) 100580989,(Rid) 200394082, (date) [1323666918], 2011.12.12-05.15.18, (Val) 0.0d
[279355], (Mid) 100580989,(Rid) 200394082, (date) [1323667818], 2011.12.12-05.30.18, (Val) 0.0d
[279356], (Mid) 100580989,(Rid) 200394082, (date) [1323668718], 2011.12.12-05.45.18, (Val) 0.0d
***********************************************
here 1005XXXXX is the DeviceID and 0.0d is the value of the device id at that time stamp.
now i want the AVG, Max and 2nd Max values for each deviceID.

Its a bit urgent!!

Thanks,
Mahi
# 2  
Old 12-14-2011
avg, max etc of what? 0.0d?
Need more info!

Provide a sample output!

--ahamed
# 3  
Old 12-14-2011
Computations on a text file
Yeah.. on 0.0d for each deviceID. but here you dont need to worry about value type (assume it as integer) i jus need the code to achieve output. the output file should look like this

#deviceID Avg Max 2ndMax
100581038 0.0 0.0 0.0

Thanks!!
# 4  
Old 12-14-2011
@mahi_mayu069: With what will you compute the avg, max and 2nd max for each device id? Please provide an example.
# 5  
Old 12-14-2011
Hi Balajesuri,

Computations to be done on the values 0.0d(assume it to be integer), last coloumn of the file i.e (Val) 0.0d.

Thanks,
Mahi
# 6  
Old 12-14-2011
Try this.

Code:
#!/usr/bin/perl

use List::Util qw(min max sum);

%d = ();

while (<STDIN>)
{
    /\(Mid\) ([0-9]+),.*\(Val\) ([0-9.-]+)d/;
    next if ($1 eq '' || $2 eq '');
    $d{$1} = [] if (!defined($d{$1}));
    push(@{$d{$1}}, $2);
}

foreach (keys %d)
{
    @a = sort {$a <=> $b} @{$d{$_}};
    $max  = (@a > 0) ? max(@a) : 0;
    $max2 = (@a > 1) ? $a[@a - 2] : 0;
    $avg = (@a > 0) ? sum(@a) / @a : 0;
    print "$_ $avg $max $max2\n";
}

Run by:

Code:
./script.pl < temp.txt

This User Gave Thanks to MacMonster For This Post:
mahi_mayu069
# 7  
Old 12-14-2011
MySQL
Thanks....but I want the script to be shell.

Thanks Again,

--Mahi
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Match text to lines in a file, iterate backwards until text or text substring matches, print to file

hi all, trying this using shell/bash with sed/awk/grep I have two files, one containing one column, the other containing multiple columns (comma delimited). file1.txt abc12345 def12345 ghi54321 ... file2.txt abc1,text1,texta abc,text2,textb def123,text3,textc gh,text4,textd... (6 Replies)
Discussion started by: shogun1970
6 Replies

2. Shell Programming and Scripting

Match text from file 1 to file 2 and return specific text

I hope this makes sense and is possible. I am trying to match $1 of panel_genes.txt with $3 of RefSeqGene.txt and when a match is found the value in $6 of RefSeqGene.txt Example: ACTA2 is $1 of panel_genes.txt ACTA2 NM_001613.2 ACTA2 NM_001141945.1 awk 'FNR==NR {... (4 Replies)
Discussion started by: cmccabe
4 Replies

3. Programming

Computations using minimum values

I have the following code and into into trying to simplifying it. Any suggestions please? pmin = min (p(1), p(2), p(3), p(4), p(5), p(6)) ni = 0 xint = 0.0 yint = 0.0 zint = 0.0 !--------------------------------------------- ! if ((0.99999 * p(1)) <= pmin) then ... (3 Replies)
Discussion started by: kristinu
3 Replies

4. UNIX for Dummies Questions & Answers

Extracting lines from a text file based on another text file with line numbers

Hi, I am trying to extract lines from a text file given a text file containing line numbers to be extracted from the first file. How do I go about doing this? Thanks! (1 Reply)
Discussion started by: evelibertine
1 Replies

5. UNIX for Dummies Questions & Answers

Numeric computations in bash

I am on Ubuntu 11.10 using bash. I want to perform some numeric computations and have the following options: # Setting verbosity levels none=0; low=1; medium=2; high=3; debug=4 (( vbLevel=medium )) if (( vbLevel > low )); then echo "Test 1, $vbLevel" fi vbLevel=$medium if ];... (3 Replies)
Discussion started by: kristinu
3 Replies

6. UNIX for Dummies Questions & Answers

Converting a text file with irregular spacing into a space delimited text file?

I have a text file with irregular spacing between values which makes it really difficult to manipulate. Is there an easy way to convert it into a space delimited text file so that all the spaces, double spaces, triple spaces, tabs between numbers are converted into spaces. The file looks like this:... (5 Replies)
Discussion started by: evelibertine
5 Replies

7. Shell Programming and Scripting

search text file in file if this file contains necessary text (awk,grep)

Hello friends! Help me pls to write correct awk and grep statements for my task: I have got files with name filename.txt It has such structure: Start of file FROM: address@domen.com (12...890) abc DATE: 11/23/2009 on Std SUBJECT: any subject End of file So, I must check, if this file... (4 Replies)
Discussion started by: candyme
4 Replies

8. Shell Programming and Scripting

Create multiple text file from a single text file on AIX

Hi I need to create multiple text files from onc text file on AIX. The data of text files is as below: ********************************************** ********************************************** DBVERIFY: Release 10.2.0.4.0 - Production on Tue Nov 10 13:45:42 2009 Copyright (c) 1982,... (11 Replies)
Discussion started by: lodhi1978
11 Replies

9. Shell Programming and Scripting

how can I bcp out a table into a text file including the header row in the text file

Hi All, I need to BCP out a table into a text file along with the table headers. Normal BCP out command only bulk copies the data, and not the headers. I am using the following command: bcp database1..table1 out file1.dat -c -t\| -b1000 -A8192 -Uuser -Ppassword -efile.dat.err Regards,... (0 Replies)
Discussion started by: shilpa_acc
0 Replies

10. Shell Programming and Scripting

Insert rows with computations of next row

Hello folks, I have data collected in every 3 hours. But, I would like to expand this to 1 hour interval by equally dividing with next row. For example, I want to keep the first value 1987-01-01-00z 2.0, but following all record should be re-written as follow. 1987-01-01-03z 5.0 becomes... (11 Replies)
Discussion started by: Jae
11 Replies
Login or Register to Ask a Question