ksh comparing current and previous lines


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting ksh comparing current and previous lines
# 1  
Old 10-12-2011
Question ksh comparing current and previous lines

Hi,

I am currently trying to work out how to compare one line with the last line I have read in via ksh. I have a file which has sorted output from a previous sort command so all the lines are in order already and the file would look something like show below. Each line has a name and a time taken to perform a certain task so I need to filter on the first column as shown below (aaa,bbb,ccc etc) and average all the data up for each matching one and for each no matching one. I am not interested in the addition part as yet as I do have that working in my script using a perl command due to the decimal places.

I.E. So if the current line does not match the previous one, write out it's name and it's data, however if the current line matches the previous one add up data and so on and so on until it does not match then write out the data. I thought it would be really easy but my scripting is just not working and I keep getting an extra line.

The result

Code:
B0000003,1.888
B0000024,0.728
B0000024,3.308
B0000024,1.948
B0000027,1.185
B0000030,3.287
B0000030,2.688

This is what I have tried to do, but does not seem to work and the result (regardless of how I place my if's and else's seems to not put the correct counter for each match.
Code:
InputFile=input_file.sort
 
    while read line
    do

        NAME=`echo $line | cut -d"," -f1`
        TIME=`echo $line | cut -d"," -f2`
 
        if [[ "$COUNT" -gt "1" ]] then
            if [[ $NAME != $PREV_NAME ]] then
               echo "$COUNT:$PREV_NAME"
               COUNT=1
            fi
        fi

 
        if [[ $NAME != $PREV_NAME ]] then
            COUNT=1
            echo "1,$NAME"
        elif [[ $NAME = $PREV_NAME ]] then
            let COUNT=$COUNT+1
        fi
 
 
        PREV_NAME=$NAME
    done <"$InputFile"


and this is the output I get from a valid input file.
Code:
Doc Name, Duration (secs)
1,Doc Name
B0000003,0.192494
1,B0000003                         < Error
B0000003,1.740529
B0000024,5.409181
2:B0000003                         < Correct
1,B0000024                         < Error
B0000024,5.409181
1,B0000024
B0000024,5.524508
B0000024,5.569696
B0000024,6.264533
B0000027,0.265375
4:B0000024                        < Correct
1,B0000027                        < Error
B0000027,0.298291
B0000027,1.157592
etc

As you can see it does work sort of, but it also puts in an extra line that is not needed.

Any help would be appreciated and Thanks!

Last edited by Franklin52; 10-13-2011 at 03:28 AM.. Reason: Please use code tags, thank you
# 2  
Old 10-12-2011
Not fully understand your request. But if you just need to calculate the average for each name, try AWK's array, no need to sort the data

Code:
awk -F, '{a[$1]++;b[$1]+=$2}END{for (i in a) print i, b[i]/a[i] |"sort -n"}'  infile

B0000003 1.888
B0000024 1.99467
B0000027 1.185
B0000030 2.9875

As a new joiner, read this first:

How to Use Code Tags in The UNIX and Linux Forums
https://www.unix.com/how-post-unix-li...code-tags.html
# 3  
Old 10-12-2011
Power You read that correctly

Holy Toledo Batman

One line instead of about 30 lines in my script, shows you my level of scripting skill. I will need to read up on what is going on there, but it does seem to work. Smilie

Yes that does work correctly, however I also need to work out the number of ocurrunces, average, minimum and maximum times for each one. So what you have there is absolutely spot on for what I asked. You are awesome.

Is that able to be done in the same line by any chance like here.

Name Number Average Minimum Maximum
B0000024 10 6.9783 1.9203 12.7652
L0000002 1 10.5767 10.5767 10.5767
L0000003 3 17.5713 9.09821 23.9881

Thanks

---------- Post updated at 12:12 PM ---------- Previous update was at 12:08 PM ----------

I will make sure I put my code into the correct code tags.Smilie
# 4  
Old 10-13-2011
Awk is wonderful Smilie

I have different opinions on trying to cram everything onto one line, especially if you're putting it into a script. It's easier to read and can be commented. Working with rdcwayx's start, here's how I'd add the extra bits, but it's not the only way:

Code:
awk -F, '
{
    if( !a[$1] )
        min[$1] = max[$1] = $2+0;   # initialise min/max on first hit of each name; adding zero ensures numeric comparison
    else
    {
        if( $2 > max[$1]  )        # capture larger/smaller values 
            max[$1] = $2+0;
        if( $2 < min[$1]  )
            min[$1] = $2+0;
    }

    a[$1]++;
    b[$1]+=$2;
}
END {
    for (i in a)
        printf( "%s %d %.3f %.3f %.3f\n", i, a[i], b[i]/a[i], min[i], max[i] ) | "sort -n"
}'  inputfile

I prefer printf() because I can control the number of decimal points for the irrational output (3 places in this case).
This User Gave Thanks to agama For This Post:
# 5  
Old 10-13-2011
Code:
awk -F, '{a[$1]++;b[$1]+=$2;min[$1]=(min[$1]==""||min[$1]>$2)?$2:min[$1];max[$1]=max[$1]>$2?max[$1]:$2}
    END{for (i in a) print i, a[i],b[i]/a[i],min[i],max[i] |"sort -n"}' infile

B0000003 1 1.888 1.888 1.888
B0000024 3 1.99467 0.728 3.308
B0000027 1 1.185 1.185 1.185
B0000030 2 2.9875 2.688 3.287

This User Gave Thanks to rdcwayx For This Post:
# 6  
Old 10-13-2011
MySQL Thank you very much rdcwayx and agama

I hate not being able to work things out myself but sometimes someone else just knows the best way to do things.

I appreciate both of your input and both ways work exactly as I wanted.

You have both been very helpfull and gave me a great insight to awk's capabalilty which I have been reading up on since this thread. It is a lot more powerfull than I expected or have been using it for in the past.

Again thank you both for you time and input!! Smilie Smilie
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to compare previous and current item in for loop in bash?

Hey, I am trying to compare formated login and logout dates from one user at a host which I have stored in a tmp directory in order to find out the total login time. I need to compare them in order to find overlapping intervals. At first I tried to store each log in and logo date in an array... (3 Replies)
Discussion started by: Mumu123
3 Replies

2. Shell Programming and Scripting

Perl for comparing numbers from previous lines in a file?

Hi everyone I have a question for you, as I am trying to learn more about Perl and work with some weather data. I have an ascii file (shown below) that has 10 lines with different columns. What I would like is have Perl find an "anomalous" value by comparing a field with the values from the last... (2 Replies)
Discussion started by: lucshi09
2 Replies

3. Shell Programming and Scripting

Perl to send previous and current value

For example, I have a file called number.txt. x y 1 1 2 4 3 9 4 6 5 5 6 6 7 9 8 4 9 1 10 0 ... And I want to print out the value of x and y, if y%4==0 and the next value of y%4==0. Thus, the sample output is: 1 1 *because the previous x before 2 is 1 2 4 *because 4%4 == 0 7 9... (2 Replies)
Discussion started by: Tzeronone
2 Replies

4. Shell Programming and Scripting

Remove previous line if next & previous lines have same 4th character.

I want to remove commands having no output. In below text file. bash-3.2$ cat abc_do_it.txt grpg10so>show trunk group all status grpg11so>show trunk group all status grpg12so>show trunk group all status GCPKNYAIGT73IMO 1440 1345 0 0 94 0 0 INSERVICE 93% 0%... (4 Replies)
Discussion started by: Raza Ali
4 Replies

5. Shell Programming and Scripting

Sed Comparing Parenthesized Values In Previous Line To Current Line

I am trying to delete lines in archived Apache httpd logs Each line has the pattern: <ip-address> - - <date-time> <document-request-URL> <http-response> <size-of-req'd-doc> <referring-document-URL> This pattern is shown in the example of 6 lines from the log in the code box below. These 6... (1 Reply)
Discussion started by: Proteomist
1 Replies

6. Shell Programming and Scripting

AWK Compare previous value with current.

Hi, I have one small doubt how to go ahead and process the below requirement. File Content 1,abc,10 2,xyz,11 3,pqr,12 4,pqr,13 5,pqr,14 Output file expected: 1,mnq,1 1,ddd,2 1,qqq,3 1,sss,4 1,ddd,5 1,eee,6 1,fff,7 1,ddr,8 1,rrd,9 (3 Replies)
Discussion started by: dikesm
3 Replies

7. UNIX for Dummies Questions & Answers

Awk to print data from current and previous line

Hi guys, I have found your forum super useful. However, right now I am stuck on a seemingly "simple" thing in AWK. I have two columns of data, the first column in Age (in million years) and the second column is Convergence Rate (in mm/yr). I am trying to process my data so I can use it to... (2 Replies)
Discussion started by: awk_noob_456
2 Replies

8. UNIX for Dummies Questions & Answers

print previous month (current month minus 1) with Solaris date and ksh

Hi folks month=`date +%m`gives current month Howto print previous month (current month minus 1) with Solaris date and ksh (7 Replies)
Discussion started by: slashdotweenie
7 Replies

9. Shell Programming and Scripting

How to use sed to search for string and Print previous two lines and current line

Hello, Can anybody help me to correct my sed syntax to find the string and print previous two lines and current line and next one line. i am using string as "testing" netstat -v | sed -n -e '/test/{x;2!p;g;$!N;p;D;}' -e h i am able to get the previous line current line next line but... (1 Reply)
Discussion started by: nmadhuhb
1 Replies

10. Shell Programming and Scripting

Print previous, current and next line using sed

Hi, how can i print the previous, current and next line using sed? current line is the matching line. The following prints all lines containing 'Failure' and also the immediate next line cat $file | sed -n -e '/Failure/{N;p;}' Now, i also want to print the previous line too. Thanks,... (8 Replies)
Discussion started by: ysrinu
8 Replies
Login or Register to Ask a Question