Visit Our UNIX and Linux User Community


Interactive filters for log file (beginner)


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Interactive filters for log file (beginner)
# 1  
Old 06-08-2011
Interactive filters for log file (beginner)

Hello,

I have a software which log all entry from internet and save it in text file.

For example (fake):
Code:
 10:02:23:124.id_0000.1:"blalba"
10:05:26:124.id_0000.1:"blalba"
10:10:32:124.id_0000.1:"blalba"
11:32:36:124.id_0000.1:"blalba"
11:33:49:124.id_0000.1:"blalba"
11:36:23:124.id_0000.1:"blalba"
12:10:21:124.id_0000.1:"blalba"

I just need to enter two time limit (start and end time).

For example with start = "11:30:00" and end = "11:40:00, should return:

Code:
11:32:36:124.id_0000.1:"blalba"
11:33:49:124.id_0000.1:"blalba"
11:36:23:124.id_0000.1:"blalba"

I tried some script with awk, but, I'm beginner and I need your help Smilie

Thanks a lot for your response,

Acid

[edit] the log file can be huge (more than 100 000 lines).
# 2  
Old 06-08-2011
Code:
awk ' /^11:[34][0-9]:[0-9][0-0]/ {print}' filename

Code:
start_line=`grep -n -m 1 "^11:30:00" filename | awk '{print $1}'`
end_line=`grep -n -m 1 "^11:40:00" filename | awk '{print $1}'`
sed '"$start_line","$end_line"p' filename

# 3  
Old 06-08-2011
Or for something more re-usable (DATA section for illustration, you'd have to open the file for reading in a programmed script.
Code:
#!/usr/bin/perl

@divs=qw(hour min sec milis);
@start{@divs} = $ARGV[0]=~/(\d+)/g;
@end{@divs} = $ARGV[1]=~/(\d+)/g;
while(<DATA>){
    chomp;
    @time{@divs} = $_ =~/(\d+):/g;
    if ((after(\%time, \%start)) && (! after(\%time, \%end))){
        print "$_\n";
    }
}
sub after{
    my ($time,$limit)=@_;
    for (@divs){
        if ($time->{$_} < $limit->{$_}){
            return 0;
        }
        elsif ($time->{$_} > $limit->{$_}){
            return 1;
        }
    }
    return 1;
}
__DATA__
10:02:23:124.id_0000.1:"blalba"
10:05:26:124.id_0000.1:"blalba"
10:10:32:124.id_0000.1:"blalba"
11:32:36:124.id_0000.1:"blalba"
11:33:49:124.id_0000.1:"blalba"
11:36:23:124.id_0000.1:"blalba"
12:10:21:124.id_0000.1:"blalba"
12:20:21:124.id_0000.1:"blalba"
12:30:21:124.id_0000.1:"blalba"

This User Gave Thanks to Skrynesaver For This Post:
# 4  
Old 06-08-2011
Quote:
Originally Posted by itkamaraj
Code:
awk ' /^11:[34][0-9]:[0-9][0-0]/ {print}' filename

Code:
start_line=`grep -n -m 1 "^11:30:00" filename | awk '{print $1}'`
end_line=`grep -n -m 1 "^11:40:00" filename | awk '{print $1}'`
sed '"$start_line","$end_line"p' filename

Thanks for your solution but it's work only for my example.
For example if I choose time range between 15:40:00 and 23:10:00
this solution doens't work Smilie

Quote:
Originally Posted by Skrynesaver
Or for something more re-usable (DATA section for illustration, you'd have to open the file for reading in a programmed script.
Code:
#!/usr/bin/perl

@divs=qw(hour min sec milis);
@start{@divs} = $ARGV[0]=~/(\d+)/g;
@end{@divs} = $ARGV[1]=~/(\d+)/g;
while(<DATA>){
    chomp;
    @time{@divs} = $_ =~/(\d+):/g;
    if ((after(\%time, \%start)) && (! after(\%time, \%end))){
        print "$_\n";
    }
}
sub after{
    my ($time,$limit)=@_;
    for (@divs){
        if ($time->{$_} < $limit->{$_}){
            return 0;
        }
        elsif ($time->{$_} > $limit->{$_}){
            return 1;
        }
    }
    return 1;
}
__DATA__
10:02:23:124.id_0000.1:"blalba"
10:05:26:124.id_0000.1:"blalba"
10:10:32:124.id_0000.1:"blalba"
11:32:36:124.id_0000.1:"blalba"
11:33:49:124.id_0000.1:"blalba"
11:36:23:124.id_0000.1:"blalba"
12:10:21:124.id_0000.1:"blalba"
12:20:21:124.id_0000.1:"blalba"
12:30:21:124.id_0000.1:"blalba"

Thanks a lot for this very nice solution Smilie
I have a lot of work to undestand in depth your code ^_^


But , anyway to do it with awk ?
# 5  
Old 06-08-2011
Code:
#!/usr/bin/perl

@divs=qw(hour min sec milis); # the names of the time parts I'm grabbing
@start{@divs} = $ARGV[0]=~/(\d+)/g; # throw the first four digit groups into a hash naming them by the keys in @div
@end{@divs} = $ARGV[1]=~/(\d+)/g; # As above but the second argument to the script
#open (DATA, '<', "$ARGV[2]"); # if you were supplying a logfile as an argument to the command
while(<DATA>){ 
    @time{@divs} = $_ =~/(\d+):/g; # as above, but for each line of the log file.
         # the record time is after start time and not after end time for the window we are looking for.
    if ((after(\%time, \%start)) && (! after(\%time, \%end))){ # see below for how after works the "\%" passes references to the hashes
        print ; # print out the line 
    }
}
sub after{ # boolean check to see if a time is after a given limit
    my ($time,$limit)=@_; # the arguments supplied
    for (@divs){ # go through the keys in order
        if ($time->{$_} < $limit->{$_}){ 
            return 0; # if the first non-equal division (hour min sec milli) is less than the limit then this is not after the time
        }
        elsif ($time->{$_} > $limit->{$_}){
            return 1; # the other case
        }
    }
    return 1; # millisecond equivalent, display it
}
__DATA__
10:02:23:124.id_0000.1:"blalba"
10:05:26:124.id_0000.1:"blalba"
10:10:32:124.id_0000.1:"blalba"
11:32:36:124.id_0000.1:"blalba"
11:33:49:124.id_0000.1:"blalba"
11:36:23:124.id_0000.1:"blalba"
12:10:21:124.id_0000.1:"blalba"
12:20:21:124.id_0000.1:"blalba"
12:30:21:124.id_0000.1:"blalba"

The above would be called as
Code:
~/$ getLogWindow.pl 11:30:00 11:40:00 application.log

If you used the open command to access the log file in ARGV[2]

I'm not sure that an awk command will give you a neater solution, however I'm frequently amazed at what some people manage to get awk to do on this site, though I do wonder if they can see what and how it is doing 6 months later Smilie
# 6  
Old 06-08-2011
This will help you...

Code:
nawk -F':' -v v1="11:30:00" -v v2="11:40:00" 'OFS=":"{v=$1":"$2":"$3}{if(v>v1 && v<v2)print}' inputfile
11:32:36:124.id_0000.1:"blalba"
11:33:49:124.id_0000.1:"blalba"
11:36:23:124.id_0000.1:"blalba"

Thanks
Sha
This User Gave Thanks to Shahul For This Post:
# 7  
Old 06-08-2011
Quote:
Originally Posted by Skrynesaver
Code:
[...]

If you used the open command to access the log file in ARGV[2]

I'm not sure that an awk command will give you a neater solution, however I'm frequently amazed at what some people manage to get awk to do on this site, though I do wonder if they can see what and how it is doing 6 months later Smilie
Oh tanks again for all this explanation... it is very helpful!

Quote:
Originally Posted by Shahul
This will help you...

Code:
nawk -F':' -v v1="11:30:00" -v v2="11:40:00" 'OFS=":"{v=$1":"$2":"$3}{if(v>v1 && v<v2)print}' inputfile
11:32:36:124.id_0000.1:"blalba"
11:33:49:124.id_0000.1:"blalba"
11:36:23:124.id_0000.1:"blalba"

Thanks
Sha
That's why I love awk so much... simple, short and perfect!
Thanks a lot SmilieSmilie

Previous Thread | Next Thread
Test Your Knowledge in Computers #848
Difficulty: Medium
The IEEE standardized Bluetooth as IEEE 802.15.1, but IEEE no longer maintains the Bluetooth standard.
True or False?

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Beginner need help how file moving script could look like

Hi there, I am an absolut beginner in scripting. I am using Ubuntu Linux 18.04 LTS: I just need some hints/suggestions how a script could look like doing the following stuff: assuming I am in directory "base_directory" like xyz@mypc:~/base_directory$ within the "base_directory there... (1 Reply)
Discussion started by: dut42
1 Replies

2. Shell Programming and Scripting

Creating an excel file with filters using shell script.

Hi All, I am new to shell scripting. I have made a script that can convert an excel file from cvs file. This excel file contains hundreds of records and i would like the reader to be able to filter. Is it possible to create an excel file with filters? or that functionality has not been... (3 Replies)
Discussion started by: Marvin159875321
3 Replies

3. Shell Programming and Scripting

Need help in filters

Hi, I have input data. 9214919702; B5; 1;20070216; 9231590437; BY; 1;20070215;9;20091022;12;20091022; 9211765888; AZ; 1;20080802;1;20080802;14;20091027; 9231592590; BY; 1;20070215;9;20091026;9;20091026; 9252412219; MM; 1;20070217; 9214917135; MM; 1;20070215; 9214917056; B5; 1;20070215;... (8 Replies)
Discussion started by: suresh3566
8 Replies

4. Homework & Coursework Questions

How to write script that behaves both in interactive and non interactive mode

Q. Write a script that behaves both in interactive and non interactive mode. When no arguments are supplied it picks up each C program from the directory and prints first 10 lines. It then prompts for deletion of the file. If user supplies arguments with the script , then it works on those files... (8 Replies)
Discussion started by: rits
8 Replies

5. Homework & Coursework Questions

Help with Interactive / Non Interactive Shell script

Q. Write a script that behaves both in interactive and non interactive mode. When no arguments are supplied it picks up each C program from the directory and prints first 10 lines. It then prompts for deletion of the file. If user supplies arguments with the script , then it works on those files... (1 Reply)
Discussion started by: rits
1 Replies

6. UNIX for Dummies Questions & Answers

(Beginner) Run c++ .exe with input to file

Hi, I've got this requirement for my homework assignment, but I'm not sure how to meet it: In the comamnd line, I need to type $ <exec-file> <input> <output_file_name> Like: test 1+2 out.txt Which should execute test.exe passing in 1+2 and directing output to out.txt. I know how... (1 Reply)
Discussion started by: JustinT
1 Replies

7. Shell Programming and Scripting

Need help with Interactive rename file.

hey all i was writing a script to 1. Rename a file upon the user's request. If the file exists, prompt the user for confirmation before renaming the file. The screen should prompt the user for a. Name of file you want to rename. Use the \c escape character. b. ... (36 Replies)
Discussion started by: keyboardkowboy
36 Replies

8. UNIX for Dummies Questions & Answers

.bashrc file is an initialization file run by each interactive invocation

I search the web and found the following statements ..... The /etc/profile file is a system wide initialization script which is run at login time for each user, while .profile is the users own login initialization. The .bashrc file is an initialization file run by each interactive invocation... (1 Reply)
Discussion started by: cy163
1 Replies

9. UNIX for Dummies Questions & Answers

filters

how to filter one particular row from one text file and copy it in another? (1 Reply)
Discussion started by: rajanandhini
1 Replies

10. UNIX for Dummies Questions & Answers

IP Filters

Anyone know where I can find good documentation for IPF on the Internet? Thanks, Chuck (1 Reply)
Discussion started by: 98_1LE
1 Replies

Featured Tech Videos