Shell Programming and Scripting

Count everything and then see how many hits that are between one hour for a user.
I have to be some line of script to make this.

That request is unclear to me. Please provide a sample file including those three connections that you want to be put out, plus some lines in between that don't match your criteria. Explain the logics with those samples.

Hi,

Let me explain again -

We're looking for users who are making more than 3 heavy searches, within the log it will appear as "SRCH=Q".

As listed below in the log, what we would like to do is whenever we find 'SRCH=Q' within the log, we will use the connection number( 13570) and trace it back to the BIND to find out who it is, and then update the user value with number of occurrence, if the occurrence is > 3, perform some action ( like add it to restricted group - we will user some internal command for adding to the group).

Note: Within the logs BIND user come before the SRCH=Q, hence we need to perform a reverse query to get the BIND user.

Code:

[04/Jun/2013:13:06:13 -0600] conn=13570 op=14 msgId=13 - BIND dn="uid=xyz123,ou=People,o=xyz.com" method=128 version=3
[04/Jun/2013:13:08:14 -0600] conn=13570 op=14 msgId=15 - RESULT err=0 tag=101 nentries=48030 etime=139 SRCH=Q
[04/Jun/2013:13:09:15 -0600] conn=13571 op=14 msgId=13 - BIND dn="uid=xyz123,ou=People,o=xyz.com" method=128 version=3
[04/Jun/2013:13:10:17 -0600] conn=13571 op=14 msgId=15 - RESULT err=0 tag=101 nentries=48030 etime=139 
[04/Jun/2013:14:09:27 -0600] conn=13572 op=14 msgId=13 - BIND dn="uid=xyz123,ou=People,o=xyz.com" method=128 version=3
[04/Jun/2013:14:33:18 -0600] conn=13572 op=14 msgId=15 - RESULT err=0 tag=101 nentries=48030 etime=139 SRCH=Q
[04/Jun/2013:15:09:22 -0600] conn=13573 op=14 msgId=13 - BIND dn="uid=xyz123,ou=People,o=xyz.com" method=128 version=3
[04/Jun/2013:15:13:31 -0600] conn=13573 op=14 msgId=15 - RESULT err=0 tag=101 nentries=48030 etime=139 SRCH=Q

Hope this makes things clear.

Thanks, John

This thread's post count has reached double-digits and you've yet to provide reasonable sample data. How do you expect someone who is volunteering their time to test their code when the only thing you've provided is two lines? Why make it more difficult than it needs to be for those willing to freely share with you their time and expertise?

Provide a larger data sample that contains log entries which match your criteria, along with the desired output that corresponds to the data sample.

Regards,
Alister

Here is the larger set of data, as mentioned previously we need to identify the occurrence and then perform an internal action, I need the code to identity the occurence :

Code:

[04/Jun/2013:13:06:13 -0600] conn=13570 op=14 msgId=13 - BIND dn="uid=xyz123,ou=People,o=xyz.com" method=128 version=3
[04/Jun/2013:13:08:14 -0600] conn=13570 op=14 msgId=15 - RESULT err=0 tag=101 nentries=48030 etime=139 SRCH=Q
[04/Jun/2013:13:09:15 -0600] conn=13571 op=14 msgId=13 - BIND dn="uid=xyz123,ou=People,o=xyz.com" method=128 version=3
[04/Jun/2013:13:10:17 -0600] conn=13571 op=14 msgId=15 - RESULT err=0 tag=101 nentries=48030 etime=139 
[04/Jun/2013:14:09:27 -0600] conn=13572 op=14 msgId=13 - BIND dn="uid=xyz123,ou=People,o=xyz.com" method=128 version=3
[04/Jun/2013:14:33:18 -0600] conn=13572 op=14 msgId=15 - RESULT err=0 tag=101 nentries=48030 etime=139 SRCH=Q
[04/Jun/2013:15:09:22 -0600] conn=13573 op=14 msgId=13 - BIND dn="uid=xyz123,ou=People,o=xyz.com" method=128 version=3
[04/Jun/2013:15:13:31 -0600] conn=13573 op=14 msgId=15 - RESULT err=0 tag=101 nentries=48030 etime=139 SRCH=Q

John,
You keep saying basically the same thing without answering the questions needed to figure out what you're trying to do. Let me give you a simple list of questions. Answer them and we may be able to help, or we may have more questions:
I assume that you already know the answers to all of the above questions and just haven't shared them with us. If you can't answer some of these questions, please quickly let us know that so we can forget about this topic and concentrate on issues for other forum members. (Then when you can answer them, start a new thread with all of the details we need to be able to help you. And, if you need to start a new thread, show us what you have tried while working on solutions to this problem.)

Hi Don,

Thanks for putting all your questions, really appreciate your effort. Here is my response to your questions :


What exactly is the output you want (using CODE tags) when your input contains 3 searches that occur in an hour and show us sample data that we can use as a test that actually contains data that would cause a report to be generated?

# re : As mentioned initially, the output will lead to execution of an internal command ( add group) - no output as such, however we can print the addition within the log. Basically, the intent is to add heavy search users within a restricted group. We need to pull out those users who perform more than 3 searches.


What determines the 1 hour time range:
3 BIND entries with timestamps <= 1 hour apart where the corresponding RESULT entries include SRCH=Q?
3 RESULT entries with SRCH=Q with timestamps <= 1 hour?
3 RESULT entries with SRCH=Q where the sum of the differences between the corresponding BIND and RESULT timestamps is >= 1 hour?
something else?

# re : 1 hour scan is to limit less log scanning, that's all. You can drop the hour part as of now, I am really interested in getting the logic first for the main issue.


Note that none of the 1st three elements in the above list will trigger a report with any of the sample data you have given us! (However, C would be met if you wanted to report a user using more than one hour of search time in three or fewer searches if we used the data provided in your initial post in this thread.)
Are all timestamps in non-decreasing order?
Are all timestamps in increasing order?
Are all timestamps in the same timezone?

# re : Don't bother about timestamp etc. The data which I provided is enough to create the condition. An output for less than too search will not trigger the execution of add group command. All timestamp will be in increasing order, and will be in same timezone.

Do we have to worry about shifts to and from daylight savings time?
Are all timestamps in an input file from the same day? If not, will there ever be:"
days that have no RESULT entries with SRCH=Q?
days that have no entries at all?
a time difference between a BIND time and a RESULT time of more than 23:59:59?
If a user has more than one 1 hour time period with 3 SRCH=Q RESULT entries:

re : We do not care about shifts or daylight saving time. If there are any heavy users, they will eventually get captured during the multiple runs, we don't care if we miss a couple of them due to daylight saving, etc.

Should there only be 1 occurrence reported for the user?
re : No, see my response below.

Should there be 1 occurrence reported for every set of 3 SRCH=Q RESULT entries that occur within an hour even if some of the SRCH=Q RESULT entries were used in an earlier report?

re : Yes, there should only 1 occurrence reported for every set of 3 SRCH=Q, once the user is added to the group, the server will not receive further request due to user being added to restricted group.

Should there be 1 occurrence reported for every set of 3 SRCH=Q RESULT entries that occur within an hour but only if none of the SRCH=Q RESULT entries were used in an earlier report?

re : see my response above.

Will every RESULT entry have a corresponding BIND entry?

re : Yes.

Could the corresponding BIND appear in a different log file? If the corresponding BIND is not found, should some kind of diagnostic be printed? (If so, exactly what should the diagnostic be?)

re: If corresponding BIND does not appear, we will not count that search, and have to drop the search.

Can there be more than one search in progress for a given user at any time? (I.e., can there be two or more BIND entries before the RESULT entry shows up corresponding to the first BIND entry?

re: Ideally, there will be only one BIND search at any given point of time.


Will conn=xxx always be in the 3rd field in every line?

re : yes.

Will "BIND" and "RESULT" always appear in the 7th field in every input line? If so, will the 7th field on any line ever contain anything other than "BIND" or "RESULT"?

re : yes. As shown within the logs, all fiels locations are static.

On RESULT lines that contain "SRCH=Q", will that string always be the last field on the line?

re : Yes.