Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Search for a specific data in a file based on a date range

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Search for a specific data in a file based on a date range
# 1  
Old 09-27-2011
Search for a specific data in a file based on a date range
Hi,

Currently I am working on a script to automate the process of converting the log file from binary into text format. To achieve this, partly I am depending on my application’s utility for this conversion and the rest I am relying on shell commands to search for directory, locate the file and search for specific data in the converted file and email that to the user.

Here is the script that will do the work.

Code:
#!/bin/ksh
 
. /home/.env
 
# Go to the location where log file is stored.
 
cd $lOg_Path
 
cd `find . -type d |sort | tail -1`
 
LogName=`ls -ltr | grep -v 'resend' | grep -E '\.dat$'|tail -1 | awk  '{ print $9 }'`
 
NewLogName=`echo $LogName | cut -f1 -d .`.txt
 
infacmd.sh isp ConvertLogFile -in $LogName -lo $NewLogName
 
 
tail -4 ${Log_File} > tmp.txt
 
#echo "=============================================================================================================================">>tmp.txt
 
#echo "Please find attached Log below">>tmp.txt
 
EndTime=`date +%H-%M-%S`
 
#(  cat tmp.txt; uuencode $NewLogName $NewLogName) | mailx -s "Subject" `echo ${Alertlist_All}`

But the problem is that the converted output file wrapped for every 80 characters and rest of the information is printed on the next line. Due to this, I am having difficulty in searching for a specific data in the log file based on a date range along with timestamp and print the output.

Here is the file output

Code:
2011-09-26 INF UM_10 UserMana Starting synchronization for security domain
14:37:31       080   gementSe [12345].
                     rvice
2011-09-26 INF LDAP_ UserMana LDAP user [xxxx] in security domain
14:37:31       10002 gementSe [12345] queried with LDAP DN
                     rvice    [CN=sreich,CN=Users,DC=DOMAIN,DC=COM].
2011-09-26 INF LDAP_ UserMana LDAP user [yyyyy] in security domain
14:37:31       10002 gementSe [12345] queried with LDAP DN
                     rvice    [CN=yyyyy,CN=Users,DC=DOMAIN,DC=COM].
2011-09-26 INF LDAP_ UserMana LDAP user [zzzzz] in security domain [12345]
14:37:31       10002 gementSe queried with LDAP DN
                     rvice    [CN=zzzzz,CN=Users,DC=DOMAIN,DC=COM].
2011-09-26 INF LDAP_ UserMana LDAP group [INFA_LDAP] in
14:37:31       10003 gementSe security domain [12345] queried from LDAP.
                     rvice
2011-09-26 INF LDAP_ UserMana LDAP group [INFA_LDAP] in
14:37:31       10004 gementSe security domain [12345] queried from LDAP
                     rvice    contains user [yyyyy] in security domain
                              [12345].
2011-09-26 INF LDAP_ UserMana LDAP group [INFA_LDAP] in
14:37:31       10004 gementSe security domain [12345] queried from LDAP
                     rvice    contains user [zzzzz] in security domain
                              [12345].
2011-09-26 INF LDAP_ UserMana LDAP group [INFA_LDAP] in
14:37:31       10004 gementSe security domain [12345] queried from LDAP
                     rvice    contains user [sreich] in security domain
                              [12345].
2011-09-26 INF LDAP_ UserMana LDAP group [INFA_ADM_LDAP] in
14:37:31       10003 gementSe security domain [12345] queried from LDAP.
                     rvice
2011-09-26 INF LDAP_ UserMana LDAP group [INFA_ADM_LDAP] in
14:37:31       10004 gementSe security domain [12345] queried from LDAP
                     rvice    contains user [yyyyy] in security domain
                              [12345].
2011-09-26 INF LDAP_ UserMana LDAP group [INFA_ADM_LDAP] in
14:37:31       10004 gementSe security domain [12345] queried from LDAP
                     rvice    contains user [zzzzz] in security domain
                              [12345].
2011-09-26 INF LDAP_ UserMana Queried [3] users based on the user search base
14:37:31       10000 gementSe [CN=Users,DC=DOMAIN,DC=COM] with filter
                     rvice    [(&(ObjectClass=user)(|(memberOf=CN=INFA_LDAP_ADM,\
          CN=Users,DC=DOMAIN,DC=COM)(m
                              emberOf=CN=INFA_LDAP,CN=U
                              sers,DC=DOMAIN,DC=COM)))] specified in
                              [12345] security domain definition.
2011-09-26 INF LDAP_ UserMana Queried [2] groups based on the group search base
14:37:31       10001 gementSe [CN=Users,DC=DOMAIN,DC=COM] with filter
                     rvice    [(&(ObjectClass=group)(|(CN=INFA_
                               LDAP_ADM)(CN=INFA_LDAP_DEV)))] 
          specified in [12345] security domain
                               definition.
2011-09-26 INF LDAP_ UserMana Security domain [12345] synchronization with
14:37:32       10007 gementSe LDAP server successful.

To overcome this problem, I am finding the row numbers of starting row and ending row which I am interested in and pass them back to the script and store them in variables. After that, I am printing the data to a output file based on the row numbers.

Here is the code I am thinking of putting into the code.

Code:
RowNum=`grep -n '$StartTime' 1_0.txt |head -1|cut -c 1`  
cat 1_0.txt|awk 'NR>$RowNum'

I feel this is a crude way of coding but I am not aware of any better technique to handle this problem.

Sri
# 2  
Old 09-27-2011
To unfold the lines using the date as the key, try this code:
Code:
#!/usr/bin/ksh
while read mDate mRest; do
  if [[ ${#mDate} -eq 10 ]]; then
    if [[ "${mOut}" != "" ]]; then
      echo ${mOut}
    fi
    mOut=''
  fi
  mOut=${mOut}' '${mDate}' '${mRest}
done < Inp_File
if [[ "${mOut}" != "" ]]; then
  echo ${mOut}
fi

# 3  
Old 09-27-2011
Hi Shell_Life,

Thanks for the code snippet. I tried your solution and I was able to unwrap the text but later I identified that the data is actually wrapped on a column by column basis but not like as mentioned before. In other words, the data in each column is wrapped after certain length and continuing in the next line.

Your script converts the data like this.

HTML Code:
2011-09-26 INF UM_10 UserMana Starting synchronization for security domain 08:45:10 080 gementSe [12345]. rvice
Is there a way to unfold the data for each column.

This is what I am looking for.

HTML Code:
2011-09-26 14:37:31 INF UM_10080 UserManagementService Starting synchronization for security domain [12345].
Thanks in advance.
Last edited by svajhala; 09-27-2011 at 04:16 PM..
# 4  
Old 09-27-2011
Here's one way to do it with Perl:

Code:
$
$
$ cat data.txt
2011-09-26 INF UM_10 UserMana Starting synchronization for security domain
14:37:31       080   gementSe [12345].
                    rvice
2011-09-26 INF LDAP_ UserMana LDAP user [xxxx] in security domain
14:37:31       10002 gementSe [12345] queried with LDAP DN
                    rvice    [CN=sreich,CN=Users,DC=DOMAIN,DC=COM].
2011-09-26 INF LDAP_ UserMana LDAP user [yyyyy] in security domain
14:37:31       10002 gementSe [12345] queried with LDAP DN
                    rvice    [CN=yyyyy,CN=Users,DC=DOMAIN,DC=COM].
2011-09-26 INF LDAP_ UserMana LDAP user [zzzzz] in security domain [12345]
14:37:31       10002 gementSe queried with LDAP DN
                    rvice    [CN=zzzzz,CN=Users,DC=DOMAIN,DC=COM].
2011-09-26 INF LDAP_ UserMana LDAP group [INFA_LDAP] in
14:37:31       10003 gementSe security domain [12345] queried from LDAP.
                    rvice
2011-09-26 INF LDAP_ UserMana LDAP group [INFA_LDAP] in
14:37:31       10004 gementSe security domain [12345] queried from LDAP
                    rvice    contains user [yyyyy] in security domain
                          [12345].
2011-09-26 INF LDAP_ UserMana LDAP group [INFA_LDAP] in
14:37:31       10004 gementSe security domain [12345] queried from LDAP
                    rvice    contains user [zzzzz] in security domain
                          [12345].
2011-09-26 INF LDAP_ UserMana LDAP group [INFA_LDAP] in
14:37:31       10004 gementSe security domain [12345] queried from LDAP
                    rvice    contains user [sreich] in security domain
                          [12345].
2011-09-26 INF LDAP_ UserMana LDAP group [INFA_ADM_LDAP] in
14:37:31       10003 gementSe security domain [12345] queried from LDAP.
                    rvice
2011-09-26 INF LDAP_ UserMana LDAP group [INFA_ADM_LDAP] in
14:37:31       10004 gementSe security domain [12345] queried from LDAP
                    rvice    contains user [yyyyy] in security domain
                          [12345].
2011-09-26 INF LDAP_ UserMana LDAP group [INFA_ADM_LDAP] in
14:37:31       10004 gementSe security domain [12345] queried from LDAP
                    rvice    contains user [zzzzz] in security domain
                          [12345].
2011-09-26 INF LDAP_ UserMana Queried [3] users based on the user search base
14:37:31       10000 gementSe [CN=Users,DC=DOMAIN,DC=COM] with filter
                    rvice    [(&(ObjectClass=user)(|(memberOf=CN=INFA_LDAP_ADM,\
                          CN=Users,DC=DOMAIN,DC=COM)(m
                          emberOf=CN=INFA_LDAP,CN=U
                          sers,DC=DOMAIN,DC=COM)))] specified in
                          [12345] security domain definition.
2011-09-26 INF LDAP_ UserMana Queried [2] groups based on the group search base
14:37:31       10001 gementSe [CN=Users,DC=DOMAIN,DC=COM] with filter
                    rvice    [(&(ObjectClass=group)(|(CN=INFA_
                           LDAP_ADM)(CN=INFA_LDAP_DEV)))]
                          specified in [12345] security domain
                           definition.
2011-09-26 INF LDAP_ UserMana Security domain [12345] synchronization with
14:37:32       10007 gementSe LDAP server successful.
$
$
$ perl -ne 'BEGIN {$fmt = "%-19s %3s %-10s %-22s %s\n"}
        chomp;
        if (/^\d{4}-\d\d-\d\d/) {
          $items[0] =~ s/^(.{10})/$1 /;
          printf($fmt, @items);
          @items = unpack ("A10 x A3 x A5 x A8 x A*");
        } else {
          if (length $_ == 26) {@y = unpack ("A10 x A3 x A5 x A*")}
          else {@y = unpack ("A10 x A3 x A5 x A8 x A*")}
          for ($i=0; $i<=$#y; $i++) {$items[$i] .= $y[$i]}
        }
        END {$items[0] =~ s/^(.{10})/$1 /; printf($fmt, @items)}
       ' data.txt
 
2011-09-26 14:37:31 INF UM_10080   UserManagementService  Starting synchronization for security domain[12345].
2011-09-26 14:37:31 INF LDAP_10002 UserManagementService  LDAP user [xxxx] in security domain[12345] queried with LDAP DN[CN=sreich,CN=Users,DC=DOMAIN,DC=COM].
2011-09-26 14:37:31 INF LDAP_10002 UserManagementService  LDAP user [yyyyy] in security domain[12345] queried with LDAP DN[CN=yyyyy,CN=Users,DC=DOMAIN,DC=COM].
2011-09-26 14:37:31 INF LDAP_10002 UserManagementService  LDAP user [zzzzz] in security domain [12345]queried with LDAP DN[CN=zzzzz,CN=Users,DC=DOMAIN,DC=COM].
2011-09-26 14:37:31 INF LDAP_10003 UserManagementService  LDAP group [INFA_LDAP] insecurity domain [12345] queried from LDAP.
2011-09-26 14:37:31 INF LDAP_10004 UserManagementService  LDAP group [INFA_LDAP] insecurity domain [12345] queried from LDAPcontains user [yyyyy] in security domain[12345].
2011-09-26 14:37:31 INF LDAP_10004 UserManagementService  LDAP group [INFA_LDAP] insecurity domain [12345] queried from LDAPcontains user [zzzzz] in security domain[12345].
2011-09-26 14:37:31 INF LDAP_10004 UserManagementService  LDAP group [INFA_LDAP] insecurity domain [12345] queried from LDAPcontains user [sreich] in security domain[12345].
2011-09-26 14:37:31 INF LDAP_10003 UserManagementService  LDAP group [INFA_ADM_LDAP] insecurity domain [12345] queried from LDAP.
2011-09-26 14:37:31 INF LDAP_10004 UserManagementService  LDAP group [INFA_ADM_LDAP] insecurity domain [12345] queried from LDAPcontains user [yyyyy] in security domain[12345].
2011-09-26 14:37:31 INF LDAP_10004 UserManagementService  LDAP group [INFA_ADM_LDAP] insecurity domain [12345] queried from LDAPcontains user [zzzzz] in security domain[12345].
2011-09-26 14:37:31 INF LDAP_10000 UserManagementService  Queried [3] users based on the user search base[CN=Users,DC=DOMAIN,DC=COM] with filter[(&(ObjectClass=user)(|(memberOf=CN=INFA_LDAP_ADM,\CN=Users,DC=DOMAIN,DC=COM)(memberOf=CN=INFA_LDAP,CN=Users,DC=DOMAIN,DC=COM)))] specified in[12345] security domain definition.
2011-09-26 14:37:31 INF LDAP_10001 UserManagementService  Queried [2] groups based on the group search base[CN=Users,DC=DOMAIN,DC=COM] with filter[(&(ObjectClass=group)(|(CN=INFA_ LDAP_ADM)(CN=INFA_LDAP_DEV)))]specified in [12345] security domain definition.
2011-09-26 14:37:32 INF LDAP_10007 UserManagementSe       Security domain [12345] synchronization withLDAP server successful.
$
$
$

tyler_durden
# 5  
Old 09-27-2011
Hi Tyler,

Thanks for your wonderful solution. It looks great. However, I do not want to do it perl since most of the script is already written in kourne shell except this data formatting piece.

Would you help me integrating this piece using shell?

Thanks
# 6  
Old 09-27-2011
Nope, sorry.

tyler_durden
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Find the count of files by last created date based on the given date range

My unix version is IBM AIX Version 6.1 I tried google my requirement and found the below answer, find . -newermt “2012-06-15 08:13" ! -newermt “2012-06-15 18:20" But newer command is not working in AIX version 6.1 unix I have given my requirement below: Input: atr files: ... (1 Reply)
Discussion started by: yuvaa27
1 Replies

2. UNIX for Dummies Questions & Answers

Search specific string logfile specific date range

Hi, I have logfile like this.. === 2014-02-09 15:46:59,936 INFO RequestContext - URL: '/eyisp/sc/skins/EY/images/pickers/comboBoxPicker_Over.png', User-Agent: 'Mozilla/5.0 (Windows NT 6.1; WOW64; Trident/7.0; rv:11.0) like Gecko': Unsupported with Accept-Encoding header === 2015-02-09... (8 Replies)
Discussion started by: kishk
8 Replies

3. UNIX for Dummies Questions & Answers

Search for a specific String in a log file for a specific date range

Hi, I have log file which rolls out every second which is as this. HttpGenRequest - -<!--OXi dbPublish--> <created="2014-03-24 23:45:37" lastMsgId="" requestTime="0.0333"> <response request="getOutcomeDetails" code="114" message="Request found no matching data" debug="" provider="undefined"/>... (3 Replies)
Discussion started by: karthikprakash
3 Replies

4. Shell Programming and Scripting

Search on date range of file based on user input

Hello I would like to ask for help with a script to search a directory that contains many log files and based on a users input after being prompted, they enter a date range down to the hour which searches the files that contain that range. I dont know how to go about this. I am hoping that the... (5 Replies)
Discussion started by: lostincashe
5 Replies

5. Shell Programming and Scripting

Extracting specific lines of data from a file and related lines of data based on a grep value range?

Hi, I have one file, say file 1, that has data like below where 19900107 is the date, 19900107 12 144 129 0.7380047 19900108 12 168 129 0.3149017 19900109 12 192 129 3.2766666E-02 ... (3 Replies)
Discussion started by: Wynner
3 Replies

6. Shell Programming and Scripting

Extract data based on specific search criteria

I have a huge file (about 2 millions records) contains data separated by “,” (comma). As part of the requirement, I can't change the format. The objective is to remove some of the records with the following condition. If the 23rd field on each line start with 302 , I need to remove that from the... (4 Replies)
Discussion started by: jaygamini
4 Replies

7. Shell Programming and Scripting

Merge two file data together based on specific pattern match

My input: File_1: 2000_t g1110.b1 abb.1 2001_t g1111.b1 abb.2 abb.2 g1112.b1 abb.3 2002_t . . File_2: 2000_t Ali england 135 abb.1 Zoe british 150 2001_t Ali england 305 g1111.b1 Lucy russia 126 (6 Replies)
Discussion started by: patrick87
6 Replies

8. Shell Programming and Scripting

Get Data Between a specific Date Range from logs

I need to extract data from logs for a mentioned date range..Its quite urgent can anyone help me out with it..its to be written in unix..just thought its better to specify.. (4 Replies)
Discussion started by: sankasu
4 Replies

9. Shell Programming and Scripting

Read Write byte range/chunk of data from specific location in file

I am new to Unix so will really appreciate if someone can guide me on this. What I want to do is: Step1: Read binary file - pick first 2 bytes, convert from hex to decimal. Read the next 3 bytes as well. 2 bytes will specify the number of bytes 'n' that I want to read and write... (1 Reply)
Discussion started by: Kbenipel
1 Replies

10. Shell Programming and Scripting

Report file extraction based on Date range

Hi all, Iam writing a script, which will extract all the files from Start_Date to End_Date. Files are date stamped as YYYYMMDD. For ex: Start_Date='20051001' End_Date='20060331' extract files such as........ ramp_20050810.rpt ramp_20050915.rpt ramp_20051001.rpt ramp_20051010.rpt... (2 Replies)
Discussion started by: ganapati
2 Replies
Login or Register to Ask a Question