[ksh] finding last file with keyword in it


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting [ksh] finding last file with keyword in it
# 1  
Old 01-15-2013
[ksh] finding last file with keyword in it

Hi,

In short : I have several log files and I need to find the last file with a certain keyword in it.
Code:
# ls -1tr logs
log_hostX.Jan01_0100.gz
log_hostX.Jan01_0105.gz
log_hostX.Jan01_0110.gz
log_hostX.Jan01_0115.gz
log_hostX.Jan01_0120.gz
log_hostX.Jan01_0125.gz
log_hostX.Jan01_0130.gz
log_hostX

Through Internet I found a way to examine the last, the previous and the previous previous file, but for some reason cannot determine the name of the file before that.
My purpose is to find the begin date & time and the end date & time.
The end time is always in the last file, but the begin time can be the last 4 files.
Because of the number of log files, I need to go from newest to oldest.

I will post my code so far in the next posting.

Greetings,

E.J.

---------- Post updated at 10:49 AM ---------- Previous update was at 10:38 AM ----------

Here the code.
Perhaps not the best you may have seen, but it almost serves it purpose.
Line 109 is where I need help.
You can copy&paste it to try it out.
But of course when there are better solutions I would like to hear them too.
Always nice to learn something.

I am on a Solaris 10 machine with ksh88 (not allowed to upgrade it).

Code:
#!/usr/bin/ksh

# Prepare log dir
mkdir -p /tmp/logs
cd /tmp/logs

rm *.gz log_hostX

# Prepare some log files to play with
echo "log1 continue 01:00:00" > log_hostX.Jan01_0100
gzip log_hostX.Jan01_0100
sleep 1
echo "log2 continue 01:05:00 \nlog2 END 01:05:00" > log_hostX.Jan01_0105
gzip log_hostX.Jan01_0105
sleep 1
echo "log3 BEGIN 01:10:00\nlog3 continue 01:10:00" > log_hostX.Jan01_0110
gzip log_hostX.Jan01_0110
sleep 1
echo "log4 continue 01:15:00\nlog4 END 01:15:00" > log_hostX.Jan01_0115
gzip log_hostX.Jan01_0115
sleep 1
echo "log5 BEGIN 01:20:00\nlog5 continue 01:20:00" > log_hostX.Jan01_0120
gzip log_hostX.Jan01_0120
sleep 1
echo "log6 continue 01:25:00" > log_hostX.Jan01_0125
gzip log_hostX.Jan01_0125
sleep 1
echo "log7 continue 01:30:00" > log_hostX.Jan01_0130
gzip log_hostX.Jan01_0130
sleep 1
echo "log8 continue 01:35:00\nlog8 END 01:35:00" > log_hostX


### Actual code ###

# Set variables to default values
BEGIN_Mmm_DD="-----"
BEGIN_HH_MM_SS="--:--:--"
END_Mmm_DD="-----"
END_HH_MM_SS="--:--:--"

HOST=hostX
FILE=log_${HOST}

# Determine end time based on last file with END
if grep -i END ${FILE} > /dev/null
then
  END_HH_MM_SS=`grep -i END ${FILE} | awk '{print $3}'`
fi

# Determine begin and end MmmDD based on log file
BEGIN_Mmm_DD=`ls -als ${FILE} | awk '{printf "%s%0.2d", $7, $8}'`
END_Mmm_DD=$BEGIN_Mmm_DD

# Try to determine begin time based on last file with BEGIN
if grep BEGIN ${FILE} > /dev/null
then
  BEGIN_HH_MM_SS=`grep BEGIN ${FILE} | awk '{print $3}'`
fi

# BEGIN not found ?
# Try to determine the begin time in the previous trace file
if [[ $BEGIN_HH_MM_SS = "--:--:--" ]]
then
  # Determine filename
  FILE=`ls -1tr log_${HOST}.*.gz 2> /dev/null | awk '{X=$0} END {print X}'`
  if [ -n "$FILE" ]
    then
    # Determine date
    BEGIN_Mmm_DD=`ls -als ${FILE} | awk '{print $7 $8}'`
    # Determine begin time
    if gzcat ${FILE} | grep BEGIN > /dev/null
    then
     	BEGIN_HH_MM_SS=`gzcat ${FILE} | grep BEGIN | awk '{print $3}'`
    else
      BEGIN_Mmm_DD="-----"
      BEGIN_HH_MM_SS="--:--:--"
    fi
  fi
fi


# BEGIN still not found ?
# Try to determine the begin time in the previous previous trace file
if [[ $BEGIN_HH_MM_SS = "--:--:--" ]]
then
  # Determine filename
  FILE=`ls -1tr log_${HOST}.*.gz 2> /dev/null | awk '{y=x "\t" $0; x=$0}; END{print y}' | awk '{print $1}'`
  if [ -n "$FILE" ]                                                      
    then
    # Determine date
    BEGIN_Mmm_DD=`ls -als ${FILE} | awk '{print $7 $8}'`
    # Determine begin time
    if gzcat ${FILE} | grep BEGIN > /dev/null
    then
     	BEGIN_HH_MM_SS=`gzcat ${FILE} | grep BEGIN | awk '{print $3}'`
    else
      BEGIN_Mmm_DD="-----"
      BEGIN_HH_MM_SS="--:--:--"
    fi
  fi
fi

# What ? BEGIN still not found ?
# Try to determine the begin time in the previous previous previous trace file
if [[ $BEGIN_HH_MM_SS = "--:--:--" ]]
then
  # Determine filename
  FILE=`echo "THAT IS THE QUESTION !!"`
  FILE=log_hostX.Jan01_0120.gz
  if [ -n "$FILE" ]
    then
    # Determine date
    BEGIN_Mmm_DD=`ls -als ${FILE} | awk '{print $7 $8}'`
    # Determine begin time
    if gzcat ${FILE} | grep BEGIN > /dev/null
    then
     	BEGIN_HH_MM_SS=`gzcat ${FILE} | grep BEGIN | awk '{print $3}'`
    else
      BEGIN_Mmm_DD="-----"
      BEGIN_HH_MM_SS="--:--:--"
    fi
  fi
fi

# Print the header lines
printf "Host    Start             End\n"
printf "\n"

# Print the hostname
printf "%-8s" ${HOST}

# Print the results
printf "%-5s  %-8s   %-5s %-8s\n" ${BEGIN_Mmm_DD} ${BEGIN_HH_MM_SS} ${END_Mmm_DD} ${END_HH_MM_SS}

Here the result:

Code:
# ./LogStat.sh
Host    Start             End

hostX   Jan15  01:20:00   Jan15 01:35:00

---------- Post updated at 04:04 PM ---------- Previous update was at 10:49 AM ----------

Okay, I was clearly on the wrong track.
There is an easier way to go through a list of files.

This is what works for me in order to find the begin date & time.
Probably it can be made even more compact, but I am served.
Code:
BEGIN_Mmm_DD="-----"
BEGIN_HH_MM_SS="--:--:--"

HOST=hostX

cd /tmp/logs
LOG_LIST=`ls -1t log_${HOST}*`

for LOG_FILE in $LOG_LIST
  do
  
  # Determine the extension of the log file
  EXTENSION=$(echo ${LOG_FILE}|awk -F\. '{print $3}')
  EXTENSION=${EXTENSION:-none}
  if [ ! ${EXTENSION} == "gz" ]
  then
    # Try to determine begin date and time 
    if grep BEGIN ${LOG_FILE} > /dev/null
    then
      BEGIN_Mmm_DD=`ls -als ${LOG_FILE} | awk '{print $7 $8}'`
      BEGIN_HH_MM_SS=`grep BEGIN ${LOG_FILE} | awk '{print $3}'`
      break # we found the date and time, so we can leave the for loop
    fi
  else
    # Try to determine begin date and time
    if gzcat ${LOG_FILE} | grep BEGIN > /dev/null
    then
      BEGIN_Mmm_DD=`ls -als ${LOG_FILE} | awk '{print $7 $8}'`
      BEGIN_HH_MM_SS=`gzcat ${LOG_FILE} | grep BEGIN | awk '{print $3}'`
      break # we found the date and time, so we can leave the for loop
    fi
  fi

done

This User Gave Thanks to ejdv For This Post:
# 2  
Old 01-15-2013
Not sure I understand your request.
So - the end time will be in the last file (key word END), and the begin (keyword BEGIN) will be in the same file or in four predecessors. Why not loop up to five times through ls -t LOG_hostx* (watch out, no -r option!), grepping each file for "BEGIN"? If there's more than one BEGIN and you need the last one, try tacing the files.
# 3  
Old 01-16-2013
[SOLVED] - [ksh] finding last file with keyword in it

Quote:
Originally Posted by RudiC
Not sure I understand your request.
So - the end time will be in the last file (key word END), and the begin (keyword BEGIN) will be in the same file or in four predecessors. Why not loop up to five times through ls -t LOG_hostx* (watch out, no -r option!), grepping each file for "BEGIN"? If there's more than one BEGIN and you need the last one, try tacing the files.
Indeed, this was the solution, I was on the wrong track.
That happens when you are expanding the code, or when you adapt the code to little changes.
In my case, first the BEGIN/END sequence was in the last file only, then spread over 2 files, then over 3 and currently over 4.
So I kept looking for the next file, not paying attention to a sound solution.
Now it can be in any file, so is future proof.
Thanks for responding.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

How to align/sort the column pairs of an csv file, based on keyword word specified in another file?

I have a csv file as shown below, xop_thy 80 avr_njk 50 str_nyu 60 avr_irt 70 str_nhj 60 avr_ngt 50 str_tgt 80 xop_nmg 50 xop_nth 40 cyv_gty 40 cop_thl 40 vir_tyk 80 vir_plo 20 vir_thk 40 ijk_yuc 70 cop_thy 70 ijk_yuc 80 irt_hgt 80 I need to align/sort the csv file based... (7 Replies)
Discussion started by: dineshkumarsrk
7 Replies

2. Shell Programming and Scripting

Bash to print if keyword not in file

I am trying to create an output file new that contains only the S5-00580 lines from list that are not in analysis_log. My attempt to do this is below. The new file would be used in the aria2c command to download only new folders. The aria2c command works to download all the files in list, but... (7 Replies)
Discussion started by: cmccabe
7 Replies

3. Shell Programming and Scripting

Need to extract the word after a particular keyword throughout the file..

Hi Everyone, Need help in extracting the hostname from the below output. Expected output: DS-TESTB-GDS-1.TEST.ABC.COM DS-TESTB-GDS-2.TEST.ABC.COM .... ... /tmp $ cat -n /tmp/patchreport 1 /usr/bin/perl /admin/bin/patch/applyPatches.pl --apply_patches... (4 Replies)
Discussion started by: thiyagoo
4 Replies

4. Shell Programming and Scripting

Search for a Keyword in file and replace another keyword or add at the end of line

Hi I want to implement something like this: if( keyword1 exists) then check if(keyword2 exists in the same line) then replace keyword 2 with New_Keyword else Add New_Keyword at the end of line end if eg: Check for Keyword JUNGLE and add/replace... (7 Replies)
Discussion started by: dashing201
7 Replies

5. Shell Programming and Scripting

Finding file pattern in ksh 88

Hi, I've to find the file which has the pattern "Delete Report for History Tables" and need to search this file pattern from directory which has sub directories as well. I'm using ksh 88 Please suggest me which command will be used to find the file pattern . Thanks. (1 Reply)
Discussion started by: smile689
1 Replies

6. Shell Programming and Scripting

Reading file contents until a keyword

Hi Guys, I need to read a file until I find a blank line. and in the next iteration I want to continue reading from the line I find a keyword. For ex: my file looks like PDS_JOB_ALIAS CRITERIA_ITEM_TYPE PDS_JOB_CRITERIA_ITEM CRITERIA_ITEM_TYPE First I want to read the file... (2 Replies)
Discussion started by: infintenumbers
2 Replies

7. Solaris

Keyword search input from a file

Hi, I have a file which got only one column and got some keywords. I have another file where the keywords used in the first file are repeated in the second file. Now I would like to know how many times each keyword from the first file is repeated in the second file. Request your help on... (1 Reply)
Discussion started by: pointers
1 Replies

8. Shell Programming and Scripting

complex if statement syntax without using 'if ..' keyword in ksh.

It saves me lot of typing and space/lines when I do not use full 'if' keyword and construct, instead use .. && <statement> || <statement> that perfectly replaces.. if ; then <statement> else <statement> fi Can I use following syntax when I want to add multiple statements under 'if'... (4 Replies)
Discussion started by: kchinnam
4 Replies

9. Shell Programming and Scripting

Ksh - finding pattern in file and generating a new file

I am trying to find for the pattern in first 5 bytes of every line in a text file and then generate a new file with the pattern found. Example: expected pattern is '-' to be serached in first 5 bytes of file only. Input File ab-cd jan-09 ddddd jan09 cc-ww jan09 dsgdq jan-09 ... (2 Replies)
Discussion started by: net
2 Replies

10. Shell Programming and Scripting

how to search a keyword within a file using a for loop

hi guys i have a problem here, im trying to stablish a relationship between a text file and an input user for example the script is going to prompt the user for some football team and what the script is going to do is return the colums in which that input is located so far this is what i have ... (6 Replies)
Discussion started by: lucho_1
6 Replies
Login or Register to Ask a Question