Login or Register to Ask a Question and Join Our Community


UNIX for Beginners Questions & Answers

Copy last 30 minutes' contents from a log file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums UNIX for Beginners Questions & Answers Copy last 30 minutes' contents from a log file
# 1  
Old 10-22-2019
Copy last 30 minutes' contents from a log file
Hi Guys,

I am writing a bash script to capture the last 30 minutes's contents from log file to a new file. This job is a scheduled job and will run every 30 minutes. The log file is db2diag.log in DB2. I am having difficulties copying the last 30 minutes's contents. Can someone please help me. Thanks

The sample log file is like

Code:
019-10-22-00.02.53.593412-300 E2310A623            LEVEL: Warning
PID     : 17432874             TID : 8134           PROC : db2sysc 0
INSTANCE: db2pdtrg             NODE : 000           DB   : GWDITRGP
APPHDL  : 0-39400              APPID: 10.33.39.144.51753.191022050231
AUTHID  : DSMMONDB             HOSTNAME: lcofndbp1
EDUID   : 8134                 EDUNAME: db2agent (GWDITRGP) 0
FUNCTION: DB2 UDB, RAS/PD component, PANotifLogColl::paGetNotifyLogFiles, probe:50
MESSAGE : ECF=0x9000001A=-1879048166=ECF_FILE_DOESNT_EXIST
          File doesn't exist
DATA #1 : String, 37 bytes
No valid notification log file found.

2019-10-22-00.02.53.630281-300 E2934A623            LEVEL: Warning
PID     : 17432874             TID : 8134           PROC : db2sysc 0
INSTANCE: db2pdtrg             NODE : 000           DB   : GWDITRGP
APPHDL  : 0-39400              APPID: 10.33.39.144.51753.191022050231
AUTHID  : DSMMONDB             HOSTNAME: lcofndbp1
EDUID   : 8134                 EDUNAME: db2agent (GWDITRGP) 0
FUNCTION: DB2 UDB, RAS/PD component, PANotifLogColl::paGetNotifyLogFiles, probe:50
MESSAGE : ECF=0x9000001A=-1879048166=ECF_FILE_DOESNT_EXIST
          File doesn't exist
DATA #1 : String, 37 bytes
No valid notification log file found.

2019-10-22-00.04.17.390138-300 E3558A576            LEVEL: Error
PID     : 17432874             TID : 152830         PROC : db2sysc 0
INSTANCE: db2pdtrg             NODE : 000           DB   : GWDITRGP
APPHDL  : 0-46568              APPID: 10.33.39.144.51779.191022050420
AUTHID  : DSMMONDB             HOSTNAME: lcofndbp1
EDUID   : 152830               EDUNAME: db2agent (GWDITRGP) 0
FUNCTION: DB2 UDB, catalog services, sqlrl_evmon_eventtable, probe:2540
MESSAGE : ADM4002W  The Event Monitor target table "LOCK_EVENT" (table schema 
          "IBM_RTMON" ) already exists.

Last edited by Scrutinizer; 10-22-2019 at 01:13 PM.. Reason: code tags
# 2  
Old 10-22-2019
How I've tackled this type of problem in the past is to have the script "Remember" where it got up to in a file and start from there next time.

Example:
Code:
LOGFILE=/DB2/db2diag.log 
NEWFILE=${LOGFILE%.log}_$(date +%Y%m%d%H%M).log
CHKFILE=/var/scriptdata/db2diagline.dat

EndLine=$(awk 'END{print NR}' $LOGFILE)
if [ -f $CHKFILE ]
then
    read StartLine < $CHFILE
    StartLine=${StartLine:-0}
    # Test for log file truncated
    [ $EndLine -lt $StartLine ] && StartLine=0
else
    StartLine=0
fi

awk -vS=$StartLine -vE=$EndLine 'NR>E{exit} NR>S' $LOGFILE > $NEWFILE

#Save EndLine for StartLine next time
echo "$EndLine" > $CHKFILE

Last edited by Chubler_XL; 10-22-2019 at 05:27 PM.. Reason: Added code to generate new filename (add datetime to original logfile name)
This User Gave Thanks to Chubler_XL For This Post:
naveed
# 3  
Old 10-25-2019
Thanks a lot for your help
# 4  
Old 10-25-2019
If you have logrotation in place you may want to know if a log-switch happened. I did this by checking if the first line of a file is still the same. (Hopefully it's not an empty line ;-) ) ... like this:

Code:
LOGFILE=/DB2/db2diag.log 
NEWFILE=${LOGFILE%.log}_$(date +%Y%m%d%H%M).log
CHKFILE=/var/scriptdata/db2diagline.dat
CHKFILE2=/var/scriptdata/db2firstline.dat
read FIRSTLINE_SAVED < $CHKFILE2
FIRSTLINE="$(head -n1 $LOGFILE)"

if [ "$FIRSTLINE" == "$FIRSTLINE_SAVED" ] ; then 
   EndLine=$(awk 'END{print NR}' $LOGFILE)
   if [ -f $CHKFILE ] ; then
       read StartLine < $CHFILE
       StartLine=${StartLine:-0}
       # Test for log file truncated
       [ $EndLine -lt $StartLine ] && StartLine=0
   else
      StartLine=0
   fi
else
   StartLine=0
   head -n1 $LOGFILE > $CHKFILE2
fi


awk -vS=$StartLine -vE=$EndLine 'NR>E{exit} NR>S' $LOGFILE > $NEWFILE

#Save EndLine for StartLine next time
echo "$EndLine" > $CHKFILE

Last edited by stomp; 10-25-2019 at 04:35 PM..
# 5  
Old 10-25-2019
Assumption is that if a logrotation occurs the resulting file will be smaller that the last run.
This is identified with [ $EndLine -lt $StartLine ] && StartLine=0

On systems I have worked with log files grow at a fairly consistent rate and logrotation only occurs every week.

Not to say it's impossible but and I've never encountered a file that got truncated and then grew bigger than it's previous size within 30mins.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Grep a log file for the last 5 minutes of contents every 5 minutes

Hi all, System Ubuntu 16.04.3 LTS i have the following log INFO 2019-02-07 15:13:31,099 module.py:700] default: "POST /join/8550614e-3e94-4fa5-9ab2-135eefa69c1b HTTP/1.0" 500 2042 INFO 2019-02-07 15:13:31,569 module.py:700] default: "POST /join/6cb9c452-dcb1-45f3-bcca-e33f5d450105... (15 Replies)
Discussion started by: charli1
15 Replies

2. Shell Programming and Scripting

Copy contents of one file to another

I need to write a script (in bash) that copies the content of the first file in each folder of a directory to the second file in the same folder. I tried this and it didn't work - it just came back with errors and I'm not sure how to fix it. Help is very much appreciated! for mpdir in... (4 Replies)
Discussion started by: LeftoverStew
4 Replies

3. Shell Programming and Scripting

Need Script to copy the contents of two files into one file

Hi i need Script to copy the contents of two files into one file i have 2 fil X1.txt / X2.txt i need script to copy the contents of X1 and X2 In AllXfile X1.txt File X1 X2.txt File X2 AllXfile.txt File X1 File X2 (2 Replies)
Discussion started by: azzeddine2005
2 Replies

4. Shell Programming and Scripting

Need help to copy contents of a file

Hi, I am stuck up with a problem of copying the contents of a directory where one of the folder name is changed daily. Problem: I have the folder structure as: RefWorlds2/LINGCC4_X64/odsdev/odessy/UTI/621GA_build_xxx/.../.. In the above path the build number (xxx) will be changed... (3 Replies)
Discussion started by: SathaKarni
3 Replies

5. UNIX for Dummies Questions & Answers

Copy entire contents of file to clipboard

Hi, I am trying to figure out how to copy the contents of a file to the clipboard, then paste into a command. i.e copy contents of file /path/filename.txt to <command> <paste text> Hope that makes sense. Basically tryting to copy the text for use in a command without having to open the... (8 Replies)
Discussion started by: JCA70
8 Replies

6. Shell Programming and Scripting

copy the contents between two keywords to a new file.

Hi All, I want to edit my gate level netlists by searching for the content between two patterns eg: ff1 \test/a0 ( .CLK(\test/ClkInt0_acb_00x1 ),.D(\test/Rakicc ), .QB(\test/X ), .VDD(1'b1), .VSS(1'b0)); ff1 \test/a1 ( .CLK(\test/medis0_acb_00x1 ),.D(\test/hedwc ), .QB(\test/X ),... (6 Replies)
Discussion started by: naveen@
6 Replies

7. Shell Programming and Scripting

Copy contents of a directory only if a file exists

I'm looking to write a script that will check the contents of a directory, and if any files exist in that directory copy them to a temporary folder. The target files are only resident for a few seconds, so I think the script needs to be running constantly. Any pointers would be really... (3 Replies)
Discussion started by: danceofillusion
3 Replies

8. Shell Programming and Scripting

Bash copy file contents into an existing file at a specific location

Hi all I need to copy the entire contents of one file into an existing file at a specific location. I know the exact line number where I need to put it. It appears I would use either sed or awk to do this, but I have been unsuccessful so far: File A line 1 line 2 line 3 line 4 ... (6 Replies)
Discussion started by: gshepherd7
6 Replies

9. Shell Programming and Scripting

Automatic Copy of File Contents to Clipboard

Could someone show me how to copy the contents of a file to the clipboard automatically without manually selecting its contents? I just want to press the "Paste Key" to show the results. I wish to use this in a ksh script. I'm using Solaris. Thanks! (5 Replies)
Discussion started by: ilak1008
5 Replies

10. Shell Programming and Scripting

Copy selected contents from file

I want to capture contents of a file between 2 strings into another file for eg all lines in between the keywords "start log" and "end log" should be copied into another file (4 Replies)
Discussion started by: misenkiser
4 Replies
Login or Register to Ask a Question