Filtering log file with lines older than 10 days. Post: 303006444

Sponsored Content

Top Forums Shell Programming and Scripting Filtering log file with lines older than 10 days. Post 303006444 by shunya on Thursday 2nd of November 2017 10:01:47 AM

11-02-2017

Registered User

Filtering log file with lines older than 10 days.

Hi,
I am trying to compare epoch time in a huge log file (2 million lines) with todays date. I have to create two files one which has lines older than 10 days and another file with less than 10 days. I am using while do but it takes forever to complete the script. It would be helpful if you can help me convert this into awk format to make it faster.

Input file field 2 has epoch time up to nano seconds.

INPUT

Code:

/kshdmjv/1/1/        1508426244789202297     101434  407896064       409265492  48817   BB      408811357       214089175
/PCD0289/0lshdmk6/1/1        1508426298102701719     101435  3744464896      3756961760 445770  KK      3756575209      1934910527
/PCD0289/0mshdml0/1/1        1508426323091171565     101436  3749707776      3762220364 446331  LL      3761465177      1943293261

Current code

Code:

#!/bin/bash
NOWT=`expr $(date +%s%3N) / 1000`
echo -e " Time now is $NOWT"
while read line ; do
FST=$(echo $line |awk '{print $1}');
SND=$(echo $line |awk '{print $2}');
RST=$(echo $line |awk '{ print substr($0, index($0,$3)) }');
OLDT=$((SND/1000000000))
DIFT=$(expr ${NOWT} - ${OLDT});

   T=$DIFT
   D=$((T/60/60/24))
   H=$((T/60/60%24))
   M=$((T/60%60))
   S=$((T%60))
   if [[ ${D} -gt 10 ]]
   then
       printf '%d days %02d:%02d:%02d %s,%d,%s\n' $D $H $M $S $FST $SND $RST >> olderthn10days.txt
   else
       printf '%d dayless %s,%d,%s\n' $D $FST $SND $RST >> lessthn10days.txt
   fi
done < 0108.txt

Please advise if there is any way to make this faster and better.

shunya

View Public Profile for shunya

Find all posts by shunya

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Delete user file(s) older then 'X' days ??

I want to delete any file in unix file system which is older then a week. Those files should not be unix system file..means it should be user created file. Any clue to this ?? ASAP. Thanks.

2. Shell Programming and Scripting

delete file older than N days

Hi, All, I'd like to delete files older than 1 day. I thought the following command find /your_directory -mtime +1-exec rm -f {} \; will do the work, but not so, it seems like it won't delete files unless it is 2 days old or older. the files between 1 day and 2 days old does not...

3. UNIX for Dummies Questions & Answers

Delete file older than three days

I am using SFTP to transmit files from the Mainframe to an UNIX server. I am looking for some kind of script that runs with SFTP to delete tranmitted files older than 3 days. Can this be done in a SFTP transmission batch job?

4. Shell Programming and Scripting

Deleting entries older than 7 days from a file

Hi, I have a file which contains entries in this format. my-bin.000140 my-bin.000141 my-bin.000142 my-bin.000143 my-bin.000144 my-bin.000145 my-bin.000146 my-bin.000147 my-bin.000148 my-bin.000149 my-bin.000150 my-bin.000151 my-bin.000152 my-bin.000153 my-bin.000154...

5. Shell Programming and Scripting

[Solved] Remove file older than 90 days

I have crontab job a tar file to a directory ( tar -cvf /tmp/backup/or.`date +%m%d%y`. /ora/db/* ) , it will do it every day . Now I don't want to keep too much files , I just want to keep the file for 90 days , can advise if I want to remove the backup file which are elder than 90 days , can...

6. Shell Programming and Scripting

Delete log files content older than 30 days and append the lastest date log file date

To delete log files content older than 30 days and append the lastest date log file date in the respective logs I want to write a shell script that deletes all log files content older than 30 days and append the lastest log file date in the respective logs This is my script cd...

7. UNIX for Dummies Questions & Answers

Find all log files under all file systems older than 2 days and zip them

Hi All, Problem Statement:Find all log files under all file systems older than 2 days and zip them. Find all zip files older than 3days and remove them. Also this has to be set under cron. I have a concerns here find . -mtime +2 -iname "*.log" -exec gzip {} Not sure if this will work as...

8. Shell Programming and Scripting

Remove lines older than 30 days

Hi Experts/Gurus, Is there a way to remove lines in a file that are older than x days (i.e. 30 days) based on the date stamp in the first column? Example. $ date Sat Jan 11 14:12:06 EDT 2014 $cat sample.txt 10-10-2013 09:00:01 AM|Line test 1234567 16-10-2013 08:30:00 AM|Line test...

9. Shell Programming and Scripting

Search 2 days older file and then delete last 10 lines

I want to search 2 day older file and then delete last 10 line of that file.

10. Shell Programming and Scripting

To nullify multiple log files older than X days.

Hi, I am writing the script for tomcat logs clean up ending with *.txt and *.log. cleanup should be applied to logs older than 7 days logs same will be scheduled in cron. after googling found below but it only help partially. find . -type f -name '*.log' -exec truncate --size 0 "{}" \; I...

LEARN ABOUT DEBIAN

lchage

lchage(8)						      System Manager's Manual							 lchage(8)

NAME

       lchage - Display or change user password policy

SYNOPSIS

       lchage [OPTION]... user

DESCRIPTION

       Displays or allows changing password policy of user.

OPTIONS

       -d, --date=days
	      Set the date of last password change to days after Jan 1 1970.

       -E, --expire=days
	      Set the account expiration date to days after Jan 1 1970.  Set days to -1 to disable account expiration.

       -i, --interactive
	      Ask all questions when connecting to the user database, even if default answers are set up in libuser configuration.

       -I, --inactive=days
	      Disable the account after days after password expires (after the user user is required to change the password).

       -l, --list
	      Only list current user's policy and make no changes.

       -m, --mindays=days
	      Require at least days days between password changes.  Set days to 0 to disable this checking.

       -M, --maxdays=days
	      Require changing the password after days since last password change.  Set days to 99999 to disable this checking.

       -W, --warndays=days
	      Start warning the user days before password expires (before the user is required to change the password).

EXIT STATUS

       The exit status is 0 on success, nonzero on error.

libuser 							    Jan 12 2005 							 lchage(8)

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Delete user file(s) older then 'X' days ??

Discussion started by: varungupta

2. Shell Programming and Scripting

delete file older than N days

Discussion started by: ericaworld

3. UNIX for Dummies Questions & Answers

Delete file older than three days

Discussion started by: Steve Carlson

4. Shell Programming and Scripting

Deleting entries older than 7 days from a file

Discussion started by: arijitsaha