Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Filtering log file with lines older than 10 days.

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Filtering log file with lines older than 10 days.
# 1  
Old 11-02-2017
Linux Filtering log file with lines older than 10 days.
Hi,
I am trying to compare epoch time in a huge log file (2 million lines) with todays date. I have to create two files one which has lines older than 10 days and another file with less than 10 days. I am using while do but it takes forever to complete the script. It would be helpful if you can help me convert this into awk format to make it faster.

Input file field 2 has epoch time up to nano seconds.

INPUT
Code:
/kshdmjv/1/1/        1508426244789202297     101434  407896064       409265492  48817   BB      408811357       214089175
/PCD0289/0lshdmk6/1/1        1508426298102701719     101435  3744464896      3756961760 445770  KK      3756575209      1934910527
/PCD0289/0mshdml0/1/1        1508426323091171565     101436  3749707776      3762220364 446331  LL      3761465177      1943293261

Current code
Code:
#!/bin/bash
NOWT=`expr $(date +%s%3N) / 1000`
echo -e " Time now is $NOWT"
while read line ; do
FST=$(echo $line |awk '{print $1}');
SND=$(echo $line |awk '{print $2}');
RST=$(echo $line |awk '{ print substr($0, index($0,$3)) }');
OLDT=$((SND/1000000000))
DIFT=$(expr ${NOWT} - ${OLDT});

   T=$DIFT
   D=$((T/60/60/24))
   H=$((T/60/60%24))
   M=$((T/60%60))
   S=$((T%60))
   if [[ ${D} -gt 10 ]]
   then
       printf '%d days %02d:%02d:%02d %s,%d,%s\n' $D $H $M $S $FST $SND $RST >> olderthn10days.txt
   else
       printf '%d dayless %s,%d,%s\n' $D $FST $SND $RST >> lessthn10days.txt
   fi
done < 0108.txt

Please advise if there is any way to make this faster and better.
# 2  
Old 11-02-2017
If there are 2 million lines this might help reduce 6 million subprocesses.
Code:
#!/bin/bash
NOWT=$(expr $(date +%s%3N) / 1000)
echo -e " Time now is $NOWT"
while read FST SND RST ; do
OLDT=$((SND/1000000000))
DIFT=$(expr ${NOWT} - ${OLDT});

   T=$DIFT
   D=$((T/60/60/24))
   H=$((T/60/60%24))
   M=$((T/60%60))
   S=$((T%60))
   if [[ ${D} -gt 10 ]]
   then
       printf '%d days %02d:%02d:%02d %s,%d,%s\n' $D $H $M $S $FST $SND $RST >> olderthn10days.txt
   else
       printf '%d dayless %s,%d,%s\n' $D $FST $SND $RST >> lessthn10days.txt
   fi
done < 0108.txt

But I cannot see that the printfs are working correctly as $RST will have embedded spaces in it.

If you are using bash4 you could optionally replace the line
Code:
NOWT=$(expr $(date +%s%3N) / 1000)

with
Code:
NOWT=$(expr $(printf "%(%s%3N)T" -2) / 1000)

Andrew
# 3  
Old 11-02-2017
Ten days to the nanosecond - please allow for a reasonable, coarser granularity. And, you should be aware that the result depends on the time of day that you run the script unless you define to use e.g. "today midnight" for the comparison...
Try
Code:
awk 'BEGIN {D10 = (srand()-864000); split ("OLDER YOUNGER", FN)} {print $0 > (FN[1+($2/1E9 > D10)])} ' file

You may want to add the desired conversions of the fields as given above.
# 4  
Old 11-02-2017
Linux Filtering log file with lines older than 10 days.
Hi RudiC,

I tried your awk but I don't get any O/P.

Code:
panz:/tmp> awk 'BEGIN {D10 = (srand()-864000); split ("OLDER YOUNGER", FN)} {print $0 > (FN[1+($2/1E9 > D10)])} ' 0108.txt

---------- Post updated at 10:12 AM ---------- Previous update was at 10:10 AM ----------

We can compare it to Today 12:00AM if that makes it easier.
# 5  
Old 11-02-2017
Sure? Didn't it create the two desired output files?
# 6  
Old 11-02-2017
Linux Filtering log file with lines older than 10 days.
Somehow...It creates only one file YOUNGER and dumps everything in that file.

---------- Post updated at 01:48 PM ---------- Previous update was at 01:38 PM ----------

Here is sample file if you like
Code:
/ECD0303/12s3tbqe/1       1494450414791890180     4566    34359738368     34475924324     4144773 4136493 116784870       115898019
/ECD0303/10s3tbqd/1       1494450429920112522     4567    34338766848     34454780244     4138613 4130341 116576831       115660526
/ECD0303/11s3tbqe/1       1494450399980138278     4568    34359738368     34476107528     4151316 4142956 117616816       116138618
/ECD0303/13s3tbqv/1       1494450274931912967     4569    4194304 4209372 503     AA      3986663 1398647
/ECD0303/14s3tbqv/1       1494450274985871012     4570    7340032 7365436 871     CC      6839625 2791870
/ECD0303/15s3tbqv/1       1494450274997690312     4571    3145728 3157044 369     FC      3041330 1124246
/ECD0303/16s3tbr1/1       1494450283497270749     4572    458227712       459733468       53679   3D      459428840       182225614
/ECD0303/17s3tbr2/1       1494450278445573393     4573    3145728 3156848 362     2C      2983292 1113543
/ECD0303/18s3tbr4/1       1494450288802435061     4574    731906048       734338440       86736   3E      733796743       322044566
/ECD0303/1as3tbuv/1       1494450584139209737     4576    34359738368     34476011096     4147872 3528559 5261036982      1944503212
/exd/doc/3/45 1509129209233091543     72      31958192128     32064917336     3807181 A       32064917336     19422111111
/exd/doc/3/46 1509129209229341353     73      31516553216     31621961332     3760202 B       31621961332     16479692952
/exd/doc/3/47 1509129209242579870     74      30883958784     30995567000     3687020 0FA     30987341948     21703509926
/exd/doc/3/48 1509129209238274317     75      30905245696     31008440092     3681224 0       31008440092     20189779623
/exd/doc/3/49 1509129209246929594     76      28803964928     28906429400     3440802 A0B     28900446928     22970462639
/exd/doc/3/50 1509129209230745672     77      28351578112     28446458816     3384655 0       28446458816     18756473322
/exd/doc/3/51 1509129209240203414     78      786698240       791016112       94114   EDC     789340848       390977698

# 7  
Old 11-02-2017
In your sample data in post#1, NO timestamp is younger than 10 days ago. For testing purposes, I modified one of them, and it worked as desired.
You may want to reverse the comparison operator to test the result...


EDIT: For your new sample, it works briliiantly: the lines with 10. May go to OLDER, the ones from 27.Oct. go to YOUNGER.

EDIT2: What's your awk version? Please post result of
Code:
awk 'BEGIN {print srand()}'
1509649848

Last edited by RudiC; 11-02-2017 at 04:11 PM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

To nullify multiple log files older than X days.

Hi, I am writing the script for tomcat logs clean up ending with *.txt and *.log. cleanup should be applied to logs older than 7 days logs same will be scheduled in cron. after googling found below but it only help partially. find . -type f -name '*.log' -exec truncate --size 0 "{}" \; I... (6 Replies)
Discussion started by: Riverstone
6 Replies

2. Shell Programming and Scripting

Search 2 days older file and then delete last 10 lines

I want to search 2 day older file and then delete last 10 line of that file. (2 Replies)
Discussion started by: sonu pandey
2 Replies

3. Shell Programming and Scripting

Remove lines older than 30 days

Hi Experts/Gurus, Is there a way to remove lines in a file that are older than x days (i.e. 30 days) based on the date stamp in the first column? Example. $ date Sat Jan 11 14:12:06 EDT 2014 $cat sample.txt 10-10-2013 09:00:01 AM|Line test 1234567 16-10-2013 08:30:00 AM|Line test... (6 Replies)
Discussion started by: brichigo
6 Replies

4. UNIX for Dummies Questions & Answers

Find all log files under all file systems older than 2 days and zip them

Hi All, Problem Statement:Find all log files under all file systems older than 2 days and zip them. Find all zip files older than 3days and remove them. Also this has to be set under cron. I have a concerns here find . -mtime +2 -iname "*.log" -exec gzip {} Not sure if this will work as... (4 Replies)
Discussion started by: saurabh.mishra
4 Replies

5. Shell Programming and Scripting

Delete log files content older than 30 days and append the lastest date log file date

To delete log files content older than 30 days and append the lastest date log file date in the respective logs I want to write a shell script that deletes all log files content older than 30 days and append the lastest log file date in the respective logs This is my script cd... (2 Replies)
Discussion started by: sreekumarhari
2 Replies

6. Shell Programming and Scripting

[Solved] Remove file older than 90 days

I have crontab job a tar file to a directory ( tar -cvf /tmp/backup/or.`date +%m%d%y`. /ora/db/* ) , it will do it every day . Now I don't want to keep too much files , I just want to keep the file for 90 days , can advise if I want to remove the backup file which are elder than 90 days , can... (1 Reply)
Discussion started by: ust3
1 Replies

7. Shell Programming and Scripting

Deleting entries older than 7 days from a file

Hi, I have a file which contains entries in this format. my-bin.000140 my-bin.000141 my-bin.000142 my-bin.000143 my-bin.000144 my-bin.000145 my-bin.000146 my-bin.000147 my-bin.000148 my-bin.000149 my-bin.000150 my-bin.000151 my-bin.000152 my-bin.000153 my-bin.000154... (2 Replies)
Discussion started by: arijitsaha
2 Replies

8. UNIX for Dummies Questions & Answers

Delete file older than three days

I am using SFTP to transmit files from the Mainframe to an UNIX server. I am looking for some kind of script that runs with SFTP to delete tranmitted files older than 3 days. Can this be done in a SFTP transmission batch job? (5 Replies)
Discussion started by: Steve Carlson
5 Replies

9. Shell Programming and Scripting

delete file older than N days

Hi, All, I'd like to delete files older than 1 day. I thought the following command find /your_directory -mtime +1-exec rm -f {} \; will do the work, but not so, it seems like it won't delete files unless it is 2 days old or older. the files between 1 day and 2 days old does not... (7 Replies)
Discussion started by: ericaworld
7 Replies

10. UNIX for Advanced & Expert Users

Delete user file(s) older then 'X' days ??

I want to delete any file in unix file system which is older then a week. Those files should not be unix system file..means it should be user created file. Any clue to this ?? ASAP. Thanks. (2 Replies)
Discussion started by: varungupta
2 Replies
Login or Register to Ask a Question