Remove lines older than 30 days


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Remove lines older than 30 days
# 1  
Old 01-13-2014
Remove lines older than 30 days

Hi Experts/Gurus,

Is there a way to remove lines in a file that are older than x days (i.e. 30 days) based on the date stamp in the first column?

Example.

Code:
$ date
Sat Jan 11 14:12:06 EDT 2014
$cat sample.txt
10-10-2013 09:00:01 AM|Line test 1234567
16-10-2013 08:30:00 AM|Line test 2345567
25-10-2013 07:30:00 AM|Line test 3456678
26-10-2013 06:00:00 AM|Line test 4567890
28-10-2013 07:30:00 AM|Line test 5678910
15-12-2013 10:00:00 PM|Line test 7891234
10-01-2014 03:00:00 PM|Line test 8901234

expected output:
Code:
$cat sample1.txt
15-12-2013 10:00:00 PM|Line test 7891234
10-01-2014 03:00:00 PM|Line test 8901234

Thank you in advance!
# 2  
Old 01-13-2014
What is your OS? Post your system information:
Code:
uname -a

# 3  
Old 01-13-2014
hi yoda - OS is aix 6.1
# 4  
Old 01-13-2014
I wrote and tested this on a AIX 6.1 box and it works, however you need to have Perl installed. Hope this helps.

Code:
#!/usr/bin/ksh
#
# -- script to remove lines from a file
# -- older than the given number of days
# -- based on the date in column 1
#

# check command line for given number of days
if [ $# -ne 2 ]
then
    echo "Usage: $0 </path/to/file> <number of days>"
    exit 1
fi

# store file and number of days
f=$1
d=$2

# get the current date in seconds past epoch
c=$(date +%s)

# new file
n="$f.new"

# read file line by line, convert the date in
# column one to seconds pase epoch and check if
# older than the given number of days
while read line
do
    c1=$(echo $line | awk '{print $1}')
    f1=$(echo "$c1" | awk '{gsub(/-/," ",$0)}{print $3" "$2" "$1}')
    secs=$(perl -e 'use Time::Local; print timelocal(0,0,0,$ARGV[2], $ARGV[1]-1, $ARGV[0]);' $f1)
    dif=$(echo "scale=0; ($c - $secs) / (24*3600)" | bc)
    if [ $dif -le $d ]
    then
        echo $line >> $n
    fi
done < $f

# remove backup original file and rename new file
mv $f $f.ORIG
mv $n $f
cat $f

# done
exit 0

./remoldlines.ksh
Usage: ./remoldlines.ksh </path/to/file> <number of days>

./remOldLines.ksh /tmp/file.txt 30
15-12-2013 10:00:00 PM|Line test 7891234
10-01-2014 03:00:00 PM|Line test 8901234

da3fb00cca3575cbce2516f73a486294
# 5  
Old 01-13-2014
Ok, take a look at this thread: Date Arithmetic in FAQ section.
# 6  
Old 01-15-2014
Calling perl and bc once and awk twice for every line of the datafile may be quite slow especially with large files.

Most of the processing can be done within perl like this:

Code:
$ cat remoldlines.pl
#!/usr/bin/env perl
use Time::Local;

while (my $ln = <STDIN>) {
    my ($d, $mt, $y, $h, $m, $s, $pm) = split(/[ :-]/, $ln);
    $h += 12 unless ($pm != "PM");
    my $age = time() - timelocal($s,$m,$h,$d,$mt-1,$y);
    print $ln unless ($age > int($ARGV[0])*3600*24);
}

$ remoldlines.pl 30 < sample.txt
20-12-2013 10:00:00 PM|Line test 7891234
15-01-2014 03:00:00 PM|Line test 8901234

# 7  
Old 01-16-2014
A python:
Code:
#!/usr/bin/env python

import sys
from time import mktime,strptime 
from datetime import datetime

now  = datetime.now()
with open('sample.txt','rb') as f:
    for line in f:
        L = line.split('|')
        before = datetime.fromtimestamp(
                 mktime(strptime(L[0],'%d-%m-%Y %I:%M:%S %p')))
        old = now - before
        if  old.days < 30:
            sys.stdout.write(line)

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Filtering log file with lines older than 10 days.

Hi, I am trying to compare epoch time in a huge log file (2 million lines) with todays date. I have to create two files one which has lines older than 10 days and another file with less than 10 days. I am using while do but it takes forever to complete the script. It would be helpful if you can... (12 Replies)
Discussion started by: shunya
12 Replies

2. Shell Programming and Scripting

Shellscript command to remove files starting with a certain string, and older than 3 days

Hi All, Need help in identifying a shellscript command to remove all files on a server directory, starting with a certain prefix and also older than 3 days. That means files created with that prefix, today or yesterday, shouldn't be removed. Thanks, Dev (3 Replies)
Discussion started by: dev.devil.1983
3 Replies

3. Shell Programming and Scripting

Search 2 days older file and then delete last 10 lines

I want to search 2 day older file and then delete last 10 line of that file. (2 Replies)
Discussion started by: sonu pandey
2 Replies

4. Shell Programming and Scripting

Remove files older than 2 days.

Hi All, I am new to the scripting and using solaris 10 OS. Please suggest me from the below script which modifications need to be done to delete the files more that 2days older. Current script is deleting existing file. # Remove old explorer runs if needed DIR=`dirname ${EXP_TARGET}` if ... (2 Replies)
Discussion started by: Navkreddy
2 Replies

5. Shell Programming and Scripting

[Solved] Remove file older than 90 days

I have crontab job a tar file to a directory ( tar -cvf /tmp/backup/or.`date +%m%d%y`. /ora/db/* ) , it will do it every day . Now I don't want to keep too much files , I just want to keep the file for 90 days , can advise if I want to remove the backup file which are elder than 90 days , can... (1 Reply)
Discussion started by: ust3
1 Replies

6. UNIX for Advanced & Expert Users

How to Remove 180 days older non-empty directories

Hi Gurus, I have command to delete more than 180days file. find /home/abc/ -name "CBST_*.txt*" -mtime +180 | xargs -n 100 rm -f Now I would like to delete more than 180days Non empty directory--What will be command? Following is non empty directory as instance CBST2010* (2 Replies)
Discussion started by: thepurple
2 Replies

7. Shell Programming and Scripting

script to remove files older than 60 days

Hi I need help in the script which looks at a contorl file which has a list of file names like xxxx.12345 and I want to take only xxxxx and search in a specific directory and remove the file if its older than 60 days I have written something like this.. but seems to be wrong... (1 Reply)
Discussion started by: antointoronto
1 Replies

8. Shell Programming and Scripting

How to tar, compress and remove files older than two days

Hi, I'm Eddy from Belgium and I've the following problem. I try to write a ksh script in AIX to tar, compress and remove the original *.wav files from the directory belgacom_sf_messages older than two days with the following commands. The problem is that I do not find a good combination... (4 Replies)
Discussion started by: edr
4 Replies

9. Shell Programming and Scripting

Need to remove files older than 30 days except directories

Hi, I need to remove files (*.trc) which are older than 30 days from one location. My problem is there I do not want to visit any of the directories at that location. I want to search files at that particular location only (need to skip directorys at that location). maxdepth option is there... (6 Replies)
Discussion started by: malaymaru
6 Replies

10. UNIX for Dummies Questions & Answers

Find files older than 5 days and remove tem after listing

need help with this ... Find files older than 5 days and remove tem after listing list "test" file older than 5 days and then remove them (1 Reply)
Discussion started by: ypatel6871
1 Replies
Login or Register to Ask a Question