removing rows from text file older than certain date


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting removing rows from text file older than certain date
# 1  
Old 02-08-2012
removing rows from text file older than certain date

Hi I need a way of removing rows from a txt file that are older than 30 days from today, going by the date in column 2, below is an example from my file. I have tried awk but don't have enough knowledge. I would really appreciate some help.
Code:
41982,15/07/2010,H833AB/0,JZ,288
41983,15/08/2010,H822AC/0,JZ,480
41984,15/08/2010,H825AC/0,JZ,480
41985,15/08/2010,H844AA/0,JZ,480
41986,15/08/2010,H887AB/0,JZ,200
41987,15/01/2012,K826AC/0,JZ,480
41988,15/01/2012,K971AF/0,JZ,100
41989,01/01/2012,Y985AE/0,JZ,100
41990,15/12/2011,Y943AB/0,JZ,80
41991,15/12/2011,Y973AB/0,JZ,100
41992,15/12/2011,Y957AB/0,JZ,150


Moderator's Comments:
Mod Comment Please use next time code tags for your code and data

Last edited by vbe; 02-08-2012 at 11:20 AM..
# 2  
Old 02-08-2012
Code:
#! /bin/bash

while IFS=',' read no dt rest
do
    row_dt=`echo $dt | awk -F'/' '{print $3$2$1}'`
    ref_dt=`date -d"-30 days" +%Y%m%d`
    [ $row_dt -ge $ref_dt ] && echo "$no,$dt,$rest"
done < inputfile

Code:
$ ./test.sh
41987,15/01/2012,K826AC/0,JZ,480
41988,15/01/2012,K971AF/0,JZ,100

# 3  
Old 02-08-2012
Try this...
Code:
awk -F, '{split($2,a,"/");sec=mktime(a[3]" "a[2]" "a[1]" 00 00 00");diff=(systime()-sec)/(60*60*24)} diff<=30' infile

--ahamed
# 4  
Old 02-08-2012
Hi balajesuri,

Thank you for the help, your code works on smaller files but when trying it on one of the bigger files I get the following error.

Code:
./test.sh: line 7: [: too many arguments

Any ideas?

Thanks again

---------- Post updated at 11:30 AM ---------- Previous update was at 11:28 AM ----------

Quote:
Originally Posted by ahamed101
Try this...
Code:
awk -F, '{split($2,a,"/");sec=mktime(a[3]" "a[2]" "a[1]" 00 00 00");diff=(systime()-sec)/(60*60*24)} diff<=30' infile

--ahamed

Thanks for the help but I get the following errors:
Code:
awk: line 2: function systime never defined
awk: line 2: function mktime never defined

# 5  
Old 02-08-2012
which is your os? if solaris, use nawk

--ahamed
# 6  
Old 02-08-2012
Quote:
Originally Posted by ahamed101
which is your os? if solaris, use nawk

--ahamed
Hi ahamed,

I am using Ubuntu server.
# 7  
Old 02-08-2012
Check if you have gawk

--ahamed

---------- Post updated at 08:58 AM ---------- Previous update was at 08:41 AM ----------

Try this...
Code:
#!/bin/bash

ref=$( date '+%s' )
while read line
do
  val=${line#*,}; val=${val%%,*}
  year=${val:6:4}; month=${val:3:2}; day=${val:0:2}
  sec=$( date -d"$year/$month/$day" +%s )
  ((diff=(ref-sec)/(60*60*24)))
  test "$diff" -le 30 && echo $line
done < infile

--ahamed
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Removing top few and bottom few rows in a file

Hi, I have a requirement where I need to delete given number of top and bottom rows in a flat file which has new line as its delimiter. For ex: if top_rows=2 & bottom_rows=1 Then in a given file which looks like: New York DC LA London Tokyo Prague Paris Bombay Sydney... (7 Replies)
Discussion started by: calredd
7 Replies

2. Shell Programming and Scripting

Removing Duplicate Rows in a file

Hello I have a file with contents like this... Part1 Field2 Field3 Field4 (line1) Part2 Field2 Field3 Field4 (line2) Part3 Field2 Field3 Field4 (line3) Part1 Field2 Field3 Field4 (line4) Part4 Field2 Field3 Field4 (line5) Part5 Field2 Field3 Field4 (line6) Part2 Field2 Field3 Field4... (7 Replies)
Discussion started by: ekbaazigar
7 Replies

3. Shell Programming and Scripting

Bash script help - removing certain rows from .csv file

Hello Everyone, I am trying to find a way to take a .csv file with 7 columns and a ton of rows (over 600,000) and remove the entire row if the cell in forth column is blank. Just to give you a little background on why I am doing this (just in case there is an easier way), I am pulling... (3 Replies)
Discussion started by: MrTuxor
3 Replies

4. Shell Programming and Scripting

Delete log files content older than 30 days and append the lastest date log file date

To delete log files content older than 30 days and append the lastest date log file date in the respective logs I want to write a shell script that deletes all log files content older than 30 days and append the lastest log file date in the respective logs This is my script cd... (2 Replies)
Discussion started by: sreekumarhari
2 Replies

5. Shell Programming and Scripting

Removing rows and chars from text file

Dear community, maybe I'm asking the moon :rolleyes:, but I'm scratching my head to find a solution for it. :wall: I have a file called query.out (coming from Oracle query), the file is like this: ADDR TOTAL -------------------- ---------- TGROUPAGGR... (16 Replies)
Discussion started by: Lord Spectre
16 Replies

6. Shell Programming and Scripting

Move file older date

hlow all i have folder with name like this 20110512 20110601 20110602 so i want move the last 1 day to other directory for example this date 20110602 so i want move folder older 1 day from this date 20110512 -----> this folder will move to other dir 20110601-----> this folder will move... (2 Replies)
Discussion started by: zvtral
2 Replies

7. UNIX for Dummies Questions & Answers

Removing duplicate rows & selecting only latest date

Gurus, From a file I need to remove duplicate rows based on the first column data but also we need to consider a date column where we need to keep the latest date (13th column). Ex: Input File: Output File: I know how to take out the duplicates but I couldn't figure out... (5 Replies)
Discussion started by: shash
5 Replies

8. Shell Programming and Scripting

Removing rows based on a another file

Hi, I am not sure if this has already been asked (I tried the search but the search was too broad). Basically I want to remove rows based on another file. So file1 looks like this (tab seperated): HHN 3 5 5 HUJ 2 2 1 JJJ 3 1 1 JUN 2 1 3 I have another file (file2)... (2 Replies)
Discussion started by: kylle345
2 Replies

9. Shell Programming and Scripting

Removing rows from a file

I have a file like below and want to use awk to solve this problem. The record separator is ">". I want to look at each record section enclosed within ">". Find the row with the 2nd and 3rd columns being 0, such as 10 0 0 I need to take the first number which in this case is 10. Then... (15 Replies)
Discussion started by: kristinu
15 Replies

10. Shell Programming and Scripting

Removing rows from a file based on date comparison

I have a '|' delimited file and want to remove all the records from the file if the date is greater than a year from sysdate. The layout of the file is as below - xxxxxxxxxxxxxx|yyyyyy|zzzzzz|2009-12-27-00:00| 000000000|N xxxxxxxxxxxxxx|yyyyyy|zzzzzz|2010-01-03-00:00| 000000000|N... (4 Replies)
Discussion started by: Max_2503
4 Replies
Login or Register to Ask a Question