Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

help with sed | awk

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting help with sed | awk
# 1  
Old 05-05-2010
Lightbulb help with sed | awk
I have this log file that I need to reduce the size by deleting the lines from the log file if the date at the beginning of the line is older than 30 days from the current system date.

Code:
01/30/2010 at 08:00:08 Successful call to script 
01/30/2010 at 08:15:06 Processing is terminated
01/30/2010 at 08:30:01 Successful MQGET call 
01/30/2010 at 08:30:07 Processing is terminated
04/30/2010 at 08:00:08 Successful call to script 
04/30/2010 at 08:15:06 Processing is terminated
04/30/2010 at 08:30:01 Successful MQGET call 
04/30/2010 at 08:30:07 Processing is terminated

I intend to use sed but I'm getting the below error:
Code:
sed -n '/^[0-9][0-9]\/[0-9][0-9]\/[0-9][0-9][0-9][0-9]/ p' log |  awk '($1); < "05/04/2010" { print }'
 Syntax Error The source line is 1.
 The error context is
                ($1); >>>  < <<<  "05/04/2010" { print }
 awk: 0602-500 Quitting The source line is 1.

Please note that "05/04/2010" can change and I plan to use a variable in its place.


Thank you in advance!
Last edited by udelalv; 05-05-2010 at 05:54 AM..
# 2  
Old 05-05-2010
I suppose the file is already sorted implicitly, because messages are stored sequentially in a time-wise order. This makes your task easier, because you can simply search for the first entry of a given date and delete all lines from there (or up to this line, depending on the file being sorted ascendingly or descendingly).

Suppose your date to search for would be stored in "$ThresholdDate" your sed-command could look like:

Code:
sed '/^'"$ThresholdDate"'/q' /path/to/infile > /path/to/outfile

This would print all lines up to (but not including) the first line containing "$TresholdDate" at the line start and then stop.

Two problems: you have to calculate the threshold-date and store it to "$ThresholdDate" and second, you have to escape all the characters with special meanings to sed which you use in your variable, especially the slashes ("/").

To calculate the correct date store the current date in several variables for year, month and day, then subtract 30 days and reassemble that to the result. Depending on how exact your result has to be you could simplify this by ignoring odd years, make every month 30 days long, etc. In either case it will be simply a matter of some additions and subtractions.

You could, of course, also use "datecalc" written by our venerable brother perderabo.

To escape the slashes add "\/" instead of the regular "/" into the reassembled date:

Code:
ThresholdDate="${month}\/${day}\/${year}"

I hope this helps.

bakunin
Last edited by bakunin; 05-05-2010 at 06:04 AM..
# 3  
Old 05-05-2010
Hello,

Or even in the file is not sorted, this could help :

Code:
cat <file> | awk '{ if ($1 > "<date>")  print }'

Regs,
Last edited by zaxxon; 05-05-2010 at 06:25 AM.. Reason: use code tags please, ty
# 4  
Old 05-05-2010
The comparison of dates in the format mm/dd/yyyy has no sense, they must be in the form of yyyymmdd.

Try and adapt this little script (requires datecalc script by Perderabo :
Code:
Days=30
Before=$(printf "%4d%02d%02d" $(datecalc -a $(date +'%Y %m %d') - ${Days}))
awk -F '[/ ]*' -v Before=${Before} '
{
   date = $3 $1 $2;
   if (date >= Before) print
}
' log

Output from your sample file :
Code:
04/30/2010 at 08:00:08 Successful call to script 
04/30/2010 at 08:15:06 Processing is terminated
04/30/2010 at 08:30:01 Successful MQGET call 
04/30/2010 at 08:30:07 Processing is terminated

Jean-Pierre.
# 5  
Old 05-06-2010
thanks everyone for the suggestion.
I used Jean-Pierre's code and adapted the dateCalc script by Pederabo.

I have another question though. If my logfile contains a line of "*****" in between, how do I get the "*****" printed?
Code:
01/30/2010 at 08:00:08 Successful call to script 
01/30/2010 at 08:15:06 Processing is terminated
*************************************
01/30/2010 at 08:30:01 Successful MQGET call 
01/30/2010 at 08:30:07 Processing is terminated
*************************************
04/30/2010 at 08:00:08 Successful call to script 
04/30/2010 at 08:15:06 Processing is terminated
*************************************
04/30/2010 at 08:30:01 Successful MQGET call 
04/30/2010 at 08:30:07 Processing is terminated
*************************************

Output file should look like this?
Code:
04/30/2010 at 08:00:08 Successful call to script 
04/30/2010 at 08:15:06 Processing is terminated
*************************************
04/30/2010 at 08:30:01 Successful MQGET call 
04/30/2010 at 08:30:07 Processing is terminated

thanks again.
# 6  
Old 05-06-2010
Try this new version of the script :
Code:
Days=30
Before=$(printf "%4d%02d%02d" $(datecalc -a $(date +'%Y %m %d') - ${Days}))
awk -F '[/ ]*' -v Before=${Before} '
{
   date = $3 $1 $2;
   if (date >= Before) {
      print;
      printSepLine = 1;
   }
} 
/^\*+/ && printSepLine {
   print;
   printSepLine = 0;
}
' log

log file :
Code:
01/30/2010 at 08:00:08 Successful call to script 
01/30/2010 at 08:15:06 Processing is terminated
************************************************
01/30/2010 at 08:30:01 Successful MQGET call 
01/30/2010 at 08:30:07 Processing is terminated
************************************************
04/30/2010 at 08:00:08 Successful call to script 
04/30/2010 at 08:15:06 Processing is terminated
************************************************
04/30/2010 at 08:30:01 Successful MQGET call 
04/30/2010 at 08:30:07 Processing is terminated

Output :
Code:
04/30/2010 at 08:00:08 Successful call to script 
04/30/2010 at 08:15:06 Processing is terminated
************************************************
04/30/2010 at 08:30:01 Successful MQGET call 
04/30/2010 at 08:30:07 Processing is terminated

Jean-Pierre.
# 7  
Old 05-06-2010
sorry... if log file contains this:

Code:
On 01/30/2010 at 08:00:08 Successful call to script 
On 01/30/2010 at 08:15:06 Processing is terminated
************************************************
On 01/30/2010 at 08:30:01 Successful MQGET call 
On 01/30/2010 at 08:30:07 Processing is terminated
************************************************
On 04/30/2010 at 08:00:08 Successful call to script 
On 04/30/2010 at 08:15:06 Processing is terminated
************************************************
On 04/30/2010 at 08:30:01 Successful MQGET call 
On 04/30/2010 at 08:30:07 Processing is terminated

How do I handle "On" to get this output:
Code:
On 04/30/2010 at 08:00:08 Successful call to script 
On 04/30/2010 at 08:15:06 Processing is terminated
************************************************
On 04/30/2010 at 08:30:01 Successful MQGET call 
On 04/30/2010 at 08:30:07 Processing is terminated
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

sed and awk giving error ./sample.sh: line 13: sed: command not found

Hi, I am running a script sample.sh in bash environment .In the script i am using sed and awk commands which when executed individually from terminal they are getting executed normally but when i give these sed and awk commands in the script it is giving the below errors :- ./sample.sh: line... (12 Replies)
Discussion started by: satishmallidi
12 Replies

2. Shell Programming and Scripting

Is this possible using SED and AWK?

Dear Geeks, I want to manipulate a file with certain modifications for that using sed or AWK how to do this process for one file i have this type of data. Input File: "Restricted and Reserved names .ANISH",3798,"TEST.CO",1201208,6/16/10 0:00,6/16/13 0:00,,,"CO","2nd"^M "Restricted and... (4 Replies)
Discussion started by: anishkumarv
4 Replies

3. UNIX for Dummies Questions & Answers

sed/awk or help please

I have a file that contain the data below: B1 1 2 3 B2 20 30 40 B3 7 8 B4 100 B5 21 22 23How can I retrieve the data for B1 into a seperate file. (8 Replies)
Discussion started by: bobo
8 Replies

4. Shell Programming and Scripting

Need help using awk or sed.

Hi All, Is there a way of comparing two columns in the same file and deleting the row if the values of the columns match. I have the sample data file as below. M024900|175309.00|968.00|17 M025001|19861.79|97.90|148 M025002|431.70|159.00|3 M025003|912.30|159.90|6 ... (6 Replies)
Discussion started by: nua7
6 Replies

5. Shell Programming and Scripting

Using sed or awk?

What if I wanted to add a word such as IT after the first character and if theres 3 characters, after the 2nd character? output would be: G, it H G, H it P G, H, P it L I'm thinking that AWK would be the easiest way to do this... Currently looking it up. Right now I'm using awk but I... (13 Replies)
Discussion started by: puttster
13 Replies

6. UNIX for Dummies Questions & Answers

sed or awk?

I've got an inventory database with eight columns with things like product name, manufacturer, UPC code, etc. on each line. Our PO (purchase order) number is in the first column. I can grep the date and get the full line of data but I would like to strip out everything but the PO number in the... (5 Replies)
Discussion started by: NetJones
5 Replies

7. UNIX for Advanced & Expert Users

Awk or Sed help

Hi, I have a data file with 5 columns - like this: "20080401 09:43:08.770798 +0100s","TEST 1","R 1","A TEST","Nov 27 2007","1" "20080401 09:43:08.770798 +0100s","THIS IS A TEST","R 2","B TEST","Nov 30 2007","10" "20080401 09:43:08.770798 +0100s","ANOTHER TEST","R 3","B TEST","Nov 05... (7 Replies)
Discussion started by: MrG-San
7 Replies

8. UNIX for Advanced & Expert Users

sed in awk ? or nested awk ?

Hey all, Can I put sed command inside the awk action ?? If not then can i do grep in the awk action ?? For ex: awk '$1=="174" { ppid=($2) ; sed -n '/$ppid/p' tempfind.txt ; }' tempfind.txt Assume: 174 is string. Assume: tempfind.txt is used for awk and sed both. tempfind.txt... (11 Replies)
Discussion started by: varungupta
11 Replies

9. Shell Programming and Scripting

sed,awk

Hi, I know sed is stream text editor and not a bit more than that. Can anyone explain its usage and advantages? How is awk different from sed? I donno i am a bit confused about it. But i have coded in awk and shell. Thanks, Nisha :confused: (7 Replies)
Discussion started by: Nisha
7 Replies

10. Shell Programming and Scripting

awk / sed

I have many messages such as the test message below: 00:00000:00021:2002/05/13 13:57:00.51 ERROR:- Test error, my test error!!! I am writing a script in which I need to get everything from the word "ERROR:-" onwards. I normally use awk for these things, but I am not an expert at it so i am... (6 Replies)
Discussion started by: baileyr1
6 Replies
Login or Register to Ask a Question