Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Remove lines having older end time

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Remove lines having older end time
# 1  
Old 09-29-2017
Remove lines having older end time
Hi,

In my bash script I want to add a code which remove all entries older than x days.

To simplify this problem, I have divided into 3 parts. (2 parts are done looking answer for 3rd part.)
  1. To find the latest log date - Done
    Latest_Time=`find . -name '*.tps' -exec sed '/endTime/!d; s/{//; s/,.*//' {} + | sort -r | head -1 | cut -d: -f2` #latest epoch time
  2. evaluate earliest epoch time. (All entries before this epoch time should be deleted) - Done
    Earliest_Time=`Latest_Time-(86400000*$No_OF_DAYS)` #earliest epoch time
  3. delete all log entries older than evaluated earliest time.

PS:
  1. there are multiple files and distributed in different sub folders.
  2. All files having extension as ".tps".
  3. time is in epoch format. endTime will be considered for calculations.("endTime":1488902735220)
  4. sample data

Code:
{"endTime":1488902734775,"startTime":1488902734775,"operationIdentity":"publishCacheStatistics","name":"murex.risk.control.excesses.cache.CacheStatisticsTracer","context":{"parentContext":{"id":-1,"parentContext":null},"data":[{"value":"excessCacheExcessKeysToContexts","key":"name"},{"value":"0","key":"hits"},{"value":"0","key":"misses"},{"value":"0","key":"count"},{"value":"0","key":"maxElements"},{"value":"0","key":"evictions"},{"value":"N/A","key":"policy"}],"id":0}}
{"endTime":1488902735220,"startTime":1488902735220,"operationIdentity":"publishCacheStatistics","name":"murex.risk.control.excesses.cache.CacheStatisticsTracer","context":{"parentContext":{"id":-1,"parentContext":null},"data":[{"value":"excessCacheExcessKeysToContexts","key":"name"},{"value":"0","key":"hits"},{"value":"0","key":"misses"},{"value":"0","key":"count"},{"value":"0","key":"maxElements"},{"value":"0","key":"evictions"},{"value":"N/A","key":"policy"}],"id":8}}
{"endTime":1488902735550,"startTime":1488902735550,"operationIdentity":"publishCacheStatistics","name":"murex.risk.control.excesses.cache.CacheStatisticsTracer","context":{"parentContext":{"id":-1,"parentContext":null},"data":[{"value":"excessCacheContextsToExcessIds","key":"name"},{"value":"0","key":"hits"},{"value":"0","key":"misses"},{"value":"0","key":"count"},{"value":"0","key":"maxElements"},{"value":"0","key":"evictions"},{"value":"N/A","key":"policy"}],"id":9}}


For Example:
  1. latest epoch time = 1488902735550
  2. earliest epoch time = 1488902735220

Problem: Now I am looking for command which delete all the entries which is older/lesses than earliest epoch time. In above example 1st line should be deleted.

Any help/suggestions are appreciated. Thank you Smilie
Last edited by rbatte1; 09-29-2017 at 10:15 AM.. Reason: Set up formatted lists and added ICODE tags where appropriate
# 2  
Old 09-29-2017
You're not telling us how the "earliest epoch time" will be transported - in a file? A shell variable? Inside awk (etc.) already?
# 3  
Old 09-29-2017
Try
Code:
awk -F, -vET="1488902735220" 'substr($1, 12, 13) >= ET' file

# 4  
Old 09-29-2017
Code:
Latest_Time=`find . -name '*.tps' -exec sed '/endTime/!d; s/{//; s/,.*//' {} + | sort -r | head -1 | cut -d: -f2`  #latest epoch time
Earliest_Time=`Latest_Time-(86400000*$No_OF_DAYS)` #earliest epoch time

Last edited by Agoyals1986; 09-29-2017 at 07:54 AM.. Reason: Added CODE tags.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove lines older than 30 days

Hi Experts/Gurus, Is there a way to remove lines in a file that are older than x days (i.e. 30 days) based on the date stamp in the first column? Example. $ date Sat Jan 11 14:12:06 EDT 2014 $cat sample.txt 10-10-2013 09:00:01 AM|Line test 1234567 16-10-2013 08:30:00 AM|Line test... (6 Replies)
Discussion started by: brichigo
6 Replies

2. Shell Programming and Scripting

Remove lines between the start string and end string including start and end string Python

Hi, I am trying to remove lines once a string is found till another string is found including the start string and end string. I want to basically grab all the lines starting with color (closing bracket). PS: The line after the closing bracket for color could be anything (currently 'more').... (1 Reply)
Discussion started by: Dabheeruz
1 Replies

3. Shell Programming and Scripting

Delete lines containing and remove the blank line at the same time

Is there a way to delete a line containing something and the blank line at the same time? If you do this it leaves a blank line behind. sed '/yum/d' .bash_historyI know this works but I would think there would be a way to do it with one command sed '/yum/d' .bash_history | sed '/^$/d'In... (2 Replies)
Discussion started by: cokedude
2 Replies

4. Shell Programming and Scripting

Remove lines that match string at end of column

I have this: 301205 0000030000041.49000000.00 2011111815505 908 301205 0000020000029.10000000.00 2011111815505 962 301205 0000010000027.56000000.00 2011111815505 3083 312291 ... (2 Replies)
Discussion started by: herot
2 Replies

5. Shell Programming and Scripting

shell script to remove all lines which exceeds a particular date & time

I have a text file which got 6th coloumn as date and 7th coloumn as time. The text contains data for last one week. I need to remove all the data whose date & time is after 03/08/2011 06:00:00 and save it on another file TEXT FILE ======== 6 dbclstr-b IXT_Web Memphis_Prod_SQL_Diff... (4 Replies)
Discussion started by: ajiwww
4 Replies

6. Shell Programming and Scripting

How to calculate time difference between start and end time of a process!

Hello All, I have a problem calculating the time difference between start and end timings...! the timings are given by 24hr format.. Start Date : 08/05/10 12:55 End Date : 08/09/10 06:50 above values are in mm/dd/yy hh:mm format. Now the thing is, 7th(08/07/10) and... (16 Replies)
Discussion started by: smarty86
16 Replies

7. Shell Programming and Scripting

Remove lines of the same time stamp leaving the highest

Hi guys, I have a log that looks like that below. Columns 2 3 4 5 6 7 is the date/time stamp separated by comma. UUU,02,06,2010,10,00,00,00,0000000000000000,0000000000000000,0000000000001224 UUU,02,06,2010,10,05,00,00,0000000000000000,0000000000000000,0000000000001502... (2 Replies)
Discussion started by: borderblaster
2 Replies

8. Shell Programming and Scripting

Remove lines, Sorted with Time based columns using AWK & SORT

Hi having a file as follows MediaErr.log 84 Server1 Policy1 Schedule1 master1 05/08/2008 02:12:16 84 Server1 Policy1 Schedule1 master1 05/08/2008 02:22:47 84 Server1 Policy1 Schedule1 master1 05/08/2008 03:41:26 84 Server1 Policy1 ... (1 Reply)
Discussion started by: karthikn7974
1 Replies

9. Shell Programming and Scripting

Start time/end time and status of crontab job

Is there anyway to get the start time and end time / status of a crontab job which was just completed? Of course, we know the start time of the crontab job since we are scheduling. But I would like to know process start and time recorded somewhere or can be fetched from a command like 'ps'. ... (3 Replies)
Discussion started by: thambi
3 Replies

10. Shell Programming and Scripting

How to remove plank spaces at the end of lines

Hello friends, I want to remove blank spaces at the end of lines. I use sed command to do this but it is not working correctly. sed ‘s/ $//’ file_name Can some body tell me what is the proper way to remove blank spaces at the end of a limes. Thanks, Mahesh Fernando. (3 Replies)
Discussion started by: maheshsri
3 Replies
Login or Register to Ask a Question