Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Extract based on timestamp

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Extract based on timestamp
# 1  
Old 12-23-2015
Extract based on timestamp
Hi,
I want take the files whose timestamp greater than "2015-12-23 22:30" using awk .Please advise
Code:
os version :Linux ip-172-31-19-240 4.1.10-17.31.amzn1.x86_64 #1 SMP Sat Oct 24 01:31:37 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

2015-12-23 20:43:17 312116 file1
2015-12-23 21:44:46 314375 file11
2015-12-23 22:46:40 308663 file12
2015-12-23 23:42:03 449440 file13
2015-12-23 23:44:19 441490 file14
2015-12-23 23:45:54 442201 file15

# 2  
Old 12-23-2015
Assuming your data is in a file named list:
Code:
awk -v start='2015-12-23 22:30' '$1 " " $2 > start { print $NF }' list

seems to do what you want (with your sample input, producing the output):
Code:
file12
file13
file14
file15

If someone else wants to try this on a Solaris/SunOS system, change awk to /usr/xpg4/bin/awk or nawk.
This User Gave Thanks to Don Cragun For This Post:
TechGyaann
# 3  
Old 12-24-2015
Hi, mohan705

I am curious. Could you, please, tell me how did you get a list like this?
Code:
cat mohan705.list
2015-12-23 20:43:17 312116 file1
2015-12-23 21:44:46 314375 file11
2015-12-23 22:46:40 308663 file12
2015-12-23 23:42:03 449440 file13
2015-12-23 23:44:19 441490 file14
2015-12-23 23:45:54 442201 file15

Using Perl, if you want to show only the file names.
Code:
perl -nlae 'print $F[-1] if $_ gt "2015-12-23 22:30"' mohan705.list

In case you want the whole line.
Code:
perl -ne 'print if $_ gt "2015-12-23 22:30"' mohan705.list

Code:
2015-12-23 22:46:40 308663 file12
2015-12-23 23:42:03 449440 file13
2015-12-23 23:44:19 441490 file14
2015-12-23 23:45:54 442201 file15

If your posted example is not a real list but a loose interpretation of a directory content, let us know.
Last edited by Aia; 12-24-2015 at 12:44 AM.. Reason: Changes the for your
# 4  
Old 12-24-2015
Hi Aia,
The files list getting in S3 AWS

Thanks,
Mohan

---------- Post updated at 03:43 AM ---------- Previous update was at 03:42 AM ----------

Thanks don ,its workingSmilie
# 5  
Old 12-26-2015
Hi Don

When passing parameter as Date its not working intact it displaying every thing ,but when hard coding the values its working .Please advise
Code:

DATE2=`date --date='1 day ago' +%Y-%m-%d`

date --date='1 day ago' -u -d  "$(date --date='1 day ago' -u "+%a %b %e 00:00:00 %Z %Y")" +%s000 >unixtime

x=`eval cat unixtime`

echo "processing of $DATE1 files with Unix timestamp $x " >$logdir/segment_$DATE1_inc.log
 
folder="s3://xx-logs/Ja/$x"
echo "yes:$folder"
echo "Source Path of S3 bucket $folder">>$logdir/segment_$DATE1_inc.log
 
chk=`eval echo $DATE2`
echo "this is : $chk 20:30"
$chk

aws s3 ls s3://xx-logs/Ja/1451001600000/|awk -v start='$(chk) 20:30' '$1" "$2 >start {print $0}'
#aws s3 ls s3://xx-logs/Ja/1451001600000/|awk -v start='2015-12-25 20:30' '$1" "$2 >start {print $0}'

Thanks,
Mohan
# 6  
Old 12-26-2015
Maybe just
Code:
start=$(date --date="1 day ago" "+%Y-%m-%d 20:30")

And then
Code:
ws s3 ls s3://xx-logs/Ja/1451001600000/|awk -v start="$start" '$1" "$2 >start {print $0}'

# 7  
Old 12-26-2015
Your immediate problems include:
  1. shell variables are not expanded when they appear between single quotes,
  2. chk is a variable name (not a command name),
  3. and the contents of that variable are a string of the form YYYY-MM-DD, not the name of a command to be executed.

Try changing:
Code:
aws s3 ls s3://xx-logs/Ja/1451001600000/|awk -v start='$(chk) 20:30' '$1" "$2 >start {print $0}'

to:
Code:
aws s3 ls s3://xx-logs/Ja/1451001600000/|awk -v start="${chk} 20:30" '$1" "$2 >start {print $0}'

You seem to like to use several levels of indirection instead of doing things directly. Do you need the file unixtime after this script ends? Or are you just using it as temporary storage for the output of a date command?

You might want to compare your script to the following (although it is completely untested):
Code:
DATE2=$(date --date='1 day ago' +%Y-%m-%d)

x=$(date --date='1 day ago' -u -d  "$(date --date='1 day ago' -u "+%a %b %e 00:00:00 %Z %Y")" +%s000)

# Uncomment the next line if the file "unixtime" will be needed later.
# echo "$x" > unixtime

echo "processing of $DATE1 files with UNIX timestamp $x" > "$logdir/segment_$DATE1_inc.log"
 
folder="s3://xx-logs/Ja/$x"
echo "yes:$folder"
echo "Source Path of S3 bucket $folder" >> "$logdir/segment_$DATE1_inc.log"
 
chk="$DATE2 20:30"
echo "this is : $chk"
# $chk	# This line commented out because I can't believe a command of the form:
	#	2015-12-26
	# ever worked.

# Note that in the following command we are expanding the variable chk; not
# executing the command chk (and it is in double quotes; not single quotes).
aws s3 ls s3://xx-logs/Ja/1451001600000/|awk -v start="$chk" '$1" "$2 > start'

Note, of course, that this script uses, but never initializes, the variables logdir and DATE1???
Last edited by Don Cragun; 12-27-2015 at 11:32 PM.. Reason: Fix typo: s/is is/is in/
This User Gave Thanks to Don Cragun For This Post:
TechGyaann
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Script to keep todays files based on Timestamp

Hi i need to keep todays files based on timestamp and archive the remaining files ex: Managerial_Country_PRD_20150907.csv Managerial_Country_PRD_20150907.csv Managerial_Country_PRD_20150906.csv Managerial_Country_PRD_20150905.csv (6 Replies)
Discussion started by: ram1228
6 Replies

2. UNIX for Dummies Questions & Answers

Display files based on particular file timestamp

Hi, I have requirement to list out files that are created after particular file. ex. I have below files in my directory. I want to display files created after /dirdat/CG1/cg004440 file. ./dirdat/CG1/cg004438 09/07/14 0:44:05 ./dirdat/CG1/cg004439 09/07/14 6:01:48 ... (3 Replies)
Discussion started by: tmalik79
3 Replies

3. Shell Programming and Scripting

Cd to Folder based on TimeStamp

Hi, ls -ltr -rw-rw---- 1 user1 admins 5000032 Jan 20 17:11 M1120252_P004640.csv Now i wish to cd to that folder amongst the below folders that would have log files for the date of the .csv file i.e. 20 Jan 17:11 ls -ltr total 53616 drwxrwx--- 2 user1 admins 20840448 Jan 19... (4 Replies)
Discussion started by: mohtashims
4 Replies

4. UNIX for Dummies Questions & Answers

Condition based on Timestamp (Date/Time based) from logfile (Epoch seconds)

Below is the sample logfile: Userids Date Time acb Checkout time: 2013-11-20 17:00 axy Checkout time: 2013-11-22 12:00 der Checkout time: 2013-11-17 17:00 xyz Checkout time: 2013-11-19 16:00 ddd Checkout time: 2013-11-21 16:00 aaa Checkout... (9 Replies)
Discussion started by: asjaiswal
9 Replies

5. UNIX for Dummies Questions & Answers

Order based on timestamp in a single file

Hi All, I have a large text file which is a combination of multiple files. This is what I used and it worked. for i in /home/docs/text/* do cat $i >> Single_File done Now wondering, if there is a way to sort that single large file based on timestamps in ascending order. Text file... (11 Replies)
Discussion started by: prrampalli
11 Replies

6. Shell Programming and Scripting

Read directories sequential based on timestamp

Hi, I have a directory structure like below Directoryname create time d1 12:00 d2 12:05 d3 12:08 I want to read the directories based on timestamp.That is oldest directory must be read first and kick off certain process. ... (7 Replies)
Discussion started by: chetan.c
7 Replies

7. Shell Programming and Scripting

Extract date from filename and set timestamp

I have lots of files in this format: dvgrab-2003.06.29_15-30-24.mpg The numbers represents the date and time (YYYY.MM.DD_HH-MM-SS) How can I extract the dates from the filenames, and use the dates in the file timestamp? I guess this can be done by using "find", "sed" and "touch"? Can... (6 Replies)
Discussion started by: qwerty1234
6 Replies

8. UNIX for Dummies Questions & Answers

How to pick only the latest files based on the timestamp?

I have a few log files which get generated on a daily basis..So, I need to pick only the ones which get generated for that particular day. -rw-r--r-- 1 staff 510732676 Apr 7 22:01 test.log040711 -rwxrwxrwx 1 staff 2147482545 Apr 7 21:30 test.log.2 -rwxrwxrwx 1 staff 2147482581 Apr 7 19:26... (43 Replies)
Discussion started by: win4luv
43 Replies

9. Shell Programming and Scripting

copy files based on creation timestamp

Dear friends.. I have the below listing of files under a directory in unix -rw-r--r-- 1 abc abc 263349631 Jun 1 11:18 CDLD_20110603032055.xml -rw-r--r-- 1 abc abc 267918241 Jun 1 11:21 CDLD_20110603032104.xml -rw-r--r-- 1 abc abc 257672513 Jun 3 10:41... (5 Replies)
Discussion started by: sureshg_sampat
5 Replies

10. Shell Programming and Scripting

How to extract timestamp from the filename?

Hi., My file name is of the format: name_abc_20100531_142528.txt where., my timestamp is of the format: yyyymmdd_hhmmss How to extract the date strring and time string into seperate variables in the shell script, after reading the file as the input? I want to get the variables... (9 Replies)
Discussion started by: av_vinay
9 Replies
Login or Register to Ask a Question