Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Working with files that are named with a timestamp

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Working with files that are named with a timestamp
# 1  
Old 12-06-2012
Working with files that are named with a timestamp
So I have a script that runs an ls on a given directory and looks for files with a timestamp that has the current hour in it and then does work with the files that it discovers

Code:
DATE=`date +'%y%m%d%H'`
HOSTIS=`hostname`
#EMAIL NOTIFICATION ALS
EMAIL=address@server.com
#

if [ $HOSTIS = 'serverhostname' ]; then
  TOHOST='destination_address@destination_server.com'
 else
#This is for the test server
 TOHOST='test_address@test_server.com'
fi

for i in `ls /transfer |grep $DATE`
do

   sftp  -vvv $TOHOST 2>&1 | tee /homedir/user/debug.out<<EFT
   cd /directory/file
      lcd /transfer
      put $i
EFT
   echo "SFTP Completed successfully."
   echo "SFTP Completed successfully." |mailx -s "SFTP completed successfully" $EMAIL
exit
done

This is then run as a cron at 58 after the hour. This has obvious implications if the user manages to generate data between hh:58:01 and HH:00:00. it was a small risk but a risk nonetheless. It only took them two weeks. So now I need to figure out how to search for $DATE -1 hour and run the cron based on that and it has me stumped...

any ideas?
# 2  
Old 12-06-2012
Just thinking...
Keep everything as is, except...
the 2nd line of the script add
Code:
sleep 180

which would make the time xx:01 but you are checking for things in the previous hour.
# 3  
Old 12-06-2012
An interesting hack to do it, but wouldn't I need to adjust the cron run time as well? set it for 59 instead?

Quote:
Originally Posted by joeyg
Keep everything as is, except...
the 2nd line of the script add
Code:
sleep 180

which would make the time xx:01 but you are checking for things in the previous hour.
# 4  
Old 12-06-2012
try assuming OSX:

Code:
date -v-[something you want] +'%y%m%d%H'

where [something you want] is a past time
-1H == one hour ago
-5M == 5 minutes ago.

So try -15M and run it at :05 after the hour.

A comment to make your life easier -- :

Those filename "timestamps" could easily be stale. ie., I started my process at 12:59.
It has 12 as the hour, because it was opened at 12:59, when open created the file.

But. It is now 13:02. My report process is still running! So, when you sftp, you are not going to get my whole file. It has not closed yet. To be safe you really should look at the mtime of the files.
# 5  
Old 12-06-2012
Further explanation
Quote:
An interesting hack to do it, but wouldn't I need to adjust the cron run time as well? set it for 59 instead?
No.
If the cron starts running at x:58
it will get the full data/time string at that moment
then wait three minutes (could be two, bu one more just to be sure)
then does everything else

Plus, if you start at x:59 perhaps (maybe?) the time will be x:00 by the time that command is returned. Why take a chance on it.
# 6  
Old 12-06-2012
Found my own solution.
This is my fault, I always forget to mention distro. This is Solaris 10. As such it doesn't have the mmtime switch in find.

What it does have is -newer that compares the results against the mtime of the file called out after the -newer switch is set.

so
Code:
DATE=`date +'%y%m%d%H'`
NEWER=/users/userid/.newer
LOG=/users/userid//sftp.log
HOSTIS=`hostname`
#EMAIL NOTIFICATION ALS
#EMAIL=user@domain.com

if [ $HOSTIS = 'hostname' ]; then
  TOHOST='user@domain.com'
 else
#This is for the validation server
 TOHOST='user@domain.com'
fi


for i in `find /transfer -name '*.txt' -newer $NEWER -print `
do
   sftp  $TOHOST <<EFT
   cd /inbound/dir
      lcd /transfer
      put $i

EFT
   touch -a -m $NEWER;
   echo "SFTP Completed successfully." |mailx -s " SFTP completed successfully" $EMAIL
exit
done

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to zip or rm the multi part named files?

Hello There, There are more than 1000 files in my log folder and i want to zip it to relase the space. But my method throwing syntax error due to the multi part file name, how to overcome in this ? ls -lart | grep "MDB_Kernel11.1_gwlog_SUN 22_09_2013" | awk '{print $9,$10,$11,$12}' | head... (8 Replies)
Discussion started by: gowthamakanthan
8 Replies

2. Shell Programming and Scripting

[Solved] awk manipulation of sequentially named files

Hello, I am a very novice user of awk, I have a set of files named file001, file002, file003, file004, etc., each contains four fields (columns of data) separated each by a uneven number of spaces. I want to substitute those spaces by a TAB, so I am using this line of awk script: awk -v OFS="\t"... (4 Replies)
Discussion started by: jaldo0805
4 Replies

3. Shell Programming and Scripting

Joining Similarly Named Files

Have a few hundred files that I have no control over. I need to join them on a common string. There are two files per unit. The file names are: AAAA_X_file1.txt AAAA_file2.txt join -a1 file1 file2 - this works. My question is, is there a way to use the join command listing the file... (4 Replies)
Discussion started by: jimmyf
4 Replies

4. Shell Programming and Scripting

Identifying files with a timestamp greater than a given timestamp

I need to be able to identify files with file timestamps greater than a given timestamp. I am using the following solution, although it appears to compare files at the "seconds" granularity and I need it at the milliseconds. When I tested my solution, it missed files that had timestamps... (3 Replies)
Discussion started by: nkm0brm
3 Replies

5. Shell Programming and Scripting

Move all files not in a directory into a subdirectory named for each given file

Hi Everyone! Looking for some help with a script that will take all files in any given root folder (which are not already in a folder) and put them into separate folders with the name of each given file. Any ideas? Thank you! (1 Reply)
Discussion started by: DanTheMan
1 Replies

6. Shell Programming and Scripting

sort the files based on timestamp and execute sorted files in order

Hi I have a requirement like below I need to sort the files based on the timestamp in the file name and run them in sorted order and then archive all the files which are one day old to temp directory My files looks like this PGABOLTXML1D_201108121235.xml... (1 Reply)
Discussion started by: saidutta123
1 Replies

7. Shell Programming and Scripting

cat certain files in directories to files named after the dir?

Hi all, I have a directory with many subdirectories each named like so: KOG0001, KOG0002, ...KOG9999. Each of these subdirectories contain a variable number two kinds of files (nuc and prot) named like so: Capitella_sp_nuc_hits.fasta (nuc) and Capitella_sp_prot_hits.fasta (prot). The... (2 Replies)
Discussion started by: kmkocot
2 Replies

8. Shell Programming and Scripting

Finding missing files that are named sequentially with Perl?

Hello I am new to Perl, in fact I am on chapter one of the book. :) However I am in need of a Perl Script faster than I can finish the book. Perhaps someone can help me with my immediate need while I read my book. I have a directory with hundreds of files that are all named like... (4 Replies)
Discussion started by: newftronics
4 Replies

9. UNIX for Dummies Questions & Answers

fifo or named pipe working?

Can someone explain to me the working of fifo() system call using simple C programs so that I can implement them in the UNIX environement? (1 Reply)
Discussion started by: lvkchaitanya
1 Replies

10. UNIX for Dummies Questions & Answers

Comparing files named by date/time

I've looked at several of the previous posts and can't seem to find any that pertain to my problem, I'd appreciate some help if possible. I have a directory with numerous logs of various names all named by heading and date ie. dog.20050529.log dog.20050530.log ... (2 Replies)
Discussion started by: gillr
2 Replies
Login or Register to Ask a Question