Login or Register to Ask a Question and Join Our Community


UNIX for Dummies Questions & Answers

Really need some help with Parsing files by date

 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Really need some help with Parsing files by date
# 1  
Old 09-09-2013
Question Really need some help with Parsing files by date
Hi,
I am new to Unix and need some help with a problem I have.

I have been asked to ftp a file(s) from a directory to another system.
I need to be able to allow the current days file to build and only send any other file across that is in the directory. I need a peice of code that will read the date from a file e.g. "LABSDATA_20130909.DAT" know that it todays file and leave it but read the next file in e.g. "LABSDATA_20130908.DAT" and recognise it has the previous days date and ftp this file and any other file that doesn't contain todays date.

I have been told I can use parsing to do but I only have basic Unix experience.

Any advice, help or tips would be greatly appreciated.

Thanks
# 2  
Old 09-09-2013
It sounds like the filename contains the date in question as opposed to actually having to open the file. So the first think to do is to construct today's date in the same format contained in the filename. So assuming the file will always have LABSDATA and DAT, you would do
Code:
today_file="LABSDATA_`date +%Y%m%d`"
if [[ ! -f ${today_file} ]
then
  echo "I can't find today's file ${today}file}
  exit 
fi
# so process the file in however you see fit, lke transfer it by way
# of example
scp ${today_file} user@host:

This User Gave Thanks to blackrageous For This Post:
darrenwg99
# 3  
Old 09-09-2013
A simpler solution is to get all the files in the directory, and discard today's file.
But, if the intent is to get files every day, how will you know that you have already retrieved the file from two days ago?
If you have write permission on the remote directory, you could run an ftp process to delete each file after you have successfully processed it on the local system.
See this thread https://www.unix.com/shell-programmin...?highlight=ftp
Last edited by jgt; 09-09-2013 at 02:42 PM..
This User Gave Thanks to jgt For This Post:
darrenwg99
# 4  
Old 09-09-2013
A script that has exactly what you asked for - to paraphrase Oscar Wilde - the greatest unhappiness comes from getting what you asked for.

This makes no assumptions about anything, it uses remote site file times. And jgt is correct, if something does not remove files over there, you will suck 'em in again and again.

Create a file .netrc in your (or the user who does this) home directory. permissions 700.
Code:
machine remotehostname user ftpusername password ftpusrnamepassword

shell script, change /bin/ksh to the system's shell.
Code:
#/bin/ksh

ftp remotenode <<EOF | awk '{print $6 "-" $7 "-" $8, $(NF)}' > t.txt
dir
bye
EOF

# change remotenode to the correct name of the ftp server, in 2 places
# echo files older than today; dynmically build a script to run
echo 'ftp remotenode <<EOF ' > tmp.shl
today=$(date '+%b-%d-%Y')
awk -v t=$today ' $1==t {next} {print "get",  $2}' t.txt >> tmp.shl
echo 'bye' >> tmp.shl
echo 'EOF' >> tmp.shl
chmod +x tmp.shl
# run dynmaic script
./tmp.shl

Stuff in red changes
This User Gave Thanks to jim mcnamara For This Post:
darrenwg99
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Getting the current time from a website and parsing date

I am trying to work on a script to grab the UTC time from a website So far I was able to cobble this together. curl -s --head web-url | grep ^Date: | sed 's/Date: //g' Which gives me the result I need. Wed, 06 Dec 2017 21:43:50 GMT What I need to is extract the 21:43:50 and convert... (4 Replies)
Discussion started by: allisterB
4 Replies

2. Shell Programming and Scripting

How to list files that are not first two files date & last file date for every month-year?

Hi All, I need to find all files other than first two files dates & last file date for month and month/year wise list. lets say there are following files in directory Mar 19 2012 c.txt Mar 19 2012 cc.txt Mar 21 2012 d.txt Mar 22 2012 f.txt Mar 24 2012 h.txt Mar 25 2012 w.txt Feb 12... (2 Replies)
Discussion started by: Makarand Dodmis
2 Replies

3. Shell Programming and Scripting

New to UNIX ... Date parsing

Hi ... extremely new to Unix scripting ... I have to get a date field from mm/dd/yyyy to yyyy/mm/dd format ... I have a variable that I want parsed. I don't seem to be doing it right?? Thanks. (19 Replies)
Discussion started by: MJKeeble
19 Replies

4. Shell Programming and Scripting

Parsing the date output

Hi fellows, I need to define a notification for SSL certificate expiration. My Command output is below: (this is the "Expiration Date") Tue Mar 15 09:30:01 2012 So, at 15th Feb (1 month before the expiration), a notification has to be triggered by a script or sth else. How can i set an... (5 Replies)
Discussion started by: oduth
5 Replies

5. Shell Programming and Scripting

Parsing Log File Based on Date & Error

I'm still up trying to figure this out and it is driving me nuts. I have a log file which has a basic format of this... 2010-10-10 22:25:42 Init block 'UA Deployment Date': Dynamic refresh of repository scope variables has failed. The ODBC function has returned an error. The database... (4 Replies)
Discussion started by: k1ko
4 Replies

6. Shell Programming and Scripting

parsing multi-date text file

Hi all: Trying to parse a log file of rsync activity to get the amount of date being transferred. The log file contains multiple dates and what I am trying to do is get the file sizes for the current date. What I have been trying to do is pipe it through awk but I am having trouble retrieving... (1 Reply)
Discussion started by: raggmopp
1 Replies

7. UNIX for Dummies Questions & Answers

Parsing Date to a file name

Hi All There is a file "apple_2008-08-15.log". I have to use grep on this file to collect my test log. There are 45 such files. Is there a command that I can use to dynamically substitute the daily date as part of the file name? As of now Iam renaming the file to the new date and running my... (6 Replies)
Discussion started by: pk_eee
6 Replies

8. Shell Programming and Scripting

Sorting Files by date and moving files in date order

I need to build a k shell script that will sort files in a directory where files appear like this "XXXX_2008021213.DAT. I need to sort by date in the filename and then move files by individual date to a working folder. concatenate the files in the working folder then start a process once... (2 Replies)
Discussion started by: rebel64
2 Replies

9. Shell Programming and Scripting

Date parsing into string, help!

I have a file that was created yesterday that I want to access... daily.log2008-02-16 I am using... curr_time=`date +"%Y %m %d"` yesterday=`./datecalc.ksh -a $curr_time - 1` the value for yesterday is now "2008 2 16" in ksh, how do I transform the string "2008 2 16" into "2008-02-16" (5 Replies)
Discussion started by: martyb555
5 Replies

10. Shell Programming and Scripting

parsing a system log file via the 'date' command

Hello, I'm trying to update some scripts here that parse our system logs daily. They report information just fine... but they just report too much info. Specifically, if there's been some failed login attempts on several different days (say Monday and Tuesday), when I get the report from... (5 Replies)
Discussion started by: cjones
5 Replies
Login or Register to Ask a Question