06-09-2008
Problem number 1 is you're mixing date math with integer math. You cannot add 1 to your "0530" and expect to get the next day. For example, there is no integer "0530". There is an integer "530", however, and if you add 1 to it you will get "531". ...NOT the "0531" that you need. And, as you mentioned, adding 1 to that will not get you into June.
The problem of shell date arithmetic is fairly difficult. See
"Date math in Linux shell script?": Tech Support from Ask Dave Taylor! .
When you do your comparison, you can test for your end condition in either of two ways:
- If the current date == the end date (string comparison)
- If the current iterations number of seconds since the epoch is less than or equal to the end date's number of seconds since the epoch.
Also, here's a couple of more gotchas you need to be aware of. If you do your date math using "number of seconds since the Epoch",
- You may want your first date to actually be midnight of the first date- that is, the 0'th second of that day.
- You may want your second date to actually be 23:59:59 on that day.
This is because if you do your "begin <= end" comparison, you may be comparing 3pm on the end day to 2pm on the end day and that iteration of the script will not run.
I mention that because it's convenient to actually do the date math by:
- Convert the first date to the number of seconds since the epoch
- Increment the date by adding 86,400 (the number of seconds in a day) to the date for each iteration.
You also must be aware of when the year changes. Don't forget you may be comparing dates in January to dates in December.
-mschwage
10 More Discussions You Might Find Interesting
1. UNIX for Dummies Questions & Answers
Hi,
I have written a script to rename a file, but I can not add the current date (in YYYYMMDD-HHMM format)
can you please look at this cript, and help?
thanks,
#!/usr/local/bin/tcsh -f
set server = "$1"
set user = "$2"
if (-f $server) then
\mv $server $server.Saar.
endif (4 Replies)
Discussion started by: sierra_aar
4 Replies
2. Shell Programming and Scripting
Hi all,
please give me the commands using which i can compare 2 sorted files and get the difference in third file, indiating where the difference is from either file1 or file2.
as:
File1 (Original file)
GARRY
JOHN
JULIE
SAM
---------------
File2
DEV
GARRY
JOHN
JOHNIEE (7 Replies)
Discussion started by: varungupta
7 Replies
3. UNIX for Dummies Questions & Answers
Hi:
I am trying to create some script that sorts the files in a subdirectory by date and concatenates the thre most recently created files.
SAy,
file1 date1
file2 date2
file3 date3
file4 date4
file5 date5
file6 date6
i only want to concatenate the first three which are the most... (4 Replies)
Discussion started by: jlarios
4 Replies
4. Shell Programming and Scripting
Say i have 2 files in the giving format:
file1
1 2 3 4
1 2 3 4
1 2 3 4
file2
1 2 3 4
1 2 3 4
1 2 3 4
I have a PERL code (loaned by one of u -i forgot who - thanks!) that extracts the 2nd column from each file and append horizontally to a new file:
perl -ane 'push @{$L->}, $F; close... (1 Reply)
Discussion started by: epi8
1 Replies
5. Programming
Hi,
I want to merge two files with sorted integers. I want to merge two files to a single file such that there are no duplicates and the merged file should contain the integers in the sorted order. Just some idea or pointer to an implementation will help me a lot.
Thanks & Regards,... (4 Replies)
Discussion started by: brett01
4 Replies
6. UNIX for Dummies Questions & Answers
the sorting is based on name of file,
file size
modification time stamps o f file
it should dislay the output in the following format
"." and ".." enteries should be ignored
please give some idea how to do it (1 Reply)
Discussion started by: pappu kumar jha
1 Replies
7. Shell Programming and Scripting
I have multiple files; each file contains a certain data in a column view
simply i want to combine all those files into one file in columns
example
file1:
a
b
c
d
file 2:
1
2
3
4
file 3:
G (4 Replies)
Discussion started by: ahmedamro
4 Replies
8. Shell Programming and Scripting
Hi
I have a requirement like below
I need to sort the files based on the timestamp in the file name and run them in sorted order and then archive all the files which are one day old to temp directory
My files looks like this
PGABOLTXML1D_201108121235.xml... (1 Reply)
Discussion started by: saidutta123
1 Replies
9. Shell Programming and Scripting
Gents,
I have lot of files in a folder where each file name includes the date of generation, then I would like to merge
all the files for each date in a complete file.
list of files in forder.
dsd01_121104.txt
dsd01_121105.txt
dsd01_121106.txt
dsd03_121104.txt
dsd03_121105.txt... (7 Replies)
Discussion started by: jiam912
7 Replies
10. Shell Programming and Scripting
Can anyone please help me i have 2 text files setup like the one below.
Textfile1:
randomemail1:randompassword1
randomemail2:randompassword2
randomemail3:randompassword3
randomemail4:randompassword4
randomemail5:randompassword5
Textfile2:
randompassword1:randomphrase1... (8 Replies)
Discussion started by: nufc
8 Replies
LEARN ABOUT DEBIAN
clfmerge
clfmerge(1) logtools clfmerge(1)
NAME
clfmerge - merge Common-Log Format web logs based on time-stamps
SYNOPSIS
clfmerge [--help | -h] [-b size] [-d] [file names]
DESCRIPTION
The clfmerge program is designed to avoid using sort to merge multiple web log files. Web logs for big sites consist of multiple files in
the >100M size range from a number of machines. For such files it is not practical to use a program such as gnusort to merge the files
because the data is not always entirely in order (so the merge option of gnusort doesn't work so well), but it is not in random order (so
doing a complete sort would be a waste). Also the date field that is being sorted on is not particularly easy to specify for gnusort (I
have seen it done but it was messy).
This program is designed to simply and quickly sort multiple large log files with no need for temporary storage space or overly large buf-
fers in memory (the memory footprint is generally only a few megs).
OVERVIEW
It will take a number (from 0 to n) of file-names on the command line, it will open them for reading and read CLF format web log data from
them all. Lines which don't appear to be in CLF format (NB they aren't parsed fully, only minimal parsing to determine the date is per-
formed) will be rejected and displayed on standard-error.
If zero files are specified then there will be no error, it will just silently output nothing, this is for scripts which use the find com-
mand to find log files and which can't be counted on to find any log files, it saves doing an extra check in your shell scripts.
If one file is specified then the data will be read into a 1000 line buffer and it will be removed from the buffer (and displayed on stan-
dard output) in date order. This is to handle the case of web servers which date entries on the connection time but write them to the log
at completion time and thus generate log files that aren't in order (Netscape web server does this - I haven't checked what other web
servers do).
If more than one file is specified then a line will be read from each file, the file that had the earliest time stamp will be read from
until it returns a time stamp later than one of the other files. Then the file with the earlier time stamp will be read. With multiple
files the buffer size is 1000 lines or 100 * the number of files (whichever is larger). When the buffer becomes full the first line will
be removed and displayed on standard output.
OPTIONS
-b buffer-size
Specify the buffer-size to use, if 0 is specified then it means to disable the sliding-window sorting of the data which improves the
speed.
-d Set domain-name mangling to on. This means that if a line starts with as the name of the site that was requested then that would be
removed from the start of the line and the GET / would be changed to GET http://www.company.com/ which allows programs like Webal-
izer to produce good graphs for large hosting sites. Also it will make the domain name in lower case.
EXIT STATUS
0 No errors
1 Bad parameters
2 Can't open one of the specified files
3 Can't write to output
AUTHOR
This program, its manual page, and the Debian package were written by Russell Coker <russell@coker.com.au>.
SEE ALSO
clfsplit(1),clfdomainsplit(1)
Russell Coker <russell@coker.com.au> 0.06 clfmerge(1)