Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Converting date string to different formats

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Converting date string to different formats
# 1  
Old 05-07-2010
Question Converting date string to different formats
Sucks to be a noob Smilie

See my last post for solution

I have 3 different log formats and the filenames contain a date. I am trying to write a script to grep these files between two date ranges.

I am now stuck at the date conversion bit. I let the user entered a date string in the format "18.05.2009" which I need to convert to these 3 different formats:

example filenames are.
Code:
WW Logs format  "access1004211716_" //the last 4 digits are the time
BC log format "HTTP_20100421_"
NC Log format "http_log.100.2010.04.28"

I know I can use awk but having read an hour up on material I still have no clue how to archive what I want in my bash script.

The next step will be even harder as I need to search between 2 date ranges Smilie

any help is apperciated.
Last edited by GermanJulian; 05-13-2010 at 11:01 AM..
# 2  
Old 05-07-2010
Hello,
maybe if the start date and end date are near you can grep for each date in the interval and for each date format.
If the difference between dates is a year...this "solution" is not very nice.

Sorry, I read again your message and is it possible you want to create different date formats from one enter by a user?

So, you can do this:
FECHA=date entered by the user y dd.mm.yyyy format
$day=$(echo $FECHA | cut -d . -f 1)
$mont=$(echo $FECHA | cut -d . -f 2)
$year=$(echo $FECHA | cut -d . -f 3)

And now, you only have to echo in the desired order..

---------- Post updated at 06:09 PM ---------- Previous update was at 05:38 PM ----------

Code:
fecha_desde=$1
fecha_hasta=$2

dia1=$(echo $fecha_desde | cut -d . -f 1)
mes1=$(echo $fecha_desde | cut -d . -f 2)
ano1=$(echo $fecha_desde | cut -d . -f 3)
dia2=$(echo $fecha_hasta | cut -d . -f 1)
mes2=$(echo $fecha_hasta | cut -d . -f 2)
ano2=$(echo $fecha_hasta | cut -d . -f 3)

fecha1=$ano1$mes1$dia1
fecha2=$ano2$mes2$dia2

# format accessDATE...
for fich in $(ls -1 access*)
do
   valor=20${fich:6:6}
   if [ $valor -ge $fecha1 ] && [ $valor -le $fecha2 ]
   then
       echo $fich " Ok!"
   fi
done

# format HTTP_DATE...

 "similar"

Last edited by albertogarcia; 05-07-2010 at 12:52 PM..
# 3  
Old 05-07-2010
ahhh I see what you did Smilie

will try the code this weekend.

The date range could be for several month... but lets see if I can figure that one out myself with the help of google Smilie
# 4  
Old 05-08-2010
Something like this. Format date yyyymmdd, then you can use numeric comparing.
You need fix the awk substr lines. Look values from real loglines.
Code:
#!/bin/ksh or bash or ...
datestr()
{
input=$1
oifs="$IFS"
IFS="."
values=($input)
IFS="$oifs"
d=${values[0]}
m=${values[1]}
y=${values[2]}
datestr="$y$m$d"
}

date1=$(datestr 12.10.2009)
date2=$(datestr 12.10.2010)

awk -v d1=$date1 -v d2=$date2 '
/access[0-9][0-9]/ {
    datestr=substr($0:7:6}
    datelong=sprintf("20%s",datestr)
    if (datelong < d1  || datelong > d2) { next }
    print $0
    next
    }
/HTTP_[0-9][0-9]/ {
    datelong=substr($0:6:6}
    if (datelong < d1  || datelong > d2) { next }
    print $0
    next
    }
/http_log.*[0-9][0-9]/ {
    y=substr($0:14:4}
    m=substr($0:19:2}
    d=substr($0:22:2}
    datelong=sprintf("%s%s%s",y,m,d)
    if (datelong < d1  || datelong > d2) { next }
    print $0
    next
    }
' logfile

# 5  
Old 05-10-2010
mhh I can not get it to work. The last example I do not even understand?

For example lets just do files which are called access, e.g.

access1004202232.merged-00.21.9b.94.43.a1.log.gz

and I want to find all all entries between date1 and date2 I can not get it to work.

Code:
#!/bin/bash

date1="100501"
date2="100502"


for i in `find /APAC/ -name *.gz* -name access*`; do awk -v d1=$date1 -v d2=$date2 ' ;done

moreover if I just try this in bash I get weird results?

Code:
 find /APAC/ -name *.gz* -name access* | awk -v /100501/,/100502/

I also get files that are 100503 and 04 etc etc.
Last edited by GermanJulian; 05-10-2010 at 01:07 PM..
# 6  
Old 05-10-2010
Quote:
Originally Posted by GermanJulian
...
For example lets just do files which are called access, e.g.

access1004202232.merged-00.21.9b.94.43.a1.log.gz

and I want to find all all entries between date1 and date2 I can not get it to work.
...
Maybe this ?

Code:
$ 
$ # show all files whose names start with "access"
$ ls -1 access*
access1004202232.merged-00.21.9b.94.43.a1.log.gz
access1005010932.merged-00.21.9b.94.43.a1.log.gz
access1005012357.merged-00.21.9b.94.43.a1.log.gz
access1005021349.merged-00.21.9b.94.43.a1.log.gz
access1005022359.merged-00.21.9b.94.43.a1.log.gz
access1005030058.merged-00.21.9b.94.43.a1.log.gz
$ 
$ # pick up files created between "100501" and "100502"
$ perl -le 'print foreach (grep /^access10050[12]/, (glob "access*"))'
access1005010932.merged-00.21.9b.94.43.a1.log.gz
access1005012357.merged-00.21.9b.94.43.a1.log.gz
access1005021349.merged-00.21.9b.94.43.a1.log.gz
access1005022359.merged-00.21.9b.94.43.a1.log.gz
$ 
$ # or using just the shell
$ ls -1 access10050[12]*
access1005010932.merged-00.21.9b.94.43.a1.log.gz
access1005012357.merged-00.21.9b.94.43.a1.log.gz
access1005021349.merged-00.21.9b.94.43.a1.log.gz
access1005022359.merged-00.21.9b.94.43.a1.log.gz
$ 
$

tyler_durden
# 7  
Old 05-13-2010
So after spending hours being stuck at this I found out that the best way to do this is to convert the filename date into unix time and then simple do a bigger then smaller then if statement with the user entered dates.
Converting to unix time was another issue as I am on solaris not GNU.

Its a blast now analyzing logs between two dates out of 3000 or more files in 1 year

Here is my solution:

User entered dates:
Code:
choosedates()
	{
		echo "----------------------------------------------------------------"
		echo "Which date range would you like to search for in region $REGION"? 
		echo "Choose your start (from) date with the below format:"
		echo "dd.mm.yyyy e.g. 08.05.2009"
		until (test "$CHECK" = "Y" || test "$CHECK" = "y")
			do 
				read STARTDATE
				echo "You have choose $STARTDATE, is this ok? [y/n]"
				read CHECK
		done
		echo "Choose your end date with the below format:"
		echo "dd.mm.yyyy e.g. 02.11.2010"
		until (test "$CHECK2" = "Y" || test "$CHECK2" = "y")
			do 
				read ENDDATE
				echo "You have choose $ENDDATE, is this ok? [y/n]"
				read CHECK2
		done
		
	CHECK=""
	CHECK2=""
	#Converting the user dates and converting them to unix epoche time	
	dayS=$(echo $STARTDATE | cut -d . -f 1)
	monS=$(echo $STARTDATE | cut -d . -f 2)
	yeaS=$(echo $STARTDATE | cut -d . -f 3)
	dayE=$(echo $ENDDATE | cut -d . -f 1)
	monE=$(echo $ENDDATE | cut -d . -f 2)
	yeaE=$(echo $ENDDATE | cut -d . -f 3)
		
	STARTDATE=`/opt/bin/perl /home/user/scripts/perldateconverter.pl $yeaS$monS$dayS`
	ENDDATE=`/opt/bin/perl /home/user/scripts/perldateconverter.pl $yeaE$monE$dayE`
	}

Reading log files and converting log file names into a timestamp then converting to unix epoch

Code:
for logfile in `ls -R1 $REGION/*/*/access*.gz` ;
			do 
			#Convert date in filename to unix timestamp
			logfiledate=$(echo $logfile | sed -e 's/.*access\(......\).*/\1/')
			logfiledate=`/opt/bin/perl /home/user/scripts/perldateconverter.pl 20$logfiledate`  ##this particular logfile has dates as 09 or 01 not 2009 or 2010
			#Analyse log files
			if [ $logfiledate -ge $STARTDATE ] && [ $logfiledate  -le  $ENDDATE ];then
				echo "Analysing file: $logfile"
				if [ -z "$GREPCOMMAND" ];then
					/usr/bin/gzcat $logfile >> ~/$FILETOWRITEWW
					else
					/usr/bin/gzcat $logfile |  (eval $GREPCOMMAND) >> ~/$FILETOWRITEWW
				fi
			fi
		done

perl date converter
Code:
use strict;
use warnings;
use DateTime::Format::Strptime;

my $str = shift;
my $parser =
  DateTime::Format::Strptime->new( pattern => '%Y%m%d' );
my $dt = $parser->parse_datetime( $str );
print $dt->epoch;

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Converting String Date into UNIX Date

Hi, I have a string date to my unix script(sun solaris). I wanted to convert it into unix date so that I can use it in a conditional statement. Please see below: MyTest.sh -s 2018-05-09 suppdt=$1 # string date passed via arguement as 2018-04-09 curryr=`date '+%Y'` nextyr=`expr... (2 Replies)
Discussion started by: Saanvi1
2 Replies

2. Shell Programming and Scripting

Perl:: mass replacement of converting C code formats to tgmath.h

hello, i have a lot of C old code I'm updating to C11 with tgmath.h for generic math. the old code has very specific types, real and complex, like cabsl, csinhl, etc usually for simple bulk replacements i would do something simple like this perl -pi -e 's/cosl/cos/g' *.c the reference... (0 Replies)
Discussion started by: f77hack
0 Replies

3. UNIX for Dummies Questions & Answers

Converting string date time to unix time in AWK

I'd like to convert a date string in the form of sun aug 19 09:03:10 EDT 2012, to unixtime timestamp using awk. I tried This is how each line of the file looks like, different date and time in this format Sun Aug 19 08:33:45 EDT 2012, user1(108.6.217.236) all: test on the 17th ... (2 Replies)
Discussion started by: bkkid
2 Replies

4. Shell Programming and Scripting

Converting a date to friday date and finding Min/Max date

Dear all, I have 2 questions. I have a file with many rows which has date of the format YYYYMMDD. 1. I need to change the date to that weeks friday date(Ex: 20120716(monday) to 20120720). Satuday/Sunday has to be changed to next week friday date too. 2. After converting the date to... (10 Replies)
Discussion started by: 2001.arun
10 Replies

5. Shell Programming and Scripting

Extracting data from a log file with date formats

Hello, I have a log file for the year, which contains lines starting with the data in the format of YYYY-MM-DD. I need to get all the lines that contain the DD being 04, how would I do this? I tried using grep "*-*04" but it didn't work. Any quick one liners I should know about? Thank you. (2 Replies)
Discussion started by: cpickering
2 Replies

6. Shell Programming and Scripting

Converting string to date in perl

Hi, I need convert a date string to date. For eaxmple $last_date=6/2/2009 and I want to change the format of the above mentioned date to "Jun 2 2009 12:00AM". Do we have any functionality or staright method to convert to the desired format? (4 Replies)
Discussion started by: siba.s.nayak
4 Replies

7. UNIX for Dummies Questions & Answers

Help with Date Formats

Hi, Following are the results of various date formats: 1. date +%h" "%d Result: Jun 02 2. date Result: Tue Jun 2 09:59:15 CDT 2009 If i use the date format as date +%h%d then i am getting the date as 02. I want the day to be displayed as "2" instead of "02". so my result should... (1 Reply)
Discussion started by: sandeep_1105
1 Replies

8. Solaris

different date formats in same server

when I ssh and run date command, it shows date in 24 hour date format. But when I telnet the same server, it shows date in 12 hour format, ie. in AM/PM (1 Reply)
Discussion started by: na75369
1 Replies

9. HP-UX

a simple way of converting a date in seconds to normal date

Hi all! I'm working on a HPUX system, and I was wondering if there is a simple way to convert a date from seconds (since 1970) to a normal date. Thanks (2 Replies)
Discussion started by: travian
2 Replies

10. UNIX for Dummies Questions & Answers

date formats

Hi, I want to generate file name in the following date format, "YYYYMMDDHHHHMISS" plz. help me how to do that? (6 Replies)
Discussion started by: harshad.katruwa
6 Replies
Login or Register to Ask a Question