Converting filenames from julian day to yyyy-mm-dd and retrieving weekly mean values


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Converting filenames from julian day to yyyy-mm-dd and retrieving weekly mean values
# 1  
Old 02-13-2012
Converting filenames from julian day to yyyy-mm-dd and retrieving weekly mean values

Hi, I need help to convert the filenames of my 9-year daily files (1999-2007) from a julian day to yyyy-mm-dd format. my original files are patterned likes the ones below.

Code:
1999001.txt
1999002.txt
1999003.txt
1999004.txt ...
1999365.txt

desired output:
19990101.txt
19990102.txt
19990103.txt
19990104.txt ...
19991231.txt

Based on the renamed files, I would like to take the average of the values of the 3rd column of the files at 7 days interval for each of the month. The daily files contains sample data like these.
Code:
1999001.txt renamed to 19990101.txt
12.3 124.4 2.0
12.1 124.2 1.2
12.4 123.5 1.5

1999002.txt renamed to 19990102.txt
12.3 124.4 1.8
12.1 124.2 2.2
12.4 123.5 1.2

1999003.txt renamed to 19990103.txt ...
12.3 124.4 1.5
 12.1 124.2 2.4
 12.4 123.5 1.5

1999007.txt renamed to 19990107.txt
12.3 124.4 1.2
12.1 124.2 2.6
12.4 123.5 1.8

hence, in Jan and Feb for instance, I would have resulting averaged files like the ones below:

199901_wk01.txt average of files in Jan 1999 from day 1-7
199901_wk02.txt average of files in Jan 1999 from day 8-14
199901_wk03.txt average of files in Jan 1999 from day 15-21
199901_wk04.txt average of files in Jan 1999 from day 22-28
199901_wk05.txt average of files in Jan 1999 from day 29-31

199902_wk01.txt average of files in Feb 1999 from day 1-7
and so on

Many thanks in advance.
# 2  
Old 02-14-2012
Easier to deal with using Python. Hope this works for you

Code:
#! /usr/bin/python

import datetime
import glob
import os


# $ cal 10 1999
#
#    October 1999
#Su Mo Tu We Th Fr Sa
#                1  2
# 3  4  5  6  7  8  9
#10 11 12 13 14 15 16
#17 18 19 20 21 22 23
#24 25 26 27 28 29 30
#31
#
# 1999-10-28 return 5
def getwk(y,m,d):
        blanks=datetime.date(y,m,1).weekday()
        return (d+blanks)/7+1


average=dict()
for source in glob.glob('1999[0-3][0-9][0-9].txt'):
        year=int(source[0:4])
        nday=int(source[4:7].lstrip('0'))
        dt=datetime.timedelta(days=nday-1)
        d=datetime.date(year,1,1)+dt
        target=str(d)+'.txt'

        os.rename(source,target)

        month=int(target[5:7].lstrip('0'))
        day=int(target[8:10].lstrip('0'))
        week=getwk(year,month,day)

        # 199903_wk03.txt
        target_wk=target[0:4]+target[5:7]+'_wk'+'%02d'%week+'.txt'

        for line in open(target):
                try:
                        average[target_wk].append(float(line.split()[2]))
                except:
                        average[target_wk]=[float(line.split()[2])]

# write out the average to week files.
for i in average:
        avg=float(sum(average[i]))/len(average[i])
        f=open(i,'w')
        f.write('%.2f\n'%avg)
        f.close()

I create the below script to generate all the sample files:
Code:
#! /bin/bash

rm -f 1999*.txt
seq -w 1 365 | while read d
do
        echo "12.3 124.4 $RANDOM
12.1 124.2 $RANDOM
12.4 123.5 $RANDOM" > 1999$d.txt
done

This User Gave Thanks to chihung For This Post:
# 3  
Old 02-14-2012
@chihung: thanks much for your reply. I havent tried running a python script, how do i execute a python script?many thanks
# 4  
Old 02-14-2012
Simply cut and paste the python code into a file and chmod 755 on that. Just like other shell scripts
This User Gave Thanks to chihung For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Pass date (YYYY-MM-DD) as parameter and get Day

Hi, I have a requirement where I have to pass Date to a script and get the day from it. Ex If parameter is 2015-09-29 The output should be Tuesday. Can you please tell me how to get that? (6 Replies)
Discussion started by: ashwin3086
6 Replies

2. Shell Programming and Scripting

Julian day to dates in YEAR-MONTH-DAY

hello, I have many files called day001, day002, day003 and I want to rename them by day20070101, day20070102, etc. I need to do it for several years and leap years as well. What is the best way to do it ? Thank you. (1 Reply)
Discussion started by: Ggg
1 Replies

3. Shell Programming and Scripting

Converting odd values to even values(or vice-versa) located in a column

Hello All, I have a below data in a .csv file where all rows where col1 is A, col2 is odd numbers, similarly even numbers for all rows where col1 is B. Note that my data has some other columns(not shown here) too (around 100) after col2. Tool,Data A,1 A,3 A,5 .... so on B,2 B,4 .... ... (4 Replies)
Discussion started by: ks_reddy
4 Replies

4. Shell Programming and Scripting

Day of year to dd.mm.yyyy format

Hi, How can I convert day of year value in format(yy,doy) to normal formatted (dd.mm.yyyy) string also all of them with awk or awk system function? in_file.txt --------- 12,043 12,044 12,045 12,046 out_file.txt ---------- 12.02.2012 13.02.2012 14.02.2012 15.02.2012 imagine... (5 Replies)
Discussion started by: kocaturk
5 Replies

5. Shell Programming and Scripting

Converting Date from YYYYMMDD to DD-MON-YYYY

Hi , I need to convert date from YYYYMMDD to DD-MON-YYYY e.g 20111214 to 14-Dec-2011 Please help. (17 Replies)
Discussion started by: ady_koolz
17 Replies

6. Shell Programming and Scripting

Converting date DD MM YYYY to DD MON YYYY

Hello, I am writing a script that parses different logs and produces one. In the source files, the date is in DD MM YYYY HH24:MI:SS format. In the output, it should be in DD MON YYY HH24:MI:SS (ie 25 Jan 2010 16:10:10) To extract the dates, I am using shell substrings, i.e.: read line ... (4 Replies)
Discussion started by: Adamm
4 Replies

7. Shell Programming and Scripting

Get current day on Julian date format

Hi guys, I know if I try to get a julian date using a specific date I can but I try to get the current date I got an error as you can see below: This one works fine: date -d "2010/10/30" +%j But I can't get the current date as below: `date -d "+%Y/%m/%d`" +%j Does somebody can... (6 Replies)
Discussion started by: edudiogo
6 Replies

8. Shell Programming and Scripting

Function to get day of week from YYYY-MM-DD date

Can't find out how to get the day of the week from a given date, anyone got a code snippet that could help please? Ta!! (4 Replies)
Discussion started by: couponmeup
4 Replies

9. Shell Programming and Scripting

converting the date field from dd/mm/yyyy to yyyy/mm/dd

How to convert the date field from dd/mm/yyyy to yyyy/mm/dd in unix my script will generate text file which have two fields one is date and another is name of the server for example this is sample date which I have to sort based on older to newer date the problem is when I found out sort will... (4 Replies)
Discussion started by: pareshan
4 Replies

10. Shell Programming and Scripting

Converting YYYYMMDD to Julian

I am writing some PERL code (and I realize this is a UNIX forum), but was wondering if anyone has a quick routine (PERL or shell scripting) to take a date in YYYYMMDD format and return the 3 digit Julian number. For instance, my program will have a variable called "$Settlement_Date" and will... (5 Replies)
Discussion started by: dfran1972
5 Replies
Login or Register to Ask a Question