Login or Register to Ask a Question and Join Our Community


UNIX for Beginners Questions & Answers

Date from filename inserted into records

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums UNIX for Beginners Questions & Answers Date from filename inserted into records
# 1  
Old 06-18-2016
Combine files & express Date in Julian days
Hi Folks,

I have below files in one directory:
Code:
Spiross-MBP:AIRTEMP spirospap$ ls -1
CPK2004001
CPK2004002
CPK2004003
etc...
JFK2003001
JFK2003002
JFK2003003
etc...
TEB1999001
TEB1999002
TEB1999003
etc...

Month/year is in Filename and also in the file Header, first line.
I only need 3 fields (out of several) and manage to filter single files and export the result to a new file with below awk one liner:
Code:
spirospap$ awk -F"," '{ print $1, $2, $9 }' CPK2004001 >CPK2004001.txt
Month:  01/2004 
Station Name:  "CENTRAL PARK" Call Sign: 
Day Time Dry Bulb Cel
01 0051    5.0
01 0151    5.0
01 0251    5.0
01 0351    5.6
01 0451    5.0
01 0551    5.0
01 0651    3.9
etc...
31 1851   -3.9
31 1951   -4.4
31 2051   -4.4
31 2151 -
31 2251    -.6
31 2351   -4.4

Tasks:
1. Combine date from Header (or Filename) and express first field in Julianday instead of month day.
2. Do this in one blow for all files in the directory, ending up with a single file for each station/year were each record looks like
Code:
Julianday Time Temperature

Any advise even for partial task(s) will be greatly appreciated.

Thanks
Last edited by spirospap; 06-18-2016 at 05:17 PM..
# 2  
Old 06-18-2016
Not that I understood half of what you wrote, but for your printing problem you might use
Code:
awk '{print $1, $2, sprintf ("%5.1f", $9?$9:-11.1)}' file

, checking if $9 IS an empty string. Please be aware that in 31 2151 - there's NO missing value.
Last edited by RudiC; 06-18-2016 at 04:23 PM..
# 3  
Old 06-18-2016
I simplified the request. Thanks for the comment and the printing suggestion although it doesn't produce the desired result.
# 4  
Old 06-18-2016
If you're trying to compute the Julian day, it would help to know what operating system and shell you're using. And, if you don' have access to the GNU date utility, do you have access to a 1993 or later version of the Korn shell.

Please show us the output you are trying to produce. From your description, I am not at all sure where you want the Julian date(s) to appear in your output.
# 5  
Old 06-18-2016
Hello,

Re kernel please see below:
Code:
Spiross-MBP:AIRTEMP spirospap$ uname -a
Darwin Spiross-MBP.fios-router.home 15.5.0 Darwin Kernel Version 15.5.0: Tue Apr 19 18:36:36 PDT 2016; root:xnu-3248.50.21~8/RELEASE_X86_64 x86_64

The file contents look like this:
Code:
Spiross-MBP:AIRTEMP spirospap$ head CPK2004001
Month:, 01/2004
Station Name:, "CENTRAL PARK" Call Sign:, NYC
Day,Time,StationType,Maint Indic,SkyConditions,Visibility,Weather Type,Dry Bulb Faren,Dry Bulb Cel,Wet Bulb Faren,Wet Bulb Cel,Dew Point Faren,Dew Point Cel,Rel Humd,Wind Speed,Wind Dir,Wind Char Gusts,Val. for Wind Char,Station Pressure,Press Tend,Sea Level Pressure,Report Type,Precip Total
01,0051,AO2 ,-,CLR                                          ,10SM   ,-,41   ,   5.0,-,   2.1,28   ,  -2.2, 60 , 6   ,VRB,-,0  ,29.92,5,180,AA,-
01,0151,AO2 ,-,CLR                                          ,10SM   ,-,41   ,   5.0,-,   2.2,28   ,  -2.2,

I only need fields 1, 2 and 9 which contain Day of the Month, Hour and Temperatures in Degrees Celsius (labeled Dry Bulb Cel). I can do this by using the code below:
Code:
Spiross-MBP:AIRTEMP spirospap$ awk -F"," '{ print $1, $2, $9 }' CPK2004001 >CPK2004001.txt
Spiross-MBP:AIRTEMP spirospap$ head CPK2004001.txt 
Month:  01/2004 
Station Name:  "CENTRAL PARK" Call Sign: 
Day Time Dry Bulb Cel
01 0051    5.0
01 0151    5.0
01 0251    5.0
01 0351    5.6
01 0451    5.0
01 0551    5.0
01 0651    3.9

Eventually, I would like to have the first field (DAY) in Julian days like this:
Code:
Month:  01/2004 
Station Name:  "CENTRAL PARK" Call Sign: 
Julian Time Dry Bulb Cel
001 0051    5.0
001 0151    5.0
001 0251    5.0
001 0351    5.6
001 0451    5.0
001 0551    5.0
001 0651    3.9
......
001 2351
002 0051   4.0
002 0151   5.0
etc...
002 0151

I hope this clarifies
Thanks
# 6  
Old 06-18-2016
Making some wild guesses about what you really want on the 1st three lines in each of your output files (.e., all of the data with <comma>s changed to <space>s instead of throwing away the call signs), the following seems to work using ksh on OS X to calculate the Julian dates for the 1st day of the month for the year and month specified by the filename:
Code:
#!/bin/ksh
for file in [A-Z][A-Z][A-Z][0-9][0-9][0-9][0-9]0[01][0-9]
do	m=${file#????????}
	y=${file#???}
	y=${y%???}
	jd1=$(printf '%(%j)T' "$m/01/$y")
	printf '%s,%s\n' "$file" "$jd1"
done | awk -F ' *, *' '
FNR == NR {
	jdb[ARGV[ARGC++] = $1] = $2 - 1
	next
}
FNR == 1 {
	if(ofn)	close(ofn)
	ofn = FILENAME ".txt"
	gsub(/,/, " ")
	print > ofn
	next
}
FNR == 2 {
	gsub(/,/, " ")
	print > ofn
	next
}
FNR == 3 {
	printf("Julian  %s  %s\n", $2, $9) > ofn
	fmt = sprintf("   %%03d  %%4.4s  %%%d.%ds\n", length($9), length($9))
	next
}
{	printf(fmt, $1 + jdb[FILENAME], $2, $9 ? $9 : "-") > ofn
}' -

With the following input files:
File: CPK2004001
Code:
Month:, 01/2004
Station Name:, "CENTRAL PARK" Call Sign:, NYC
Day,Time,StationType,Maint Indic,SkyConditions,Visibility,Weather Type,Dry Bulb Faren,Dry Bulb Cel,Wet Bulb Faren,Wet Bulb Cel,Dew Point Faren,Dew Point Cel,Rel Humd,Wind Speed,Wind Dir,Wind Char Gusts,Val. for Wind Char,Station Pressure,Press Tend,Sea Level Pressure,Report Type,Precip Total
01,0051,AO2 ,-,CLR                                          ,10SM   ,-,41   ,   5.0,-,   2.1,28   ,  -2.2, 60 , 6   ,VRB,-,0  ,29.92,5,180,AA,-
01,0151,AO2 ,-,CLR                                          ,10SM   ,-,41   ,   5.0,-,   2.2,28   ,  -2.2,

File: JFK2003003
Code:
Month:, 03/2003
Station Name:, "JFK AIRPORT" Call Sign:, JFK
Day,Time,StationType,Maint Indic,SkyConditions,Visibility,Weather Type,Dry Bulb Faren,Dry Bulb Cel,Wet Bulb Faren,Wet Bulb Cel,Dew Point Faren,Dew Point Cel,Rel Humd,Wind Speed,Wind Dir,Wind Char Gusts,Val. for Wind Char,Station Pressure,Press Tend,Sea Level Pressure,Report Type,Precip Total
01,0051,AO2 ,-,CLR                                          ,10SM   ,-,41   ,      ,-,   2.1,28   ,  -2.2, 60 , 6   ,VRB,-,0  ,29.92,5,180,AA,-
10,0151,AO2 ,-,CLR                                          ,10SM   ,-,41   ,      ,-,   2.2,28   ,  -2.2,

File: TEB2000012
Code:
Month:, 12/2000
Station Name:, "TETERBORO AIRPORT" Call Sign:, TEB
Day,Time,StationType,Maint Indic,SkyConditions,Visibility,Weather Type,Dry Bulb Faren,Dry Bulb Cel,Wet Bulb Faren,Wet Bulb Cel,Dew Point Faren,Dew Point Cel,Rel Humd,Wind Speed,Wind Dir,Wind Char Gusts,Val. for Wind Char,Station Pressure,Press Tend,Sea Level Pressure,Report Type,Precip Total
01,0051,AO2 ,-,CLR                                          ,10SM   ,-,41   ,   5.0,-,   2.1,28   ,  -2.2, 60 , 6   ,VRB,-,0  ,29.92,5,180,AA,-
31,0151,AO2 ,-,CLR                                          ,10SM   ,-,41   ,  -5.0,-,   2.2,28   ,  -2.2,

it produces the following output files:
File: CPK2004001.txt
Code:
Month:  01/2004
Station Name:  "CENTRAL PARK" Call Sign:  NYC
Julian  Time  Dry Bulb Cel
   001  0051           5.0
   001  0151           5.0

File: JFK2003003.txt
Code:
Month:  03/2003
Station Name:  "JFK AIRPORT" Call Sign:  JFK
Julian  Time  Dry Bulb Cel
   060  0051             -
   069  0151             -

File: TEB2000012.txt
Code:
Month:  12/2000
Station Name:  "TETERBORO AIRPORT" Call Sign:  TEB
Julian  Time  Dry Bulb Cel
   336  0051           5.0
   366  0151          -5.0

Note that this code only works with a 1993 or later version of the Korn shell. (OS X comes with a BSD-based date utility; not a GNU date utility. And, although BSD date has a -d option, it sets the kernel's idea of daylight savings time offsets; it does not have a way to use the GNU date -d option to specify an alternative date to process.
This User Gave Thanks to Don Cragun For This Post:
spirospap
# 7  
Old 06-18-2016
Oh, this is perfection beyond belief! Thank you!

Is it possible to concatenate all the .txt files of a single year into a single file?
For example, all below files into one CPK2004.txt file?

Code:
Spiross-MBP:AIRTEMP spirospap$ ls -1 *.txt
CPK2004001.txt
CPK2004002.txt
CPK2004003.txt
CPK2004004.txt
CPK2004005.txt
CPK2004006.txt
CPK2004007.txt
CPK2004008.txt
CPK2004009.txt
CPK2004010.txt
CPK2004011.txt
CPK2004012.txt

Thank you
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

How to change existing date to current date in a filename?

Suppose i have a list of files in a directory as mentioned below 1. Shankar_04152019_ny.txt 2. Gopi_shan_03122019_mi.txt 3. Siva_mourya_02242019_nd.txt .. . . . . 1000 . Jiva_surya_02282019_nd.txt query : At one shot i want to modify the above all filenames present in one path with... (4 Replies)
Discussion started by: Shankar455
4 Replies

2. Shell Programming and Scripting

Comparing two files to get only records to be inserted and updated

Hello all, Please help me for a script that compares two files and reads only those records that are to be inserted and updated. File1: c_id name place contact_no 1 abc xyz 34567 10 efg uvw 82725 6 hjk wth 01823 2 iuy ... (4 Replies)
Discussion started by: T@ni@
4 Replies

3. Shell Programming and Scripting

How to append date to filename, but base it on yesterday's date?

Hello, I'd like to write a monthly archive script that archives some logs. But I'd like to do it based on yesterday's date. In other words, I'd like to schedule the script to run on the 1st day of each month, but have the archive filename include the previous month instead. Here's what I... (5 Replies)
Discussion started by: nbsparks
5 Replies

4. UNIX for Advanced & Expert Users

query display number lines or records present in file only numeric value -without filename

Hi all Thanks in advance........... Please help me for this issue............ I have a file it has 11 records . I used the command like .... >$ wc -l file 11 file I'm getting output like 11 file (no.of records along with filename) here my requirement is, I want to display only... (3 Replies)
Discussion started by: ksrivani
3 Replies

5. Shell Programming and Scripting

Get the oldest date based on date in the filename

I am using ksh93 on Solaris. Ok, this may seem like a simple request at first. I have a directory that contains sets of files with a YYYYMMDD component to the name, along with other files of different filespecs. something like this: 20110501_1.dat 20110501_2.dat 20110501_3.dat... (2 Replies)
Discussion started by: gary_w
2 Replies

6. Shell Programming and Scripting

date from filename

Hi all I have the following question: With this command, I get the latest file in a directory. lastfile =`ls -1tr | tail -n 1` echo $lastfile The output is then: partner131210.txt (meaning 13th December 2010) My goal is to get the date into a variable and to obtain a final variable... (4 Replies)
Discussion started by: davis77
4 Replies

7. UNIX for Dummies Questions & Answers

Date in filename

how do i add the date for the filename? for example filename20080917 (3 Replies)
Discussion started by: khestoi
3 Replies

8. Shell Programming and Scripting

Get date from filename

Hi all, I have this files: aaa20080714.log bbbb20080714.log ccccccc20080714.log Can i get the 20080714 from each file? (6 Replies)
Discussion started by: icy_blu_blu
6 Replies

9. Shell Programming and Scripting

display filename with date

Hi buddies, I have a doubt. I want to display filename with date in the following format.Is there any way to do this. Kindly give me the solution. I want to display the result in the following manner. test1.txt 03/28/2008 testlog.log 02/20/2008 Please let me know one solution how to do... (1 Reply)
Discussion started by: pstanand
1 Replies

10. Shell Programming and Scripting

filename to contain date

hello, can anyone tell me how to rename a filename in a script to contain the current date? i have searched for the answer but with little success! many thanks rkap (4 Replies)
Discussion started by: rkap
4 Replies
Login or Register to Ask a Question