Parsing a column of text file - best practices

 
Thread Tools Search this Thread
Top Forums UNIX for Beginners Questions & Answers Parsing a column of text file - best practices
# 36  
Old 06-11-2017
Hi Don -

Thank you for the note! You're in CA responding at 130am - do you ever sleep? LOL

I apologize, I thought I included my environment file but I didn't.

Here it is:
Code:
#!/bin/bash
#::--:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
#::-- Script Name: _env.sh                                                    --::
#::--                                                                         --::
#::-- Description: This environment file is set to hold all variables         --::
#::--              for all Hyperion/Oracle scripts requiring:                 --::
#::--                                                                          --::
#::--              1. Path variables                                          --::
#::--              2. User ID(s)                                              --::
#::--              3. Logon(s)                                                --::
#::--              4. Password(s)                                              --::
#::--              5. Server(s)                                                   --::
#::--              6. Application(s)                                          --::
#::--               7. Database(s)                                             --::
#::--               8. etc                                                      --::
#::--                                                                          --::
#::--  Calls:      Not Applicable                                             --::
#::--  Called By:  All Scripts                                                --::
#::--                                                                              --::
#::-- Parameters:  Not Applicable                                               --::
#::--                                                                              --::
#::-- Author:      Name (Company )                                             --::
#::-- Date:                                                                       --::
#::                                                                              --::
#::--:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::

#::::::::::::::::::::::::::::
#::-- Set Path Variables --::
#::::::::::::::::::::::::::::

cd $HOME

_MAINPATH=$(pwd)/Hyperion_Batch/
_LOGPATH=Logs/
_ERRORPATH=Errors/
_FILEPATH=Files/
_FLAGFILEPATH=${_MAINPATH}${_FILEPATH}FlagFiles/
_BATCHPATH=${_MAINPATH}Scripts/Batch/
_MAXLSCRIPTPATH=${_MAINPATH}Scripts/MaxL/
_SUBVARPATH=${_MAINPATH}${_FILEPATH}Subvar/

#:::::::::::::::::::::::::::::::
#::-- Set Essbase Variables --::
#:::::::::::::::::::::::::::::::

_ESSB_USER=user
_ESSB_PSWD=password
_ESSB_SRVR=exalytics-madc-01.server.lan

_STARTMAXLPATH=/u02/EssbaseServer/essbaseserver1/bin/startMaxl.sh

#::::::::::::::::::::::::::::
#::-- Set Time Variables --::
#::::::::::::::::::::::::::::

_DAY=$(date +%d)
_MONTH=$(date +%m)
_QUARTER=$(((_MONTH+2)/3))
_YEAR=$(date +%Y)
_DATESTAMP=${_YEAR}${_MONTH}${_DAY}
_HOUR=$(date +%H)
_MINUTE=$(date +%M)
_SECOND=$(date +%S)
_TIME=${_HOUR}${_MINUTE}${_SECOND}
_DATETIMESTAMP=${_DATESTAMP}_${_TIME}

_PREV_DAY=$(date --date yesterday "+%d")
_PREV_MONTH=$(date --date yesterday "+%m")

#:::::::::::::::::::::::::::::::::::
#::-- Export Relevant Variables --::
#:::::::::::::::::::::::::::::::::::

export _MAINPATH _LOGPATH _ERRORPATH  _FILEPATH _FLAGFILEPATH _SCRIPTPATH _MAXLSCRIPTPATH _SUBVARPATH
export _ESSB_USER _ESSB_PSWD _ESSB_SRVR _STARTMAXLPATH
export _DAY _MONTH _QUARTER _YEAR _DATESTAMP _HOUR _MINUTE _SECOND _TIME _DATETIMESTAMP _PREV_DAY _PREV_MONTH

That should answer all your questions above but please let me know if you need anything else. Again thank you so much for helping me!

Last edited by SIMMS7400; 06-11-2017 at 09:05 AM..
# 37  
Old 06-11-2017
Sleep. What's that? Smilie

The way that _env.sh calculates _QUARTER only works if the quarter you are interested in is based on the calendar month in which you run your script. I thought the quarter used in your script was supposed to be based on the fiscal month (not the calendar month). For example, if you had run your script on Saturday, December 31, 2016 do you want that to be 1Q2017 (based on the fiscal month) or 4Q2016 (based on the calendar month)?

Do any of the other scripts that source _env.sh depend on the way it sets _QUARTER? The code I'm writing currently waits to set the variables it uses that specify quarters until after it has determined in which fiscal quarter the most recent Saturday before the date on which the script was run (or the date given to it as an operand if an operand was found on the command line) is located.

Last edited by Don Cragun; 06-11-2017 at 04:08 PM.. Reason: Fix typo: s/interested is/interested in is/
This User Gave Thanks to Don Cragun For This Post:
# 38  
Old 06-11-2017
Hi Don -

If I run my script on Saturday, December 31st, 2016, it should be 1Q2017 - correct.

Also, please do what you need to with the _QUARTER variable. I only made that variable for this process so please build as necessary. Nothing else is dependent on that.

Thank you!
# 39  
Old 06-11-2017
Please just remove the definition of _QUARTER from _env.sh (so no other scripts see it and are lured into using it incorrectly).

My script won't be depending on any of the time variables defined in _env.sh; it will only be using the path and Essbase variables it sets.

Note that the way you're setting each of your time variables with separate invocations of date can lead to confusing results. For example if you start your script at ~23:59:59 on 12/31/2016 the value assigned to _DATETIMESTAMP could be any of the following:
Code:
20161231_235959
20161231_235900
20161231_230000
20161231_000000
20171231_000000
20170131_000000
20170101_000000

assuming that all of the invocations of the date utiity are completed in a period of 1 second (showing results that cover a period of about 30 days).

When getting a group of related time variables, it is safer to do so with one invocation of date. For example:
Code:
read _DAY _MONTH _YEAR _HOUR _MINUTE _SECOND <<-EOF
        $(date '+%d %m %Y %H %M %S')
EOF
_DATESTAMP=${_YEAR}${_MONTH}${_DAY}
_TIME=${_HOUR}${_MINUTE}${_SECOND}
_DATETIMESTAMP=${_DATESTAMP}_${_TIME}

which will reduce the possibilities to a consistent date and time stamp with all values at the end of 12/31/2016 or all values at the start of 1/1/2017.

Are we having fun yet?
# 40  
Old 06-11-2017
Hi Don -

This is great - I learn every time you post. It's helped me immensely and I can't thank you enough.

Will do I will remove QUARTER variable going forward. Thank you again and I'm pumped to see the end results!
# 41  
Old 06-16-2017
Hi SIMMS7400,
I still have a lot of work to do to clean some things up and to correctly set up your log files and configuration files, but I would like for you to verify that the stuff I have done so far is correctly setting variables the way you want them set. The early part of the following output is debugging information just meant for me (but may give you some insight into how it works). The last part of the following output contains the variable assignments I think you want in the weekly configuration file you hope to generate if you run your script on Saturday, June 17, 2017. Please verify that I have correctly set the variables you want, that the values assigned to those variables are what you want (correct placement of 4 digit years and 2 digit years, etc.), and that I haven't missed any variables that you want to be included in your configuration files.
Code:
$ tester 06/17/2017
Date given as operand: "06/17/2017"
Saturday at the start of the week is 06/17/2017
Friday at the end of this week is 06/23/2017
Therefore, current week is in 2Q2017(17) and 1H2017(17)
FNQ=3Q2017(17)
FPQ=1Q2017(17), FPPQ=4Q2016(16)
Fiscal Month: JUN2017(17)
Previous Fiscal Month: MAY2017(17)
2nd Previous Fiscal Month: APR2017(17)
Calculated 1st Saturday of 2Q2017(17): 04/01/2017(4/1/17)
Calculated 1st Saturday of 3Q2017(17): 07/01/2017(7/1/17)
Calculated last Saturday of 2Q2017(17): 06/24/2017(6/24/17)

Calculations are done & Variables are set, print results:

"1PeriodPrior",MAY17
"1PeriodPriorq",'"MAY17"'
"2PeriodPrior",APR17
"2PeriodPriorq",'"APR17"'
ALLC_CurrentPeriod,JUN17
ALLC_CurrentPeriodq,'"JUN17"'
ALLC_CurrentWeek,'06/17/17'
ALLC_CurrentWeekq,'"06/17/17"'
CurrentHalfq,'"FY 1H2017"'
CurrentPeriod,JUN17
CurrentPeriodq,'"JUN17"'
CurrentPlanYear,'FY 2017'
CurrentPlanYearq,'"FY 2017"'
CurrentQtrInput,'FY 2Q2017_input'
CurrentQtrInputq,'"FY 2Q2017_input"'
CurrentQuarter,'FY 2Q2017'
CurrentQuarterq,'"FY 2Q2017"'
CurrentWeek,'06/17/17'
CurrentWeekq,'"06/17/17"'
CurrentYear,'FY 2017'
CurrentYearq,'"FY 2017"'
FirstQtrWeek,'04/01/2017'
LastQtrWeek,'06/24/2017'
PriorQtrInput,'FY 1Q2017_input'
PriorQuarter,'FY 1Q2017'
PriorQuarterq,'"FY 1Q2017"'
PriorQuarterAD,'AD1-17'
PriorQuarterADq,'"AD1-17"'
PriorQuarterMnth1,JAN17
PriorQuarterMnth1q,'"JAN17"'
PriorQuarterMnth2,FEB17
PriorQuarterMnth2q,'"FEB17"'
PriorQuarterMnth3,MAR17
PriorQuarterMnth3q,'"MAR17"'
$ 

This User Gave Thanks to Don Cragun For This Post:
# 42  
Old 06-18-2017
Hi Don -

Thank you for all your help. I reviewed your above post and everything looks 100% - I see no issues or anything that needs correcting.

Thank you!!!!
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Parsing a fixed column text file in sed

I have a text file with records of the form: A X1 Y1 X2 Y2 X3 Y3 where A is character length 10, Xi is character length 4 and Yi is numeric length 10. I want to parse the line, and output records like: A X1 Y1 A X2 Y2 A X3 Y3 etc Can anyone please give me an idea of how to do this. ... (4 Replies)
Discussion started by: wvdeijk
4 Replies

2. Shell Programming and Scripting

Parsing text file

Hi Friends, I am back for the second round today - :D My input text file is this way Home friends friendship meter Tools Mirrors Downloads My Data About Us Help My own results BLAT Search Results ACTIONS QUERY SCORE START END QSIZE IDENTITY CHRO STRAND ... (7 Replies)
Discussion started by: jacobs.smith
7 Replies

3. Shell Programming and Scripting

Parsing text file

I'm totally stumped with how to handle this huge text file I'm trying to deal with. I really need some help! Here is what is looks like: ab1ba67c331a3d731396322fad8dd71a3b627f89359827697645c806091c40b9 0.2 812a3c3684310045f1cb3157bf5eebc4379804e98c82b56f3944564e7bf5dab5 0.6 0.6... (3 Replies)
Discussion started by: comp8765
3 Replies

4. Programming

Parsing a Text file using C++

I was trying to parse the text file, which will looks like this ###XYZABC#### ############ int = 4 char = 1 float = 1 . . ############ like this my text file will contains lots of entries and I need to store these entries in the map eg. map.first = int and map.second = 4 same way I... (5 Replies)
Discussion started by: agupta2
5 Replies

5. UNIX for Dummies Questions & Answers

Replacing a specific column of a text file with another column

Hi, I have a text file in the following format: Code: 13412 NA06985 0 0 2 46.6432798439 4 4 4 4 13412 NA06991 NA06993 NA06985 2 48.8478948517 4 4 2 4 13412 NA06993 0 0 1 45.8022601455 4 4 2 4 13401 NA06994 0 0 1 48.780669145 4 4 4 4 13401 NA07000 0 0 2 47.7312017846 2 4 4 4 ... (2 Replies)
Discussion started by: evelibertine
2 Replies

6. UNIX for Dummies Questions & Answers

Replacing a specific column of a text file with another column

I have a text file in the following format: 13412 NA06985 0 0 2 46.6432798439 4 4 4 4 13412 NA06991 NA06993 NA06985 2 48.8478948517 4 4 2 4 13412 NA06993 0 0 1 45.8022601455 4 4 2 4 13401 NA06994 0 0 1 48.780669145 4 4 4 4 13401 NA07000 0 0 2 47.7312017846 2 4 4 4 13402 NA07019... (3 Replies)
Discussion started by: evelibertine
3 Replies

7. Shell Programming and Scripting

Need help parsing a text file

I have a text file: router1#sh ip blah blah | incl --- Gi2/8 10.60.4.181 --- 10.60.123.175 11 0000 0000 355K Gi2/8 10.60.83.28 --- 224.10.10.26 11 F9FF 3840 154K Gi2/8 10.60.83.198 --- ... (1 Reply)
Discussion started by: streetfighter2
1 Replies

8. Shell Programming and Scripting

Column wise file parsing.

Shell script for the below operation : File "A" contains : SEQ++1' MOA+9:000,00:ABC' RFF+AIK:000000007' FII+PH+0170++AA' NAD+PL+++XXXXXXXXXXX XXXXXXX XX++XXX XXXX XXXX X.X. XXXXXXXXX+++NL' SEQ++2' MOA+9:389,47:ABC' RFF+AIK:02110300000008' FII+PH+0PSTBNL2A:25:5+BB'... (5 Replies)
Discussion started by: navojit dutta
5 Replies

9. Shell Programming and Scripting

Parsing text from file

Any ideas? 1)loop through text file 2)extract everything between SOL and EOL 3)output files, for example: 123.txt and 124.txt for the file below So far I have: sed -n "/SOL/,/EOL/{p;/EOL/q;}" file Here is an example of my text file. SOL-123.go something goes here something goes... (0 Replies)
Discussion started by: ndnkyd
0 Replies

10. Shell Programming and Scripting

Text File Parsing

Hey Guys.I am a newbie on Bash Shell Scripting and Perl.And I have a question about file parsing. I have a log file which contains reports about a communication device.I need to take some of the reports from the log file.Its hard to explain the issue.but shortly I can say that, the reports has a... (2 Replies)
Discussion started by: Djlethal
2 Replies
Login or Register to Ask a Question