UNIX for Dummies Questions & Answers

Hello,

I'm working on a little project to extract weather data from website.
At this point, I've stored each weather information on one line in a UNIX standard file.
Each line has the structure:

Code:

dd-mm-yy:City:dd-mm-yy:kind Of Data:value Of Data

the first dd-mm-yy (it's the french format because I'm french) is the date of the download of the forecasts weather, the second dd-mm-yy is the day for the forecasts (it could be the day of the download, the day after the download, the day after the day of the download etc...).

I'll now give an example of forecast, I use 5 lines:

Code:

08-12-10:Paris:10-12-10:Windspeed:20 km/h
08-12-10:Paris:10-12-10:Windcourse:Wind north north-east
08-12-10:Paris:10-12-10:Sky:Some breaks
08-12-10:Paris:10-12-10:Max:5°C
08-12-10:Paris:10-12-10:Min:-3°C

Well, now my problem, I need to catch the different values on some variables:
for example, I'd like to have in 5 variables ($downloadDay, $City, $forDate etc...) the 5 arguments of a line (only for 1 line!) and I need a little tips.

Actually, I get the line using the sed command and I store the line in a variable.
Then I get the values with the expr command, my commands are:

Code:

downloadDay=`expr "$myline" : '\(........\)'`
city=`expr "$myline" : '.*\([A-Z][a-z]*\):[0-9]'`
forecastday= # I cut it to display the command, in the script it's linked
`expr "$myline" : '.*[a-z]:\([0-9][0-9]-[0-9][0-9]-[0-9][0-9]-[0-9][0-9]\):[A-Z]'`
kindOfData=`expr "$myline" : '.*[0-9]:\([A-Z][a-z]*\)'` 
valueOfData=`expr "$myline" : '.*[a-z]:\(.*\)'`

the two elements which fail are: the forecastday and the valueOfData

for the forecastday, I've more than 100 lines in my weather base, sometimes it's run, but sometimes it fails:

Run:

Code:

03-12-10:Mulhouse:04-12-10:Ciel:Soleil voilé

Fail:

Code:

03-12-10:Versaille:03-12-10:Ciel:Quelques éclaircies

for the valueOfData:

Actually, the data is cut when I've a space: 50 km/h -> 50
some breaks-> some etc...

I think that my problem comes from the 5 regex with expr, I'm new in this domain, so if somebody could have a look and explain me what's wrong and how to fix it (if possible with a regex explanation to understand why that not run) It'd be very very very nice!

Big thanks in advance!

Moderator's Comments:

Please use code tags when posting data and code samples!

One way....
Depending on the final objective, you might be better off with other tools ( e.g. awk) - but that's the start.

Code:

#!/bin/ksh

while IFS=: read downloadDay City forecastDay kindOfData valueOfData junk
do
     # do your stuff with the extracted fields here
     echo "downloadDay -> [${downloadDay}]  City -> [${City}]"
done < myForecastFile

Thanks for this first answer,

actually, my "strategy" was to store the data line by line and to separate each element by a flag, typically the

Code:

:

So my idea was to read the content between two flags and to do this for each element.
Unfortunately, I don't know how to proceed and I don't know advanced tools such as awk, I commonly use expr, sed and grep.
So I've tried to make some regex but that doesn't work as well as I would like...

When I extracted the data from the web page, I used the same regex structure using the mark out and tag to do the extraction such that:

Code:

'FlagOne\(.*\)FlagTwo'

and that was good but in my case here, I've not the slightest idea...

Are my regex not valid or not efficient and should I try another method?

Thanks!

You have right strategy, but a bad choice of tools that are both are insufficient and that would consume quite a bit of the CPU.

Think of your data as records/lines separated by the InputFileSeparator (IFS) ':'. Shell itself (without using any other tools) provides the vehicles to deal with field separated records.

If/when your processing requirements become more 'mature' you might consider migrating to using other *NIX scripting tools, e.g. awk, perl, python etc...

Take a look at what I've provided and try to run it to see if it satisfies your requirements 'as-is'.

Thanks for the advice, I think it's going well!

I've just found this example:

http://www.livefirelabs.com/unix_tip...3/10132003.htm and that's typically what I want to do, with one exception... I don't want to "echo" but I'd like to store in 5 variables... If I play the handyman again, I'll set my element's variable and a "tmp" var out of the for loop and increase its value for each passage in the loop, and by using a "if" structure I could select in the loop what variable will be fill by the current record.

But I find that very dirty and spoiling the CPU time (exactly like you said).

Have you a more elegant method to set and fill my element's variables?

Thanks again!

look at what I've posted originally.

Woo oo,
I though that was the awk command but you gave me the solution actually by the shell itself

I'll try it tomorrow because it's late in my country and I'm tired.
I'll come back later if necessary...

You've probably fix the problem, so thanks a lot again