Shell script to read multiple options from file, line by line
Hi all
I have spent half a day trying to create a shell script which reads a configuration file on a line by line basis.
The idea of the file is that each will contain server information, such as IP address and various port numbers. The line could also be blank (The file is user created). Here is an example
The script should ignore any empty lines, obviously. For the line with server 172.18.1.1, default settings should be used as nothing else is specified (Default is UDP mode port 500). For the line 172.18.50.1, the specified settings are that tcp mode is to be used, and SSH and SSL on port 8443 options are also to be used.
This is what I have created, as an example. The final product will obviously do much more, it is the reading in of the lines and variables that I need help with.
The idea here is that each line will be read in. grep will attempt to find a valid IP address in that line, and if it does ( if [ ! -z $IP ] ) then the line will be checked for all possible options. Otherwise if there is no valid IP address on the line it will go to the next line.
I know that I am going wrong somewhere. I cannot use the $SRV string to get any information, although this pasted script simply gives no response. I also believe that using
will not work with the servers file where the line contains spaces. Is this true?
Can anyone advise on the commands I should be using to read the lines and gain these variables. Could I use a for loop instead?
Thanks for any help
PS also, is the usage of grep ok? I had used awk before but this seems to work ok and be shorter. Is there any advantage of one over the other? Thanks
Last edited by haggismn; 06-09-2012 at 06:05 PM..
Reason: fixed ssl,ssh,http if -z statements
Ok, let us start at the end, because this is easiest:
Quote:
Originally Posted by haggismn
PS also, is the usage of grep ok? I had used awk before but this seems to work ok and be shorter.
Actually neither awk nor grep is recommended. If you call an external program (regardless which one) from the shell you start a fork()-ed process. To create such a process costs an awful lot of time. You might want to read this thread where i learned the same lesson the very hard way.
To come back to your main question: what you need is a "parser". If you really want to indulge in the theory and practice of (recursively) parsing arbitrary languages you might want to read the classic "Dragon Book" ("Principles of Compiler Design"; Aho, Sethi, Ullman). It is a phantastic book about an intriguing field but for your rather limited purposes a very simple approach will suffice.
Lets start with a few thoughts about your input file:
1. We want empty space, like leading blanks, trailing blanks, empty lines, etc., not to influence the outcome, because this would make for a really awkward handling of files which users should prepare. A single mis-paced space char, which would be invisible, would prevent correct parsing, so we don't want that.
Well, the best way to prevent space from having any meaning is to remove it prior to even looking at the file. Question: do we have to handle quoted strings? If not, we can even throw all successive whitespace between words out and replace them with a single space. So let us start with a sketch of a script. We use sed for this, because it is only called ONCE for the whole input (replace <spc> and <tab> with literal space/tab chars):
The "print" line will later be removed, it is just there to let us see what we do. On to the next part:
2. We want to have comments, because it is easier for people to be able to comment directly in place what they do instead of having to use separate documents. As configuration files get longer some comments might be practical. We could implement multi-line comments like in C, but this would be overkill, so we settle for the same comments as in shell scripts: everything following a "#" is considered a comment. Now, it might be that "#" is part of a word and we do not want to remove half the word because it could be a comment, therefore we consider "#" to be introducing a comment only if it is either on the start of a line or preceeded by a space.
Let us change the sed-statement accordingly, to remove everything we don't need our parser to see:
3. Now that we have taken care of the preliminaries we have to start on the real work: what should our config file look like? Which format do we want be able to recognize?
We start first with identifying necessary and optional values: the IP-address is obviously mandatory. The port-list is optional and we define a default for that. Then the mode: is it optional and we create a default? Is it mandatory? Is there any other information which should/could come up on a line? You want to consider this first and prepare a list like the following:
Be very thorough with this list, you will see why.
There are three basic layouts for your config file: stanza, delimited file and what i call option-file. The easiest to parse is the option-file, which contains only declarations of the form "identifier=value". For instance, it could look like:
The problem (or advantage?) with this is that it only can contain a single system. You could put all these config-files to a directory and cycle through these. For some problems this is a good choice, you decide if this is good in your case. A parser would look like this (i have left out consistency checks to make it easier to follow, we will fill these in later):
It is a good idea btw., to put the processing of the system to a separate function and call that instead of doing all the work in one single program. It makes the code better readable and easier to maintain.
The next possibility would be the delimited file. It is a table with a certain delimiter character as field separator. Spreadsheet programs use this format for data interchange frequently ("comma-separated file"). It will allow for all the configuration data in a single file, but optional values will have to be left explicitly empty. In the option file you could simply leave out an optional value for which a default exists, not so here. Furthermore you have to decide on a delimiter char which cannot be used in text, unless we want to further complicate matters by introducing escaping:
This file type is relatively easy to parse, we chop off from the start of the line to the next delimiter until we reach the end. Because we have a fixed succession of fields we do not need field names like in the first type, but this also makes it easier for people to make errors by exchanging field values, if the fields get more. This is what a parser could look like:
The last possibility is the stanza file format. It allows for easy handling of default options because fields can simply be left out. It is also possible to have multiple entries in a single file (which - see above - might be a good or bad thing, depending on your environment).
The stanza file looks like this:
In your case it could look like this:
Unfortunately this is the most complicated to parse of the three formats, but it is definitely the most flexible. Let's get to it:
We start with an identifier (in our case the IP address) and read and store one line after the other until we encounter another identifier (or the end of the input file). This tells us we have read the whole record and we process it before we start over to read. We will - for the purpose of the example - suppose that "mode" is mandatory to show how mandatory fields are handled.
OK, as you see there is a lot of pseudo-code in there, which you have to fill with your checks. This post is getting very long so i would like to discuss this in a separate post. Please give me some kind of feedback first, it is quite some work to write this and i wouldn't want to do this unwanted.
Some last suggestions:
1) You should decide what to do with doubled directives, which could occur in the option-file and the stanza-file. For instance:
You could: let the last one take precedence; warn the user and skip the record for ambiguousity; add all the options up to one, so that the example would be equivalent to "ports=5,6,7,8,9,10".
2) you might want to allow for spaces between the equal signs and the field names/values:
To achieve this it is only necessary to put the following directive into the sed-statement (which throws these out so that the provided code would go unchanged):
3) A similar device could be employed in the delimited file, where blanks surrounding delimiter chars could be thrown out previous to parsing:
i have a text file as belows, it includes 2 columns, 1st is the column name, 2nd is the file_name
data_file.txt
column_name file_name
col1 file1
col2 file2
col3 file1
col4 file1
col5 file2
now, i would like to... (4 Replies)
Hello,
I'm works on Ubuntu server
My goal : I would like to read file line per line, but i want to started at the end of file.
Currently, I use instructions :
while read line;
do
COMMAND
done < /var/log/apache2/access.log
But, the first line, i don't want this. The file is long... (5 Replies)
Hi All,
Am trying to write wrapper shell/bash script on a utility tool for which i need to pass 2 files as arugment to execute utility tool.
Wraper script am trying is to do with above metion 2 files.
utility tool accepts :
a. userinfo file : which contains username
b. item file : which... (2 Replies)
I have a LOG file which looks like this
Import started at: Mon Jul 23 02:13:01 EDT 2012
Initialization completed in 2.146 seconds.
--------------------------------------------------------------------------------
--
Import summary for Import item: PolicyInformation... (8 Replies)
Hi ,
I am trying to write an shell, which reads a text file (from a location) having a list of numbers of strictly 5 digits only ex: 33144
Now my script will check :
1) that each entry is only 5 digits & numeric only, no alphabets, & its not empty.
2)then it executes a shell script called... (8 Replies)
I am using the while-loop to read a file.
The file has lines with null-terminated strings (words, actually.)
What I have by that reading - just a first word up to '\0'!
I need to have whole string up to 'new line' - (LF, 10#10, 16#A)
What I am doing wrong?
#make file 'grb' with... (6 Replies)
Hi,
I need to read a text file from shell script line by line and copy the feilds of each line.
Below is the complete requirement.
I've text file which contains ...
pgm1 file11 file12 file13
pgm2 file21 file22
pgm3 file31 file32 file33
I'll give input as... (4 Replies)
Hi,
I have gps receiver log..its giving readings .like below
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
GPSD,R=1
$GPGSV,3,1,11,08,16,328,40,11,36,127,00,28,33,283,39,20,11,165,00*71... (3 Replies)