Need help separating file lines into three classes


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Need help separating file lines into three classes
# 8  
Old 10-28-2012
Any suggestions on my process?
I can get the array to read out,

I even used a cut to pull the first two characters:
Code:
START=`echo $data[$INDEX] | cut -c1-2

But even once I tested that $START was collecting the correct data, I could not get my if statements to work.

Add to that, I know there is a more efficient way, but I don't know that method either.

Marc
# 9  
Old 10-28-2012
Quote:
Originally Posted by Marc G
Any suggestions on my process?
I can get the array to read out,

I even used a cut to pull the first two characters:
Code:
START=`echo $data[$INDEX] | cut -c1-2

But even once I tested that $START was collecting the correct data, I could not get my if statements to work.

Add to that, I know there is a more efficient way, but I don't know that method either.

Marc
I'm working on it. I have a way to do it in awk and I almost have a way to do it entirely in ksh, but I still have some testing to do...

I'm using arrays in a completely different way than you did.
# 10  
Old 10-28-2012
I eventually want to learn awk too.
So if you can explain it, I'd like that

But thanks for your help

Marc
# 11  
Old 10-28-2012
OK. Finally, here it is. The following Korn shell script provides a way to read a data file (named data), extract data items from that file, create a menu of the data items, and allows the user to interactively request the data associated with each menu entry. It actually provides two sample solutions for this problem and runs both solutions so the user can see the differences in behavior.

The first method is an awk script. It only uses features in awk that are included in the POSIX Standards. (Note that on Solaris systems, you'll need to use nawk or /usr/xpg4/bin/awk rather than /usr/bin/awk.)

The second method is a Korn shell script. It makes use of a few features that are not available in old versions of ksh; you'll need a version newer than November 16, 1988. It also makes use of some features that are not required by the POSIX standards.

The awk and ksh scripts use the same variable names. A summary of the variable names and their uses is provided in comments between the two scripts.

Here it is:
Code:
#!/bin/ksh
echo '*** 1st method: Using awk script... ***'
# Note that the variable names used in this awk script use the same variable
# names as the shell variables in the ksh script below.  A list of the
# variables and their use is included below for the ksh script.
#
# Note also that the echo and cat here are critical to allow awk to detect the
# EOF on the data file and display the prompt for the first round of
# interactive user input...
(echo;cat)|awk 'BEGIN{m = "Menu of options:\n"}
function mp(){printf("\n%s", m)}
{       # Throw away leading and trailing  whitespace on all input.
        sub(/^[[:space:]]*/, "")
        sub(/[[:space:]]*$/, "")
}
FNR == NR {
        # We are reading the data file here...
        if($0 == "") next # skip blank lines, sub() above made blank lines empty
        if($0 == "!")
                # Note that next non-empty data line starts a menu entry
                d1=1
        else if(d1) {
                # First line in data for menu choice mc, create menu entry
                d[++mc] = $0 "\n"
                d1=0
                m = sprintf("%s%d %s\n", m, mc,
                        match($0, /[^[:alpha:]]/) > 0 ? \
                                substr($0, 1, RSTART - 1) : $0)
        } else  # Subsequent line in data for menu choice mc
                d[mc] = d[mc] $0 "\n"
        next
}
FNR == 1 {
        # We are reading the line produced by the echo here.  Finish up the
        # menu, print the menu, and wait for the user to enter a selection:
        m = sprintf("%sEnter your choice 1 to %d (ctl-D to exit):\n", m, mc)
        mp()
        next
}
{       if($0 ~ /[^[:digit:]]/ || $0 < 1 || $0 > mc) printf("Invalid choice.\n")
        else printf("%s", d[$0])
        mp()
}' data -

sleep 1
echo '*** 2nd method: Just using ksh... ***'

# Shell variable data dictionary:
# d[x]  "d"ata for menu item x
# d1    != 0 => looking for "d"ata line #"1" for menu entry $mc
# in    current "in"put line
# m     "m"enu
# mc    "m"enu entry "c"ount
d1=0
m='Menu of options:'
while read -r in
do      if [[ "$in" == '' ]]
        then    continue        # skip empty lines in data file
        elif [[ "$in" == '!' ]]
        then    d1=1            # Next non-blank line is where we'll find the
                                # menu text and the start of the data text.
        elif [[ $d1 == 1 ]]
        then    # Gather data from 1st data line for a menu item.
                ((mc++))
                d[mc]="$(printf '%s\n' "$in")" # 1st line of data
                # Add a selection line to the menu.
                m="$(printf '%s\n%d %s' "$m" $mc "${in/%[^[:alpha:]]*/}")"
                d1=0            # no longer looking for the 1st data line
        else    # Add another line of data to the current entry.
                d[mc]="$(printf '%s\n%s' "${d[mc]}" "$in")"
        fi
done < data
m="$(printf '%s\nEnter your choice 1 to %d (ctl-D to exit): ' "$m" $mc)"
while printf '\n%s' "$m"        # Print the menu
do      if ! read -r in         # read the user's menu selection
        then    echo            # user entered ctl-d, reposition cursor & exit
                exit
        fi
        # Verify that the user's respons was not an empty line, didn't contain
        # any non-digit characters, and is a number between 1 and $mc inclusive.
        if [[ in == '' || ${in//[[:digit:]]/} != '' || in -lt 1 || in -gt mc ]]
        then    echo 'Invalid choice.'
                continue
        fi
        # We have a verified entry, print the selected data item.
        printf '%s\n' "${d[$in]}"
        # Give the user a few seconds to read the results before reprinting the
        # menu...
        sleep 2
done

and here is the data file I used to test it:
Code:
!
version 12.4

service timestamps debug datetime msec
service timestamps log datetime msec

no service password-encryption

!
hostname MarcsTestbox
!

boot-start-marker
boot-end-marker
!
  logging message-counter syslog
        enable secret {FIELD DELETED] [FIELD DELETED]
!
no aaa new-model
clock timezone
        !
etc.
.
.



.

The first part of it is data that was provided in the 1st message in this thread, but adds some spaces and tabs at the beginning and end of some non-empty lines, and adds some blank lines to test the ability of these scripts to handle blank lines, leading whitespace, and trailing whitespace.

I hope this helps you compare some of the features of awk and ksh for performing similar tasks.
# 12  
Old 10-28-2012
Don,

Thanks!
I need to look up "POSIX", but thank you!

Marc

Quote:
Originally Posted by Don Cragun
OK. Finally, here it is. The following Korn shell script provides a way to read a data file (named data), extract data items from that file, create a menu of the data items, and allows the user to interactively request the data associated with each menu entry. It actually provides two sample solutions for this problem and runs both solutions so the user can see the differences in behavior.

The first method is an awk script. It only uses features in awk that are included in the POSIX Standards. (Note that on Solaris systems, you'll need to use nawk or /usr/xpg4/bin/awk rather than /usr/bin/awk.)

The second method is a Korn shell script. It makes use of a few features that are not available in old versions of ksh; you'll need a version newer than November 16, 1988. It also makes use of some features that are not required by the POSIX standards.

The awk and ksh scripts use the same variable names. A summary of the variable names and their uses is provided in comments between the two scripts.

Here it is:
Code:
#!/bin/ksh
echo '*** 1st method: Using awk script... ***'
# Note that the variable names used in this awk script use the same variable
# names as the shell variables in the ksh script below.  A list of the
# variables and their use is included below for the ksh script.
#
# Note also that the echo and cat here are critical to allow awk to detect the
# EOF on the data file and display the prompt for the first round of
# interactive user input...
(echo;cat)|awk 'BEGIN{m = "Menu of options:\n"}
function mp(){printf("\n%s", m)}
{       # Throw away leading and trailing  whitespace on all input.
        sub(/^[[:space:]]*/, "")
        sub(/[[:space:]]*$/, "")
}
FNR == NR {
        # We are reading the data file here...
        if($0 == "") next # skip blank lines, sub() above made blank lines empty
        if($0 == "!")
                # Note that next non-empty data line starts a menu entry
                d1=1
        else if(d1) {
                # First line in data for menu choice mc, create menu entry
                d[++mc] = $0 "\n"
                d1=0
                m = sprintf("%s%d %s\n", m, mc,
                        match($0, /[^[:alpha:]]/) > 0 ? \
                                substr($0, 1, RSTART - 1) : $0)
        } else  # Subsequent line in data for menu choice mc
                d[mc] = d[mc] $0 "\n"
        next
}
FNR == 1 {
        # We are reading the line produced by the echo here.  Finish up the
        # menu, print the menu, and wait for the user to enter a selection:
        m = sprintf("%sEnter your choice 1 to %d (ctl-D to exit):\n", m, mc)
        mp()
        next
}
{       if($0 ~ /[^[:digit:]]/ || $0 < 1 || $0 > mc) printf("Invalid choice.\n")
        else printf("%s", d[$0])
        mp()
}' data -

sleep 1
echo '*** 2nd method: Just using ksh... ***'

# Shell variable data dictionary:
# d[x]  "d"ata for menu item x
# d1    != 0 => looking for "d"ata line #"1" for menu entry $mc
# in    current "in"put line
# m     "m"enu
# mc    "m"enu entry "c"ount
d1=0
m='Menu of options:'
while read -r in
do      if [[ "$in" == '' ]]
        then    continue        # skip empty lines in data file
        elif [[ "$in" == '!' ]]
        then    d1=1            # Next non-blank line is where we'll find the
                                # menu text and the start of the data text.
        elif [[ $d1 == 1 ]]
        then    # Gather data from 1st data line for a menu item.
                ((mc++))
                d[mc]="$(printf '%s\n' "$in")" # 1st line of data
                # Add a selection line to the menu.
                m="$(printf '%s\n%d %s' "$m" $mc "${in/%[^[:alpha:]]*/}")"
                d1=0            # no longer looking for the 1st data line
        else    # Add another line of data to the current entry.
                d[mc]="$(printf '%s\n%s' "${d[mc]}" "$in")"
        fi
done < data
m="$(printf '%s\nEnter your choice 1 to %d (ctl-D to exit): ' "$m" $mc)"
while printf '\n%s' "$m"        # Print the menu
do      if ! read -r in         # read the user's menu selection
        then    echo            # user entered ctl-d, reposition cursor & exit
                exit
        fi
        # Verify that the user's respons was not an empty line, didn't contain
        # any non-digit characters, and is a number between 1 and $mc inclusive.
        if [[ in == '' || ${in//[[:digit:]]/} != '' || in -lt 1 || in -gt mc ]]
        then    echo 'Invalid choice.'
                continue
        fi
        # We have a verified entry, print the selected data item.
        printf '%s\n' "${d[$in]}"
        # Give the user a few seconds to read the results before reprinting the
        # menu...
        sleep 2
done

and here is the data file I used to test it:
Code:
!
version 12.4

service timestamps debug datetime msec
service timestamps log datetime msec

no service password-encryption

!
hostname MarcsTestbox
!

boot-start-marker
boot-end-marker
!
  logging message-counter syslog
        enable secret {FIELD DELETED] [FIELD DELETED]
!
no aaa new-model
clock timezone
        !
etc.
.
.



.

The first part of it is data that was provided in the 1st message in this thread, but adds some spaces and tabs at the beginning and end of some non-empty lines, and adds some blank lines to test the ability of these scripts to handle blank lines, leading whitespace, and trailing whitespace.

I hope this helps you compare some of the features of awk and ksh for performing similar tasks.
# 13  
Old 10-29-2012
Quote:
Originally Posted by Don Cragun
Rudi,
In recent versions of ksh:
Code:
x=4
for i in {0..$x}
do echo $i
done

produces:
Code:
0
1
2
3
4

Sorry, I obviously missed the ksh remark in Marc's post #1!
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Help in separating a multilingual file

Hello, I have a text file running into around 100 thousand+ lines which has the following rigid structure: Each field is separated by a comma. Some examples are given below: 23,Chinttaman Pagare,चिंतमण पगारे 24, Chinttaman Pateel,चिंतामण पाटल 25, Chinttaman Rout,चिंतामण राऊत 26,... (3 Replies)
Discussion started by: gimley
3 Replies

2. Shell Programming and Scripting

How to generate a csv files by separating the values from the input file based on position?

Hi All, I need help for doing the following. I have a input file like: aaaaaaaaaabbbbbbbbbbbbbbbbbbbb cccbbbbbaaaaaadddddaaaabbbbbbb now I am trying to generate a output csv file where i will have for e.g. 0-3 chars of each line as the first column in the csv, 4-10 chars of the line as... (3 Replies)
Discussion started by: babom
3 Replies

3. Shell Programming and Scripting

AWK separating a file into an array

Is there a way to have awk put successive records into an array in a bash script? I have files that say things like name :title :salary Bob :Instructor :30,000 Joyce :Instructor :30,000 Patrick :Manager :40,000 What I want to do is seperate this file into an array so that... (8 Replies)
Discussion started by: tgidzak
8 Replies

4. Programming

Separating two classes in two files

I have a file Map.hh shown below. I want to put the two classes Phase and Map in two different files Phase.hh and Map.hh. I have forward declaration before the Map class. How can I tackle this situation? ////////////////////////////////////////////////////////////////////////// #ifndef... (3 Replies)
Discussion started by: kristinu
3 Replies

5. Shell Programming and Scripting

Need help separating a file

Hi all, I have a single text file, Contig3.fasta, that looks like this: >NAME1 ACCTGGTA >NAME2 GGTTGGACA >NAME3 ATTTTGGGCCAnd It has about 100 items like this in it. What I would like to do is copy each item into 100 different text files, and have them named a certain way Output... (4 Replies)
Discussion started by: repiv
4 Replies

6. Shell Programming and Scripting

Separating list of input files (*.file) with a comma in bash script

Hi all, I'm trying to get a bash script working for a program (bowtie) which takes a list of input files (*.fastq) and assembles them to an output file (outfile.sam). All the .fastq files are in one folder in my home directory (~/infiles). The problem is that the 'bowtie' requires that... (7 Replies)
Discussion started by: TuAd
7 Replies

7. Shell Programming and Scripting

Separating delimited file by pattern with exclusion list

I have a file with the contents below jan_t=jan;feb_t=feb;mar_t=mar;year=2010 jan_t=null;feb_t=feb;mar_t=mar;year=2010 jan_t=jan;feb_t=feb;mar_t=mar;year=2010 I want to extract out all the fields values ending with "_t" , however, i want to exclude feb_t and mar_t from the results In... (6 Replies)
Discussion started by: alienated
6 Replies

8. Programming

Use of C++ Classes

I was wondering if I could put the section at the beginning rather than at the end before the definition of the class. const REAL ModMisfit::DefMinDT = 0.01; const REAL ModMisfit::DefSigma0 = 0.01; const double ModMisfit::DefDAngSh = 2; const REAL ModMisfit::DefKBeta = 5;... (2 Replies)
Discussion started by: kristinu
2 Replies

9. Shell Programming and Scripting

Merging files into a single tab delimited file with a space separating

I have a folder that contains say 50 files in a sequential order: cdf_1.txt cdf_2.txt cdf_3.txt cdf_3.txt . . . cdf_50.txt. I need to merge these files in the same order into a single tab delimited file. I used the following shell script: for x in {1..50}; do cat cdf_${x}.txt >>... (3 Replies)
Discussion started by: Lucky Ali
3 Replies

10. Shell Programming and Scripting

Separating values from a file and putting to a variable

I am writing into a file testfile.txt values like ./XXXXXXCZ1/tprcm10c.bin ./XXXXXXCZ1_HOT/tprcm09c.bin ./XXXXXXCZ_cold/tprcm05c.bin I want to store the values of tprcm*.bin and XXXXXXCZ* in separate variables Can anybody Pls hlp me out with this ... Thanks (2 Replies)
Discussion started by: ultimatix
2 Replies
Login or Register to Ask a Question