Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

ksh - Need Help Reducing Nested Loops

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting ksh - Need Help Reducing Nested Loops
# 1  
Old 10-23-2014
ksh - Need Help Reducing Nested Loops
Hello, I pulled out some old code from an unfinished project the other day and wanted to stream line it better. I know anything beyond a double loop is usually bad practice, and I came up with some logic for later that would no longer require the first loop in the following code that works:

Code:
#!/bin/ksh

for j in $(ls -l ./lists | awk '{ print $9 }');
   do
      while read line;
        do
          A=`echo $line | sed 's/.*://;s/-/ /g;'`;
          B=`echo $A | awk '{ print $1 }' | cut -d "." -f4`;
          C=`echo $A | awk '{ print $2 }' | cut -d "." -f4`;
          D=`echo $A | awk '{ print $1 }' | cut -d "." -f2-`;

          while [ $B -le $C ];
            do
              echo $D.$B >> first_list_parsed (( B++ ));
          done;
      done;

      sleep 1;
done < /path/to/lists/first_list

I've tried changing this to the following, but it writes blank output to the first_list_parsed file:

Code:
#!/bin/ksh

while read line;
     do
          A=`echo $line | sed 's/.*://;s/-/ /g;'`;
          B=`echo $A | awk '{ print $1 }' | cut -d "." -f4`;
          C=`echo $A | awk '{ print $2 }' | cut -d "." -f4`;
          D=`echo $A | awk '{ print $1 }' | cut -d "." -f2-`;

          while [ $B -le $C ];
            do
              echo $D.$B >> first_list_parsed (( B++ ));
          done;

done;< /path/to/lists/first_list

Any suggestions welcome
# 2  
Old 10-23-2014
This is bad practice in far worse ways than a triply nested loop. You are using awk and sed to process individual lines. They are full-fledged languages -- I get the impression this program could be rewritten as one awk call; so this is like running, loading, and quitting a web browser to read each individual word on a web page. Seven times per word. Monstrously inefficient, not how it's intended to be used at all. The use of awk to split columns on single, individual lines is especially troublesome, shell read and text substitution is fully capable of doing that.

ls -l | awk '{ print $9 }' is also dangerous and wrong. Why ask ls for extended information then throw it all away? This will also break apart filenames which contain spaces. for j in ./lists/* does the same thing far more safely and without using ls at all.

Except that you don't appear to be using the value of j anywhere even though you loop over it. What are you trying to do? That part of the loop looks like a complete no-op right now. The actual loop is over /path/to/lists/first_list and nothing else.

All things considered, three loops isn't so bad if written well. Please explain what this code is for and I'll help improve it. The contents of /path/to/lists/first_list would also be extremely helpful to figuring out why your current program isn't working.
Last edited by Corona688; 10-23-2014 at 02:36 PM..
# 3  
Old 10-23-2014
As I mentioned I'm trying to remove the for loop with "j" and "ls -l | awk '{ print $9 }'". Originally I was going to loop through multiple files, but have decided there are better ways to do this.

The lists contain thousands of lines of many different ip ranges. Since the pf firewall does not handle ranges outside CIDR, I'm using sed to remove the text in lines like these that are not ips and drop the hyphen between the ips:

Marchex, Inc:8.20.104.0-8.20.107.255

Then I used the B variable to hold the last octet of the first ip and C to hold the last octet of the second ip. D holds the first three octets of the ip addresses. After that the last while loop writes the individual ip address to a file to be used for a table in pf.

I'm sure there are easier and better ways to do a lot of this. I just thought I'd focus on one thing and not bother anyone with totally overhauling this. However, I'm open to any and all suggestions. As this was my first Ksh, its very convoluted and not where I'd like it to be. Thanks
# 4  
Old 10-23-2014
Ah, so for 8.20.104.0-8.20.107.255 you wanted
Code:
8.20.104.0
8.20.104.1
...
8.20.104.255

?


Code:
OLDIFS="$IFS" ; IFS=".-" # Alter the default splitting in the shell
while IFS=":" read CORP IPS
do
        # Splits on . and -, so we get $1=8, $2=20, $3=104, $4=0, $5=8, ...
        set -- $IPS
        # Print a sequence of numbers, with $1.$2.$3. prepended
        seq -f "$1.$2.$3.%1.0f" $4 $8

        # I'm not sure this is a portable 'seq' option.
        # If it doesn't work,
        # N=$4 ; while [ "$N" -le "$8" ] ; do echo "$1.$2.$3.$((N++))" ; done
done < inputfile
IFS="$OLDIFS"  # Restore splitting to default

And if you wanted to loop over multiple files:

Code:
OLDIFS="$IFS" ; IFS=".-" # Alter the default splitting in the shell

cat inputfolder/* | while IFS=":" read CORP IPS # Read a line splitting on ":" alone
do
        # Splits on . and -, so we get $1=8, $2=20, $3=104, $4=0, $5=8, ...
        set -- $IPS
        # Print a sequence of numbers, with $1.$2.$3. prepended
        seq -f "$1.$2.$3.%1.0f" $4 $8

        # I'm not sure this is a portable 'seq' option.
        # If it doesn't work,
        # N=$4 ; while [ "$N" -le "$8" ] ; do echo "$1.$2.$3.$((N++))" ; done
done
IFS="$OLDIFS" # Restore splitting to default

I think I had a script to convert ranges to CIDR somewhere, though. Probably easy enough to catch the 0-255 case.
Last edited by Corona688; 10-23-2014 at 03:07 PM..
This User Gave Thanks to Corona688 For This Post:
Azrael
# 5  
Old 10-23-2014
@Azrael

The key to improve your ksh script is understanding this part
Quote:
Originally Posted by Corona688
[...] shell read and text substitution is fully capable of doing that.
Otherwise, use only awk. It can read every file without any need for ls, cut, read, echo or any built-in or external shell call.
This User Gave Thanks to Aia For This Post:
Azrael
# 6  
Old 10-23-2014
Had to leave for a long doctor's appointment. That worked perfectly Corona688! Learning IFS is something I've put off for much too long.

I'm going to use the code for individual lists you provided. The code works for calling all files in that directory, but I plan to link this to some lower-level code that initialize them all in separate threads at the same time.

I'll let you know how this turns out. Thank you very much!!!
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Nested Loops for text file

Hi A text file containing data something likeVehicle: BMW Class checkin_note: Tyre : Four path_id : 11 vault_file_size: -1 Vehicle: Toyota Class checkin_note: Tyre : Four path_id : 11 vault_file_size: -1 Vehicle: Chevrolet Class checkin_note: Tyre : Five path_id :... (7 Replies)
Discussion started by: vipinHasija
7 Replies

2. Shell Programming and Scripting

two while nested loops

for server in $(echo `cat /tmp/ScanHosts_${USERSNAME}.TXT`) do for portnumber in $(echo `cat /tmp/ScanPorts_${USERSNAME}.TXT`) do #echo ${server} ${portnumber} ... (3 Replies)
Discussion started by: SkySmart
3 Replies

3. UNIX for Dummies Questions & Answers

Executing nested loops+foreach

It's been a while since I used csh formatting and I am having a little bit of trouble with a few things. Things seem so much easier to execute in Matlab, however I need to do this on the terminal because of the programs I am trying to interact with. So here's what I want to do: I have a file... (0 Replies)
Discussion started by: katia
0 Replies

4. Shell Programming and Scripting

Nested for loops

Greetings All, The following script attempts to enumerate all users in all groups in the group file(GROUP) and echo the following information: GROUP ---> USER The script is as follows: IFS="," for GROUP in `ypcat -k group | cut -d" " -f1` do for USER in `ypcat -k group... (13 Replies)
Discussion started by: jacksolm
13 Replies

5. Shell Programming and Scripting

KSH nested loops?

KSH isn't my strong suit but it's what my company has to offer. I've got a script with two nested loops, a FOR and UNTIL, and that works fine. When I add a CASE into the mix I end up getting "Unexpected 'done' at line xx" errors. Any suggestions on this? for divi in at ce ci cm co de di fl... (9 Replies)
Discussion started by: mrice
9 Replies

6. Shell Programming and Scripting

Nested while loops (ksh scripting)

You can use one while inside another? I made the following script (without really knowing if I can use two while) to get 3 numbers different from each other at random: num1=$(( $RANDOM % 10 )) num2=$num1 while do num2=$(( $RANDOM % 10 )) done num3=$num1 while do while do... (1 Reply)
Discussion started by: ale.dle
1 Replies

7. Shell Programming and Scripting

nested for loops

I need help getting over this bump on how nested for loops work in shell. Say i was comparing files in a directory in any other language my for loop would look like so for(int i=0;to then end; i++) for(int y = i+1; to the end; y++) I can't seem to understand how i can translate that... (5 Replies)
Discussion started by: taiL
5 Replies

8. Shell Programming and Scripting

file reading in nested loops

I have to to read files simultaneously in two nested loops,but am getting error can anyone do the needful. useridFile=userIds.txt fname=kiran.txt exec<$useridFile while read line do echo "User IDs are..$line" USER_ID=$line REMOTE_DIR_LOCATION="/home/test/$USER_ID" SOURCE_DIR=$USER_ID... (1 Reply)
Discussion started by: KiranKumarKarre
1 Replies

9. Shell Programming and Scripting

Grepping within nested for loops

Good morning - I have publication lists from 34 different faculty members. I need to end up with the numbers of publications in common across all 34 faculty. I need to grep person1 (last name) in list2, person1 in list3, person1 in list 4, etc., then person2 in list3, person 2 in list4, etc.,... (2 Replies)
Discussion started by: Peggy White
2 Replies

10. Shell Programming and Scripting

Complex problem about nested for loops

Hey, I'm writing this bash script that will test print me many copies of the same program but with different combos of 4 variables being between 1 and 100. Here's the code: #! /bin/bash x=0 for ((a=1; a < 101; a++)) do for ((b=1; b < 101; b++)) do for ((c=1; c < 101; c++)) do for... (4 Replies)
Discussion started by: Silverlining
4 Replies
Login or Register to Ask a Question