Loops & Variable


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Loops & Variable
# 8  
Old 06-29-2015
This solves my problem
Code:
#!/bin/bash
TMPArray1=( NULL "GGGGGG|TTTTT" "AAAAAA|CCCCCC" "CCAAAA|CCCCAA" )
TMPArray2=( NULL "XXXXX|YYYYYY" "ZZZZZZZ|BBBBBBB" "KKKKKK|LLLLLL" )
TMPArray3=( NULL "SSSSS|FFFFFF" "EEEEEE|JJJJJJ" "PPPPPP|UUUUUU" )
for e in {1..3}
do 
    awk -v"REP=${TMPArray1[$e]}" '{gsub("^.*" REP ".*$","",$0)} 1' 1.$e > 1.Result$e
    awk -v"REP=${TMPArray2[$e]}" '{gsub("^.*" REP ".*$","",$0)} 1' 2.$e > 2.Result$e
    awk -v"REP=${TMPArray3[$e]}" '{gsub("^.*" REP ".*$","",$0)} 1' 3.$e > 3.Result$e
done

However, I really would like to understand why this does not work
Code:
#!/bin/bash
TMPArray1=( NULL "GGGGGG|TTTTT" "AAAAAA|CCCCCC" "CCAAAA|CCCCAA" )
TMPArray2=( NULL "XXXXX|YYYYYY" "ZZZZZZZ|BBBBBBB" "KKKKKK|LLLLLL" )
TMPArray3=( NULL "SSSSS|FFFFFF" "EEEEEE|JJJJJJ" "PPPPPP|UUUUUU" )
for y in {1..3}
do
    for e in {1..3}
    do
        awk -v"REP=eval ${TMPArray[$y][$e]}" '{gsub("^.*" REP ".*$","",$0)} 1' $y.$e > $y.Result$e
    done
done


Last edited by Xterra; 06-29-2015 at 10:01 PM..
# 9  
Old 06-29-2015
Quote:
Originally Posted by Xterra
However, I am curious to know why this does not work
Code:
awk -v"REP=eval ${TMPArray[$y][$e]}" '{gsub("^.*" REP ".*$","",$0)} 1' $y.$e > $y.Result$e

I have told you above that "eval" is a command. Take another command, for example "ls", and ask yourself if you expect

Code:
awk -v"REP=ls ${TMPArray[$y][$e]}" '{gsub("^.*" REP ".*$","",$0)} 1' $y.$e > $y.Result$e

to work. You are aware that -v"REP=<something>" is a declaration of a variable, aren't you? It declares "REP" to hold a certain value inside awk. Would you expect "$x" here:

Code:
x="ls"

to hold a directory listing or rather the string "ls", hmm?

/PS: Is there any reason why you introduce an element "NULL" in every array which you do not use at all?

Code:
...
TMPArray1=( "GGGGGG|TTTTT" "AAAAAA|CCCCCC" "CCAAAA|CCCCAA" )
...
for y in {0..2}

would do the same, no?

I hope this helps.

bakunin

Last edited by bakunin; 06-29-2015 at 10:16 PM..
# 10  
Old 06-29-2015
I could have the following
Code:
#!/bin/bash
TMPArray1=( NULL "GGGGGG|TTTTT" "AAAAAA|CCCCCC" "CCAAAA|CCCCAA" )
for e in {1..3}
do 
    awk -v"REP=${TMPArray1[$e]}" '{gsub("^.*" REP ".*$","",$0)} 1' 1.$e > 1.Result$e
done
 
TMPArray2=( NULL "XXXXX|YYYYYY" "ZZZZZZZ|BBBBBBB" "KKKKKK|LLLLLL" )
for e in {1..3}
do 
    awk -v"REP=${TMPArray2[$e]}" '{gsub("^.*" REP ".*$","",$0)} 1' 2.$e > 2.Result$e
done
 
TMPArray3=( NULL "SSSSS|FFFFFF" "EEEEEE|JJJJJJ" "PPPPPP|UUUUUU" )
for e in {1..3}
do 
    awk -v"REP=${TMPArray3[$e]}" '{gsub("^.*" REP ".*$","",$0)} 1' 3.$e > 3.Result$e
done

But I would like to simplify the whole thing. That's why I was wondering if there was anyway to use a double nested loop
This is not working either
Code:
#!/bin/bash
TMPArray1=( NULL "GGGGGG|TTTTT" "AAAAAA|CCCCCC" "CCAAAA|CCCCAA" )
TMPArray2=( NULL "XXXXX|YYYYYY" "ZZZZZZZ|BBBBBBB" "KKKKKK|LLLLLL" )
TMPArray3=( NULL "SSSSS|FFFFFF" "EEEEEE|JJJJJJ" "PPPPPP|UUUUUU" )
x="eval"
for y in {1..3}
do
    for e in {1..3}
    do
        awk -v"REP=$x ${TMPArray[$y][$e]}" '{gsub("^.*" REP ".*$","",$0)} 1' $y.$e > $y.Result$e
    done
done

but this works
Code:
#!/bin/bash
a='$b'
b="hello world"
x="eval"
for y in {1..3}
do
    for e in {1..3}
    do
        $x echo $a $y [$e]
    done
done


Last edited by Xterra; 06-29-2015 at 11:57 PM..
# 11  
Old 06-30-2015
Quote:
Originally Posted by Xterra
But I would like to simplify the whole thing. That's why I was wondering if there was anyway to use a double nested loop
Good idea.

This is not working either

Quote:
Originally Posted by Xterra
Code:
#!/bin/bash
TMPArray1=( NULL "GGGGGG|TTTTT" "AAAAAA|CCCCCC" "CCAAAA|CCCCAA" )
TMPArray2=( NULL "XXXXX|YYYYYY" "ZZZZZZZ|BBBBBBB" "KKKKKK|LLLLLL" )
TMPArray3=( NULL "SSSSS|FFFFFF" "EEEEEE|JJJJJJ" "PPPPPP|UUUUUU" )

Later on you reference these three arrays as:

Code:
"REP=$x ${TMPArray[$y][$e]}"

Lets put aside the business with awk and the eval for a moment. ${TMPArray[$y][$e]} is supposed to be a two-dimensional array, a matrix, yes? The two variables "$y" and "$e" should hold row and column, correct?

Now look again at how you defined your arrays: this is not one two-dimensional array (like the one i sketched out for your above, in ksh), but its three one-dimensional arrays which just happen to have similar names.

Let us simplify our requirements for a moment. Instead of using complex parameter-passing calls to awk we simply want to print the content of the arrays using a nested loop to generate the indexes. We also simplify the contents of the arrays:

Code:
arr1=( "11" "12" "13" )
arr2=( "21" "22" "23" )
arr3=( "31" "32" "33" )

Now let us construct the outer loop. This one should generate the names of the arrays:

Code:
arr1=( "11" "12" "13" )
arr2=( "21" "22" "23" )
arr3=( "31" "32" "33" )

for row in {1..3} ; do
     echo "arr"${row}
done

You are still with me? Good, now for the second, inner loop to count the colums. Notice that array elements are numbered starting with 0, not 1, therefore we let the for-loop run from 0-2, not 1-3. This way we do not need a NULL element which we would ignore.

Code:
arr1=( "11" "12" "13" )
arr2=( "21" "22" "23" )
arr3=( "31" "32" "33" )

for row in {1..3} ; do
     echo "arr"${row}
done

Code:
arr1=( "11" "12" "13" )
arr2=( "21" "22" "23" )
arr3=( "31" "32" "33" )

for row in {1..3} ; do
     for column in {0..2} ; do
          echo "arr"${row}"[$column]"
     done
done

Notice that this outputs the names of the variables correctly. But we do not want the names, we want the contents. This is why we restart the process of substituting a variables name with its content a second time. We put an "eval" in front of the command and a (protected by a "\") "$"

Code:
arr1=( "11" "12" "13" )
arr2=( "21" "22" "23" )
arr3=( "31" "32" "33" )

for row in {1..3} ; do
     for column in {0..2} ; do
          eval echo "\${arr${row}[$column]}"
     done
done

The only thing left to explain is why the "$" had to be protected: well, picture how the process of substituting variables names with their contents happens:

Code:
eval echo "\${arr${row}[$column]}"     # original line
echo $arr1{[0]}                        # after one processing
echo 11                                # after the second processing

The "$" had to be protected so that it would not be translated in the first processing pass.

I hope this helps.

bakunin
# 12  
Old 06-30-2015
Finally! I have to keep the NULL statement to facilitate the handling of the input files
Code:
#!/bin/bash
TMPArray1=( NULL "GGGGGG|TTTTT" "AAAAAA|CCCCCC" "CCAAAA|CCCCAA" )
TMPArray2=( NULL "XXXXX|YYYYYY" "ZZZZZZZ|BBBBBBB" "KKKKKK|LLLLLL" )
TMPArray3=( NULL "SSSSS|FFFFFF" "EEEEEE|JJJJJJ" "PPPPPP|UUUUUU" )
for y in {1..3}
do
    for x in {1..3}
    do
        eval echo "\${TMPArray${y}[$x]}" > tmp | awk -v "REP=$(awk '{print}' tmp)" '{gsub("^.*" REP ".*$","",$0)} 1' $y.$x > $y.Result$x
    done
rm tmp
done

It still looks horrible but much better than before and working. Anyway to simplify the awk script? If possible, I would like to avoid the 'tmp' file
Thanks!
# 13  
Old 07-01-2015
Given a file, say, tmp, holds
Code:
cat tmp
GGGGGG|TTTTT
AAAAAA|CCCCCC
CCAAAA|CCCCAA
XXXXX|YYYYYY
ZZZZZZZ|BBBBBBB
KKKKKK|LLLLLL
SSSSS|FFFFFF
EEEEEE|JJJJJJ
PPPPPP|UUUUUU

, this might work for you:
Code:
mapfile -tO1 TMPArray <tmp
for y in {1..3}
  do for x in {1..3}    
    do  awk -v"REP=${TMPArray[$x+($y-1)*3]}" '{print "^.*\"" REP "\".*$"}' file
    done 
  done

# 14  
Old 07-01-2015
Quote:
Originally Posted by Xterra
Finally!
I hate to say it, but: not quite.

Programming (writing a script, however small, IS programming) is a complex activity and you need to do it right, the same way you need to establish a protocol.

When establishing a certain method - say, i want to use a Taqman PCR to establish the expression of a certain protein - we need to not only get one good try but we need to work on the method as long as it takes to get good tries reliably. To show that one certain mouse had a higher expression level means nothing - we would need to show that all the knock-down specimen have significantly higher/lower expression levels than the control group, yes? When writing programs you do not stop once it did once what you want but you work until it does what you want even under the most adverse conditions (or nothing at all, but definitely not doing something you do NOT want), you work until you can understand it well and until a certain "elegance" in the way it is written is reached. *)

Quote:
Originally Posted by Xterra
It still looks horrible but much better than before and working. Anyway to simplify the awk script? If possible, I would like to avoid the 'tmp' file
You are right, it looks ugly - or, as i prefer to say, rather "unelegant". The usage of a (completely unnecessary) "tmp"-file is one of the reasons. But let us start at a basic topic:

Code:
for y in {1..3}
do
    for x in {1..3}

Of course for the system it doesn't matter what variable names you use, but YOU - the human reader trying to understand what the program does - it matters big time. Do yourself a favour and give variables a meaningful name: "row" and "column" is more easily understood than "x" and "y". It will not matter how you name variables, but do so in a way you yourself can easily understand its purpose and it possible contents from the name.


Quote:
Originally Posted by Xterra
I have to keep the NULL statement to facilitate the handling of the input files
No! Programming is about keeping order and you should NEVER use such devices. If you want a list of replacements then create a list of replacements, not a list of anything with replacements interspersed. if you need a separate name for a file then create it using a separate variable - there is no shortage of them, for example:

Code:
for outerloop in {1..3} ; do
     for innerloop in {0..2} ; do
          (( filenamepart = innerloop + 1 ))
          eval command \${array${outerloop}[$innerloop]} > file.${filenamepart}.${outerloop}
     done
done

At last, let us talk about the intermediate "tmp" file:
Code:
eval echo "\${TMPArray${y}[$x]}" > tmp | awk -v "REP=$(awk '{print}' tmp)"[...]

First, the pipe is not necessary at all and most probably keeping the line from doing what you want to do. Picture a process (any process - "awk" or any other program) like a garden hose: you pour data into it at the top, and some (processed) data flows out at the bottom. A pipe, to carry on with this picture, means to connect two such garden hoses so that the outflow of one pours into the other.

Some programs take their data from either this input ("stdin") or from files and "awk" is among them. So the following two lines do the same:

Code:
awk '....' inputfile
cat inputfile | awk '...'

"cat" just takes an input file (or "stdin") and spews out what is there. Notice that the following - though not making very much sense - is also doing the same:

Code:
cat inputfile | cat | awk '...'

the first "cat reads from a file and puts its contents to "stdout", which is connected to the "stdin" of the second "cat". This again puts everything going in there to its own "stdout" (we are now through two hoses of the same type) and finally this is connected to "stdin" of awk.

To come back to your line: there is nothing you want to pass through this mechanism to "awk" and therefore this is just redundant. Its just two commands:

Code:
eval echo "\${TMPArray${y}[$x]}" > tmp
awk -v "REP=$(awk '{print}' tmp)"[...]

Now, when you look closely at the first command, what does go into this file "tmp"? In fact, after all the "eval"s and expansions, it is a certain array element, say "GGGGGG|TTTTT". So, effectively you have there a line that says:

Code:
echo "GGGGGG|TTTTT" > tmp

(yes, the values differ for each execution of the loop, but the schema stays the same). You could - if you don't want to convolute your "awk" statement too much, which is a good idea, always do one thing in a step, but do it good - put it in a variable:

Code:
eval replacement=\${TMPArray${y}[$x]}
awk -v "REP=${replacement}"[...]

I hope this helps.

bakunin
__________________
*) this is a difficult topic to explain to a novice and you will probably just have to believe me - at least for the moment. I can tell you that "elegantly written" programs tend to work better and are more easily maintained than "unelegant" ones.
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Bash: How to use read with conditions & loops

Hello, Below I try to control that the input is good an IP : #!/bin/bash cp /home/scripts/choice_interfaces.txt /home/scripts/interfaces.txt chmod 644 /home/scripts/interfaces.txt echo -e "Please enter the network informations into the /etc/network/interfaces file, complete them below... (9 Replies)
Discussion started by: Arnaudh78
9 Replies

2. UNIX for Advanced & Expert Users

Passing variable as input & storing output in other variable

I have a below syntax its working fine... var12=$(ps -ef | grep apache | awk '{print $2,$4}') Im getting expected output as below: printf "%b\n" "${VAR12}" dell 123 dell 456 dell 457 Now I wrote a while loop.. the output of VAR12 should be passed as input parameters to while loop and results... (5 Replies)
Discussion started by: sam@sam
5 Replies

3. UNIX for Dummies Questions & Answers

Bash loops and variable scope

Hi All, I've been researching this problem and I am pretty sure that the issue is related to the while loop and the piping. There are plenty of other threads about this issue that recommend removing the pipe and using redirection. However, I haven't been able to get it working using the ssh and... (1 Reply)
Discussion started by: 1skydive
1 Replies

4. Shell Programming and Scripting

Re-assign variable's value through which FOR LOOP loops

Hi, I've a requirement where I want to re-assign the value in the variable through which FOR LOOP loops. For e.g. Snippet of code --------------- for i in $var do echo $i >> $tempFile var=`echo $another_var | awk -F" " '{print $1}'` done I am re-assigning var so... (2 Replies)
Discussion started by: dips_ag
2 Replies

5. Shell Programming and Scripting

sed -i '7 c\$variable' file ....(?!@#$%^&*!)

I have tried everything I can think of to get sed to change line N of a file to the contents of a variable. I have Googled the Internet, and I find lots of people telling how to use variables with the "Substitute" command, but no one telling how to use variables with the "Change" command. I... (4 Replies)
Discussion started by: Mr.Lauren
4 Replies

6. UNIX for Dummies Questions & Answers

Passing KSH variable to AWK with two loops

Hi, I need some help to figure out why an outer for loop KSH variable does not decode in AWK but inner for loop does. Below is my code, If I hard code variable 'SUBSEQ' in AWK it works but if I try to pass the SUBSEQ from KSH, it does not and when I pass the variable 'NAM' from KSH it works: I... (1 Reply)
Discussion started by: chowdhut
1 Replies

7. Shell Programming and Scripting

Insert a line including Variable & Carriage Return / sed command as Variable

I want to instert Category:XXXXX into the 2. line something like this should work, but I have somewhere the wrong sytanx. something with the linebreak goes wrong: sed "2i\\${n}Category:$cat\n" Sample: Titel Blahh Blahh abllk sdhsd sjdhf Blahh Blah Blahh Blahh Should look like... (2 Replies)
Discussion started by: lowmaster
2 Replies

8. Shell Programming and Scripting

scripting headache... loops & variables

Surely there's an easier way to do this, lets see if anyone knows! I am new to scripting so go easy on me! I have the following script and at the moment it doesn't work and I believe the problem is that I am using a while loop within a while loop. When I run the script using sh -x I can see... (6 Replies)
Discussion started by: StevePace
6 Replies

9. Shell Programming and Scripting

search & replace in variable

Can I use search & replace in any variable? Suppose I have one variable named var1 which holds value "abcabc" I need to search 'a' in var1 and want to replace with 'x' like 'xbcxbc'. Is it possible? Can you provide me an example? Malay (3 Replies)
Discussion started by: malaymaru
3 Replies

10. Shell Programming and Scripting

korn shell "loops & arrays"

Hi, I am trying to write a script which will loop until a certain action has been performed. I have two files i would like to compares. For example: file1 has a list of user ids (about 900) from the company's e-mail server. file2 has a list of user ids (about 50 or so) from... (7 Replies)
Discussion started by: muzica
7 Replies
Login or Register to Ask a Question