Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Loop through variables

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Loop through variables
# 1  
Old 08-03-2013
Loop through variables
I am pretty new to Unix. Trying to pick up some slack while a coworker is out on vacation.

Basically the script is working fine however when I go through the testing phase and have to make mods it is a pita.

Here is an example of what I have

Code:
#!/bin/ksh
if [ ! -s "${DATAPATH}/${src_file_1}.csv" ]
then
    echo "${src_file_1}.csv is missing" >> ${file_log}
    fi

That is a small piece. There are several other areas alot more complex. There are 36 csv files that we are loading into a reporting website every quarter. Each variable ties to a file name. That being said we have variable names from src_file_1 through src_file_36. My thought is that I could generate a loop that would change the variable name with each cycle.

Thanks in advance for any insight.

---------- Post updated at 09:24 AM ---------- Previous update was at 07:28 AM ----------

Got it figured out...

Code:
for i in ${src_file_1} ${src_file_2} ${src_file_3} ${src_file_4} ${src_file_5} ${src_file_6} \
${src_file_7} ${src_file_8} ${src_file_9} ${src_file_10} ${src_file_11} ${src_file_12} \
${src_file_13} ${src_file_14} ${src_file_15} ${src_file_16} ${src_file_17} ${src_file_18} \
${src_file_19} ${src_file_20} ${src_file_21} ${src_file_22} ${src_file_23} ${src_file_24} \
${src_file_25} ${src_file_26} ${src_file_27} ${src_file_28} ${src_file_29} ${src_file_30} \
${src_file_31} ${src_file_32} ${src_file_33} ${src_file_34} ${src_file_35} ${src_file_36} 
 
do 
if [ ! -s "${DATAPATH}/${i}.csv" ]
then
    echo "${i}.csv is missing" >> ${file_log}
    fi
    
done

# 2  
Old 08-03-2013
That's painful. See if you have "seq" command in your system:

Code:
$ 
$ seq 1 10
1
2
3
4
5
6
7
8
9
10
$

If you do, then it could be used in a "for" loop:

Code:
for i in $(seq 1 36)
do
  echo "src_file_$i"
  # perform the test
done

Or you can always go for the plain old-fashioned "while" loop:

Code:
first=1
last=36
while [[ $first -le $last ]]
do
  echo "src_file_$first"
  # perform the test and then increment the iterator variable
  (( first += 1 ))
done

This User Gave Thanks to durden_tyler For This Post:
biobill
# 3  
Old 08-03-2013
Quote:
Originally Posted by durden_tyler
That's painful. See if you have "seq" command in your system:
I tested seq and I dont have it so I would have to use the old school method

Code:
for i in $(seq 1 36)
do
  echo "src_file_$i"
  # perform the test
done

So in this example which I think will work, but confuses me a bit.

How do i get
Code:
echo "src_file_$i"

to work in a test like this

Code:
if [ ! -s "${DATAPATH}/${src_file_1}.csv" ]

So how do I go about replacing ${src_file_1} with echo "src_file_$i" for the test. Hope that makes sense.

This would be relevant for either the "for" or "do while" methods

---------- Post updated at 01:04 PM ---------- Previous update was at 10:39 AM ----------

I am not one to sit idle especially when I am trying to figure something out so please look at the following:

Code:
Test: eval echo \$${src_file}
the_right_text
test: src_file_name=`eval echo \$${src_file}`
test: echo ${src_file_name}
29199{src_file}

so why when I manually do the eval it comes up with the right text, but when assigning the eval to a variable and then echo it it comes out wierd?
# 4  
Old 08-03-2013
Quote:
Originally Posted by biobill
...
Code:
Test: eval echo \$${src_file}
the_right_text
test: src_file_name=`eval echo \$${src_file}`
test: echo ${src_file_name}
29199{src_file}

so why when I manually do the eval it comes up with the right text, but when assigning the eval to a variable and then echo it it comes out wierd?
I am unable to reproduce your test in my system. This looks like the output of some shell script. I'll need to look at the script to figure out what's going on.

However, what's happening at the end is that the value of the variable "src_file_name" is the following two values appended to each other in the order shown:
(1) the value of $$
(2) the constant text "{src_file}"

Now, $$ is a built-in shell variable which holds the value of the PID, the process id. You can check the PID of each process in your shell by executing the following command and looking at the value in the 2nd column:

Code:
ps -ef

You can also check the value of your current shell PID:

Code:
echo $$

And view it and its descendent processes in the process list:

Code:
ps -ef | grep <the_number_you_saw_for_$$_above>

As for your problem, you are creating variable names within the loop, and you want to access their values for your file names. There is a construct "${!variable}" for that.

A simple test follows. I create and assign a variable and echo its value:

Code:
$ 
$ pointer_to_val=123456
$ echo $pointer_to_val
123456
$

Now, I assign the text "pointer_to_val" to another variable and echo it:

Code:
$ 
$ val=pointer_to_val
$ echo $val
pointer_to_val
$ 
$

So far, everything is as expected.

But in the last echo command above, let's say I want to see the value of pointer_to_val, i.e. I want to see the value 123456 that pointer_to_val points to.

To do that, I'd use ${!variable} construct:

Code:
$ 
$ echo ${val}
pointer_to_val
$ 
$ echo ${!val}
123456
$

This idea could be incorporated in your script as follows:

Code:
$ 
$ cat -n check_for_files_1.sh
     1    #!/usr/bin/ksh
     2    DATAPATH="/my/data/dir"
     3    src_file_1=file_one.txt
     4    src_file_2=file_two.sql
     5    src_file_3=file_three.py
     6    src_file_4=file_four.java
     7    src_file_5=file_five.c
     8    first=1
     9    last=5
    10    while [[ $first -le $last ]]
    11    do
    12      var=src_file_$first
    13      file=${!var}
    14      echo "${DATAPATH}/${file}"
    15      (( first += 1 ))
    16    done
$ 
$ 
$ . check_for_files_1.sh
/my/data/dir/file_one.txt
/my/data/dir/file_two.sql
/my/data/dir/file_three.py
/my/data/dir/file_four.java
/my/data/dir/file_five.c
$ 
$ 
$

And if the ${!variable} construct is not supported in your shell, you could use the old-fashioned eval construct:

Code:
$ 
$ cat -n check_for_files.sh
     1    #!/usr/bin/ksh
     2    DATAPATH="/my/data/dir"
     3    src_file_1=file_one.txt
     4    src_file_2=file_two.sql
     5    src_file_3=file_three.py
     6    src_file_4=file_four.java
     7    src_file_5=file_five.c
     8    first=1
     9    last=5
    10    while [[ $first -le $last ]]
    11    do
    12      var=src_file_${first}
    13      file=$(eval "echo \$$var")
    14      echo "${DATAPATH}/${file}"
    15      (( first += 1 ))
    16    done
$ 
$ 
$ . check_for_files.sh
/my/data/dir/file_one.txt
/my/data/dir/file_two.sql
/my/data/dir/file_three.py
/my/data/dir/file_four.java
/my/data/dir/file_five.c
$ 
$

# 5  
Old 08-03-2013
I ended up getting it working. I wish the exclamation point piece of code would work on our box. Would have saved me hours of searching on the net.

This is wheat I ended up with. Might be a little crude but it works.

Code:
i=1
last=36
while [[ $i -le $last ]]
do
src_file="src_file"
ctl="_ctl"
eval src_file_name="\$src_file_$i"
eval ctl_file_name="\$src_file_$i$ctl"
      
    echo "************************************************************************" >> ${cron_log} 
    echo "File Details" >> ${cron_log}
    echo "------------------------------------------------------------------------" >> ${cron_log}
    echo "File Name: ${src_file_name}.csv" >> ${cron_log}
    echo "File Location: ${cron_dir}" >> ${cron_log}
    FILE_LINE_COUNT=`wc -l < ${cron_dir}/${src_file_name}.csv`
    echo "File Line Count:  $FILE_LINE_COUNT"   >> ${cron_log}
    echo " "  >> ${cron_log}
  
    echo "------------------------------------------------------------------------" >> ${cron_log}
    echo "Data Load Details:" >> ${cron_log}
    $ORACLE_HOME/bin/sqlldr ${ORA_DBCU}/${ORA_DBCP} control=${control_dir}/${ctl_file_name}.ctl
    grep -i "logical records read" ${cron_dir}/${ctl_file_name}.log >> ${cron_log}
    grep -i "logical records skipped" ${cron_dir}/${ctl_file_name}.log >> ${cron_log}
    grep -i "successfully loaded" ${cron_dir}/${ctl_file_name}.log >> ${cron_log}tina
    grep -i "not loaded" ${cron_dir}/${ctl_file_name}.log >> ${cron_log}
    echo " " >> ${cron_log}
    err_cnt=`grep "not loaded due to data error"  ${cron_dir}/${ctl_file_name}.log | cut -f3-3 -d" "`
    echo "Records not loaded due to errors: $err_cnt" >> ${cron_log}
    echo " " >> ${cron_log}
    
if ${src_file_name} -eq ${src_file_1}
then
      tot_err_cnt=$err_cnt
else
      tot_err_cnt=$(($tot_err_cnt+$err_cnt))
fi
  i=$(($i+1))
done

This User Gave Thanks to biobill For This Post:
MadeInGermany
# 6  
Old 08-04-2013
That's the correct use of eval. Thanks for posting it!
# 7  
Old 08-04-2013
Quote:
Originally Posted by biobill
I ended up getting it working. I wish the exclamation point piece of code would work on our box. Would have saved me hours of searching on the net.

This is wheat I ended up with. Might be a little crude but it works.

Code:
i=1
last=36
while [[ $i -le $last ]]
do
src_file="src_file"
ctl="_ctl"
eval src_file_name="\$src_file_$i"
eval ctl_file_name="\$src_file_$i$ctl"
      
    echo "************************************************************************" >> ${cron_log} 
    echo "File Details" >> ${cron_log}
    echo "------------------------------------------------------------------------" >> ${cron_log}
    echo "File Name: ${src_file_name}.csv" >> ${cron_log}
    echo "File Location: ${cron_dir}" >> ${cron_log}
    FILE_LINE_COUNT=`wc -l < ${cron_dir}/${src_file_name}.csv`
    echo "File Line Count:  $FILE_LINE_COUNT"   >> ${cron_log}
    echo " "  >> ${cron_log}
  
    echo "------------------------------------------------------------------------" >> ${cron_log}
    echo "Data Load Details:" >> ${cron_log}
    $ORACLE_HOME/bin/sqlldr ${ORA_DBCU}/${ORA_DBCP} control=${control_dir}/${ctl_file_name}.ctl
    grep -i "logical records read" ${cron_dir}/${ctl_file_name}.log >> ${cron_log}
    grep -i "logical records skipped" ${cron_dir}/${ctl_file_name}.log >> ${cron_log}
    grep -i "successfully loaded" ${cron_dir}/${ctl_file_name}.log >> ${cron_log}tina
    grep -i "not loaded" ${cron_dir}/${ctl_file_name}.log >> ${cron_log}
    echo " " >> ${cron_log}
    err_cnt=`grep "not loaded due to data error"  ${cron_dir}/${ctl_file_name}.log | cut -f3-3 -d" "`
    echo "Records not loaded due to errors: $err_cnt" >> ${cron_log}
    echo " " >> ${cron_log}
    
if ${src_file_name} -eq ${src_file_1}
then
      tot_err_cnt=$err_cnt
else
      tot_err_cnt=$(($tot_err_cnt+$err_cnt))
fi
  i=$(($i+1))
done

I'm glad it is working for you, but I'm surprised that the line I marked in red is working. It looks like you intended to execute test with the three operands trying to determine if two filenames were the same. But:
  1. you are missing the test or [ and ] that would be required to make it a test command, and
  2. when comparing strings, you should use = (or == if you were using [[ and ]] instead of test) instead of -eq.
The way you have this if statement written, it is executing the 1st filename with the two arguments -eq and the 2nd filename. If that returns a zero exit status, the then branch is executed; otherwise the else branch is executed.

If the intent was to use one branch the 1st time through the loop, why not test for the value of i being 1 instead of comparing filenames? You might consider changing:
Code:
if ${src_file_name} -eq ${src_file_1}
then
      tot_err_cnt=$err_cnt
else
      tot_err_cnt=$(($tot_err_cnt+$err_cnt))
fi
  i=$(($i+1))

close to the end of your script to:
Code:
    if [ $i -eq 1 ]
    then
        tot_err_cnt=$err_cnt
    else
        ((tot_err_cnt += err_cnt))
    fi
    ((i++))

This User Gave Thanks to Don Cragun For This Post:
biobill
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Help with variables in loop

Hello, please assist: users="test1 test2" keytest1="abcd" keytest2="dbcd" for i in $users do echo "$key${i}" > fileout done So, my objective is to take the current user (ie test1) in loop and echo its associated keyname (ie keytest1) variable to a file. The echo... (2 Replies)
Discussion started by: motdman
2 Replies

2. Shell Programming and Scripting

Need to loop three variables

Hi, I have a out from a command i need to grep a report. For that i need loop 3 variable for that. How i can loop need help. Symmetrix ID : 123456 Masking View Name : Host16 Last updated at : 04:13:06 PM on Thu Mar 17,2011 Initiator Group Name : Host16 Host... (3 Replies)
Discussion started by: ranjancom2000
3 Replies

3. Shell Programming and Scripting

for loop with 2 variables

i am having a file contants as below my requirement is for file in `awk -F "," '{print $8,$9}'` <temp.txt echo "$file" echo "$file">test.txt a=`awk -F "," '{print $1}' `<test.txt b=`awk -F "," '{print $2}' `<test.txt but script reads , i want both the vales for further... (5 Replies)
Discussion started by: sagar_1986
5 Replies

4. Shell Programming and Scripting

Help with a For loop and variables

Greetings. I'm completely new to shell scripting and quickly trying to catch on. Here's my scenario: I have a text file, named ip.txt, containing IP addresses. I want to automatically perform a whois query on each address in the file, search the output for the country, and then put both the IP... (4 Replies)
Discussion started by: molnir
4 Replies

5. Shell Programming and Scripting

Need help in for loop with 2 variables

Hi, I need help on for loop need to add domain and IP In domain list 1.com 2.com 3.com In Ip list 1.1.0.1 1.2.0.1 1.3.0.1 1.com 1.1.0.1 2.com 1.2.0.1 3.com 1.3.0.1 I need to excute this command (4 Replies)
Discussion started by: ranjancom2000
4 Replies

6. Shell Programming and Scripting

Two variables in a for loop

Can we assign two variables in a for loop? I have an input file: 000301|20100502 835101|20100502 I want to read this file in a for loop and assign values to two different variables. I did this now but did not work for STORE,RUNDATE in `awk -F\| '{print $1,$2}' inputfile ... (4 Replies)
Discussion started by: gpaulose
4 Replies

7. Shell Programming and Scripting

SH: two variables in for loop

Hi, say I have a simple sh script like this: for i in a b c d do for j in 1 2 3 4 do echo "$i $j" done done and the output is a 1 a 2 a 3 a 4 b 1 (20 Replies)
Discussion started by: marcpascual
20 Replies

8. Shell Programming and Scripting

Using variables created sequentially in a loop while still inside of the loop [bash]

I'm trying to understand if it's possible to create a set of variables that are numbered based on another variable (using eval) in a loop, and then call on it before the loop ends. As an example I've written a script called question (The fist command is to show what is the contents of the... (2 Replies)
Discussion started by: DeCoTwc
2 Replies

9. Shell Programming and Scripting

Is there a better way I could have run this loop. (For loop with two variables)

Sorry for such a dreadful title, but I'm not sure how to be more descriptive. I'm hoping some of the more gurutastic out there can take a look at a solution I came up with to a problem, and advice if there are better ways to have gone about it. To make a long story short around 20K pieces of... (2 Replies)
Discussion started by: DeCoTwc
2 Replies

10. Shell Programming and Scripting

using variables outside a while loop

Hi Guys, I have a scripts that uses a while loop to read a file and set 2 variables. How can I do this so the variables can be used outside the while loop ? Below is an example....# ./junk2 -m -e user EXE=user master=TRUE DB_TAG=PRODUCT In loop MST=MST=testsvr1:3110 In loop ARGS=... (2 Replies)
Discussion started by: Tornado
2 Replies
Login or Register to Ask a Question