awk call in bash function called with arugments not working, something lost in translation?


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting awk call in bash function called with arugments not working, something lost in translation?
# 1  
Old 02-01-2020
awk call in bash function called with arugments not working, something lost in translation?

Hello,

I have this awk code in a bash script to perform a find and replace task. This finds one unique line in a file and substitutes the found line with a replacement.
Code:
#! /bin/bash

# value determined elsewhere
total_outputs_p1=100
# file being modified
filename='./common_depend/UTILPARAMS.DAT'
# comment lines to ignore
comment_ch="C"
# line being searched for to replace
look_for="      PARAMETER \(SIZEXX = "
# value of replacement line
replace_with="      PARAMETER (SIZEXX = $total_outputs_p1)"
# temporarily add variable values to environment varialbes
export comment_ch  look_for  replace_with

# process file with awk to find $look_for and replace with $replace_with
# if the first character(s) are are a comment, print the unmodified line
# if the line contains the string we are looking for, print the replacement line
# in all other cases, print the unmodified line
awk ' { if( substr($0,1,1) == ENVIRON["comment_ch"] || if( substr($0,1,2) == ENVIRON["comment_ch"] )
           print $0;
        else if($0 ~ ENVIRON["look_for"])
           print ENVIRON["replace_with"];
        else
           print $0;
      } ' $filename > tmp
# overwrite original file
mv tmp $filename

# clear variables set for this call to awk
unset ENVIRON["comment_ch"]
unset ENVIRON["look_for"]
unset ENVIRON["replace_with"]

This works well. It is rather involved for a simple find and replace but there are other conditions to consider. First, the line with $look_for is only unique in live code. The string may also exist in lines that have been commented out. Lines with a leading comment character need to be ignored. Also, $look_for often contains spaces and/or special characters that need to be escaped. I have had allot of trouble defining such values in awk using -v. Exporting the value of the string I am looking for to an environment variable allows me to use test the value in the awk call, including any spaces and necessary escape characters. I unset the environment variables immediately afterwords.

I have a dozen or so of these tasks to perform, so the logical structure is to have this code in a function to which the values for $filename, comment_ch, $look_for, and $replace_with are passed in the call. I have set it up in a function and printed the values from inside the function.

the code looks like,
Code:
#! /bin/bash

# function to call awk and implement find and replace
function find_and_replace () {

   # leading comment character (ignore line and print)
   local f_comment_ch=$1
   # line in file to find and replace
   local f_look_for=$2
   # replacement line
   local f_replace_with=$3
   # original filename
   local f_filename=$4

echo "     in function"
echo "  f_comment_ch: " $f_comment_ch
echo "    f_look_for: " $f_look_for
echo "f_replace_with: " $f_replace_with
echo "    f_filename: " $f_filename
echo ""

   # add variables to ENVIRON
   export f_comment_ch  f_look_for  f_replace_with

   # process file with awk to find $look_for and replace with $replace_with
   awk ' { if( substr($0,1,1) == ENVIRON["comment_ch"] || if( substr($0,1,2) == ENVIRON["comment_ch"] )
              print $0;
           else if($0 ~ ENVIRON["f_look_for"])
              print ENVIRON["f_replace_with"];
           else
              print $0;
         } ' $f_filename > tmp
   # overwrite original file
   mv tmp $f_filename

   # clear variables set for this call to awk
   unset ENVIRON["f_comment_ch"]
   unset ENVIRON["f_look_for"]
   unset ENVIRON["f_replace_with"]

}

# value determined elsewhere
total_outputs_p1=100
# file being modified
filename='./common_depend/UTILPARAMS.DAT'
# comment lines to ignore
comment_ch="C"
# line being searched for to replace
look_for="      PARAMETER \(SIZEXX = "
# value of replacement line
replace_with="      PARAMETER (SIZEXX = $total_outputs_p1)"

# call find and replace
find_and_replace  "$comment_ch"  "$look_for"  "$replace_with"  "$filename"

This correctly prints the argument values from inside the function but the output I get is just a few lines with a single space. This doesn't make any sense. I am guessing that there may be some issue with the local environment inside the subshell for the function or the call to awk, but if ENVIRON["f_look_for"] evaluates to uninitialized, based on the logic in the awk code I would expect the entire unmodified file to be printed since neither the $comment_ch or $look_for would ever be found.

I'm not sure what to try next so I thought I would post.

Thanks,

LMHmedchem
# 2  
Old 02-01-2020
I see a couple of potential issues with this approach.
I still don't understand why you can't use the usual paradigm of passing vars with -v to awk and have to use this somewhat convoluted approach.
Any chance you can attach a sample file?

Last edited by vgersh99; 02-01-2020 at 02:51 PM..
# 3  
Old 02-01-2020
Quote:
Originally Posted by vgersh99
I see a couple of potential issue with this approach.
I still don't understand why you can't use the usual paradigm of passing vars with -v to awk and have to use this somewhat convoluted approach.
Any chance you can attach a sample file?
I will put together a sample directory and post it.

The -v method works for the comments and replace line, no matter if there are special characters or spaces. It is the $look_for line that is the problem. This version of the code does not work. The $look_for line is never found.
Code:
#! /bin/bash

# value determined elsewhere
total_outputs_p1=100
# file being modified
filename='./common_depend/UTILPARAMS.DAT'
# comment lines to ignore
comment_ch="C"
# line being searched for to replace
look_for="      PARAMETER \(SIZEXX = "
# value of replacement line
replace_with="      PARAMETER (SIZEXX = $total_outputs_p1)"

# process file with awk to find $look_for and replace with $replace_with
awk -v comment="$comment_ch" \
    -v find_line="$look_for" \
    -v replace_line="$replace_with" \
' { if( substr($0,1,1) == comment || if( substr($0,1,2) == comment )
       print $0;
    else if($0 ~ find_line)
       print replace_line;
    else
       print $0;
  } ' $filename > tmp
# overwrite original file
mv tmp $filename

If you replace,

else if($0 ~ find_line)

with,

else if($0 ~ / PARAMETER \(SIZEXX = /)

then it will function. This is hard coding specifically for each item to find and so can't be used in a function. I read that the -v option presented difficulties when using characters that need to be escaped and that is when I switched to the temporary environment variables that I read about on a stackexchange post (I think). I don't understand why it works, but not in the function.

I will post some samples in a few minutes. I need another cup of coffee.

LMHmedchem
# 4  
Old 02-01-2020
for one I can see different number of blank spaces in look_for=" PARAMETER \(SIZEXX = " and in the inline version: else if($0 ~ / PARAMETER \(SIZEXX = /)...
# 5  
Old 02-01-2020
There is a syntax error in the awk code:
Code:
if( substr($0,1,1) == comment || if( substr($0,1,2) == comment )

should probably be
Code:
if( substr($0,1,1) == comment || substr($0,1,2) == comment )

# 6  
Old 02-01-2020
Quote:
Originally Posted by vgersh99
for one I can see different number of blank spaces in look_for=" PARAMETER \(SIZEXX = " and in the inline version: else if($0 ~ / PARAMETER \(SIZEXX = /)...
This is probably just a copying error by me. I also fixed the error noted by Scrutinizer. I added the || because some comments are two characters.

I have attached a .zip with test files. The attachment has a directory with the script set_size_defs.sh and a sub-directory with the two files that are modified in this version. There is also a second sub-directory with a coppies of the files being modified as they are changed when the script is run.

This is working now, though I am not entirely sure what I fixed.

This is the working version of the script,
Code:
#! /bin/bash

# script to find lines and replace, lines commented out are ignored

# function to call awk and implement find and replace
function find_and_replace () {

   # comment character, lines to ignore and print with a leading comment
   local f_comment_ch=$1
   # line in file to find and replace
   local f_look_for=$2
   # replacement line
   local f_replace_with=$3
   # original filename
   local f_filename=$4

   # make sure the file exists
   if [ ! -f "$f_filename" ]; then
      echo ""
      echo $f_filename
      echo "cannot be found"
      echo ""
      exit 1
   fi

   # test print
   echo "   in function"
   echo "  f_comment_ch: " $f_comment_ch
   echo "    f_look_for: " $f_look_for
   echo "f_replace_with: " $f_replace_with
   echo "    f_filename: " $f_filename
   echo ""

   # add variables to ENVIRON
   export f_comment_ch  f_look_for  f_replace_with

   # process each file with awk to find and replace
   awk ' { if( substr($0,1,1) == ENVIRON["f_comment_ch"] ||
               substr($0,1,2) == ENVIRON["f_comment_ch"] )
              print $0;
           else if($0 ~ ENVIRON["f_look_for"])
              print ENVIRON["f_replace_with"];
           else
              print $0;
         } ' $f_filename > tmp
   # overwrite original file
   mv tmp $f_filename

   # clear variables set for this call to awk
   unset f_comment_ch
   unset f_look_for
   unset f_replace_with

}


#----------------------------------------------------------------------------------------
# this value is calculated by other code
total_outputs='2099'
# add 1 to total outputs for the rest of the values
total_outputs_p1=$(($total_outputs + 1))

#----------------------------------------------------------------------------------------
# replace value of SIZEXX in src_common_depend/UTILPARAMS.DAT

# file being modified
filename='./common_depend/UTILPARAMS.DAT'
# comment lines to ignore
comment_ch="C"
# line being searched for to replace
look_for="      PARAMETER \(SIZEXX = "
# value of line being replaced
replace_with="      PARAMETER (SIZEXX = $total_outputs_p1)"

# call find and replace
find_and_replace  "$comment_ch"  "$look_for"  "$replace_with"  "$filename"


#----------------------------------------------------------------------------------------
# replace value of LASTINDEX in ./common_depend/sizedefs.h

# file being modified
filename='./common_depend/sizedefs.h'
# comment lines to ignore
comment_ch='//'
# line being searched for to replace, may contain escape characters
look_for="\#define LASTINDEX"
# value of line being replaced, characters do not need to be escaped here
replace_with="#define LASTINDEX   $total_outputs"

# call find and replace
find_and_replace  "$comment_ch"  "$look_for"  "$replace_with"  "$filename"

#----------------------------------------------------------------------------------------
echo "finished find and replace"
echo ""

This works and is quite fast. It will accommodate comments that are either 1 or 2 characters but the comments have to be at the beginning of the line unless I add code to trim leading whitespace. It doesn't matter if the replacement line has spaces or escaped characters.

I could go with this unless someone suggests the method is a bad idea.

LMHmedchem
This User Gave Thanks to LMHmedchem For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Call user defined function from awk

My requirement is to call function ("fun1") from awk, and print its returned value along with $0. fun1() { t=$1 printf "%02d\n", $t % 60; } echo "Hi There 23" | awk '{print $0; system(fun1 $3)}' Any suggestions what to be modified in above code to achieve requirement.. (5 Replies)
Discussion started by: JSKOBS
5 Replies

2. Shell Programming and Scripting

Bash script from makefile - it is called each time i call make

I've created a tag in the makefile: mytag: $(shell ${PWD}/script.sh) When i do: make clean - the script is executed When i perform make or make mytag the script is again executed with the output: make: Nothing to be done for mytag What i want ? I want script.sh to be executed only... (0 Replies)
Discussion started by: Pufo
0 Replies

3. Shell Programming and Scripting

Ceil not working as function in awk statement

Hi, I have the following code in which i am trying to find ceil of 10th & 11th fields. For finding ceil i have a function in the awk statement. When i test it for some values say on command line it gives correct response(say $10=0 & $11=750). But when the same value occurs in a file having more 3... (5 Replies)
Discussion started by: siramitsharma
5 Replies

4. Shell Programming and Scripting

error when call function in bash script

Dear all, Could you please advice as I when call function i found the following error " refills: command not found" note that refills is function name. following also the function and how i call it function refills { echo "formatting refills and telepin" >> $log awk -F,... (20 Replies)
Discussion started by: ahmed.gad
20 Replies

5. Shell Programming and Scripting

Return a value from called function to the calling function

I have two scripts. script1.sh looks -------------------------------- #!/bin/bash display() { echo "Welcome to Unix" } display ----------------------------- Script2.sh #!/bin/bash sh script1.sh //simply calling script1.sh ------------------------------ (1 Reply)
Discussion started by: mvictorvijayan
1 Replies

6. Shell Programming and Scripting

awk , function call problem

#!/bin/bash awk ' function ad(t,r){ return (t+r); } BEGIN{ print ad(5,3); } { print ad(5,3); } ' Doesn't print anything for the last print ad(5,3); (6 Replies)
Discussion started by: cola
6 Replies

7. Infrastructure Monitoring

diffrence between method call and function call in perl

Hello, I have a problem with package and name space. require "/Mehran/DSGateEngineLib/general.pl"; use strict; sub System_Status_Main_Service_Status_Intrusion_Prevention { my %idpstatus; my @result; &General_ReadHash("/var/dsg/idp/settings",\%idpstatus); #print... (4 Replies)
Discussion started by: Zaxon
4 Replies

8. Shell Programming and Scripting

Bash: how to call function having it's name in variable?

Hello. Looking for a method of modularizing my bash script, I am stuck with such a problem. For example, I have: MODULE_NAME="test" FUNCTION_NAME="run" How do I can a function with name test_run? (4 Replies)
Discussion started by: FractalizeR
4 Replies

9. UNIX for Dummies Questions & Answers

How to call a local function within Awk

Hi, I have the following statement which parses a string for me and prints it out: l_comp="dc000.runksh.test.ksh| $g_sql/dc0000.runksh_test.sql|new.dat|control.ctl" echo $l_comp | awk -F"|" '{ for ( i = 1; i <= NF; i++) { print $i; } } ' Rather then printing the data, I would like to... (5 Replies)
Discussion started by: CAGIRL
5 Replies

10. Shell Programming and Scripting

Passing global variable to a function which is called by another function

Hi , I have three funcions f1, f2 and f3 . f1 calls f2 and f2 calls f3 . I have a global variable "period" which i want to pass to f3 . Can i pass the variable directly in the definition of f3 ? Pls help . sars (4 Replies)
Discussion started by: sars
4 Replies
Login or Register to Ask a Question