Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

need to pass parameters to working and tested awk script

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting need to pass parameters to working and tested awk script
# 1  
Old 11-02-2010
need to pass parameters to working and tested awk script
I have a working and tested AWK script that removes duplicates from an input file and generates an output file without the duplicates.

I had help from my other post to develop it:

https://www.unix.com/shell-programmin...st-column.html

I want to now make this working AWK script more dynamic and execute with parameters from a shell script (a .sh file).


My UNIX Directory

/home/usr/script

This directory contains:
  • rem_dups.awk (this is the functional awk script)
  • dups_file.txt (this file contains duplicate records)
  • rem_dups.sh (the shell script file that I want to use to execute the awk script.)


CONTENTS of rem_dups.awk (this is the functional AWK script)
Code:
#!/bin/sh

awk '{split($NF,a,"_"); key=$1;site=a[3];keysite=key "_" site;
if (b[keysite]<=a[4]a[5]) {b[keysite]=a[4]a[5];c[keysite]=$0;}}
END{for (i in b) print c[i]}' dups_file.txt


CONTENTS of dups_file.txt (this is the file to be input into rem_dups.awk, it contains duplicates)
Code:
1238646 ,QO,SO,IN,PA,PO,SH,BL,DO,IS file_937_20101024_065520.txt
1239560 ,QO,SO,IN,PA,PO,SH,BL,DO,IS file_937_20101024_065520.txt
1240650 ,QO,SO,IN,PA,PO,SH,BL,DO,IS file_937_20101024_065520.txt

1238646 ,QO,SO,IN,PA,PO,SH,BL,DO,IS file_937_20101025_054320.txt
1239560 ,QO,SO,IN,PA,PO,SH,BL,DO,IS file_937_20101025_054320.txt
1240650 ,QO,SO,IN,PA,PO,SH,BL,DO,IS file_937_20101025_054320.txt

CONTENTS of rem_dups.sh (this is the file that I want to use to pass parameters to the AWK script)(this file DOES NOT WORK)
Code:
#!/usr/bin/sh
#--------------------------------------------------------------------- 
# Program ....... rem_dups.sh
# Function ...... removes duplicates from EDI files 
# Developer ..... script_op2a 
# Date .......... November 2 2010 
# Parameters .... $1 = Position of key column in input file (Required)
#                        $2 = Unix Script directory (Required)
#                        $3 = Input file name of file to remove duplicates from (Required)
#        
# Dependencies .. None
# 
# Notes: This program was built on awk code.
#
# 1) The position of the key column valid values are $1 and greater (the $1 in this case would be the AWK $1 or $2 ect..)
# 2) The name of the UNIX directory where the input file is located
# 3) Input file name must be the name of the file containing duplicate records

pos=$1
filedir=$2
filename=$3

awk '{FS="";split($NF,a,"_"); key=pos;site=a[3];keysite=key "_" site;
if (b[keysite]<=a[4]a[5]) {b[keysite]=a[4]a[5];c[keysite]=$0;}}
END{for (i in b) print c[i]}' filedir "/" filename

UNIX COMMAND LINE

From the UNIX command line I want to execute the shell like this
Code:
./rem_dups.sh $2 /home/usr/script/ dups_file.txt > no_dups.txt

This should remove the duplicates in the file based on the AWK $2 second field and the $NF (last field as shown in the AWK code) however
should the file change I would be able to specify AWK $1 or or AWK $3 as the key column for the AWK script in the shell script.

It's confusing because the shell accepts parameters like $1 $2 ect, and AWK uses $1 $2 etc for the fields of the input file.

I want to say with the shell parameters, that OK, now I choose AWK field $1 or no, now I choose AWK field $2 to insert into the value for the variable "key" in the awk script.

I also want to specify the directory and filename of the input file that contains the duplicates to be removed.
Last edited by Franklin52; 11-03-2010 at 09:00 AM.. Reason: Please use code tags
# 2  
Old 11-02-2010
Assuming you have a modern awk (nawk on Sun) this should work. I've added some 'fluff' to illustrate how to capture the command line parameters and use them in your script (don't mean to imply that you don't know how, but thought it made for a more complete example).

Code:
#!/usr/bin/env ksh

pos=$1
infile="$2/$3"
outfile=$infile.new

if [[ ! -r $infile ]]
then
        echo "file is not readable: $infile"
        exit 1
fi

# pass the key position using -v
awk -v key_col="$pos" '
        {
                split($NF,a,"_"); 
                key=$(key_col);   # this should be the only internal change (not needed -- see 2nd example)
                site=a[3];
                keysite=key "_" site;
                if (b[keysite]<=a[4]a[5]) 
                {
                        b[keysite]=a[4]a[5];
                        c[keysite]=$0;
                }
        }
        END{
                for( i in b ) 
                        print c[i];
        }' <$infile >$outfile

CAUTION: I've not tested this.

Since you only use the variable key to construct keysite, you could eliminate the first assignment and just code this:
Code:
keysite=$(key_col) "_" site;

Hope this helps get you started.
This User Gave Thanks to agama For This Post:
script_op2a
# 3  
Old 11-03-2010
Hello,

Thank you, I am now testing your file.

I'm trying to run the it from the UNIX Command Line using both the following ways:

Code:
./rem_dups.sh "$1" /home/usr/script dups_file.txt

Code:
./rem_dups.sh $1 /home/usr/script dups_file.txt



Command Line Error
Code:
ksh: 1: parameter not set

----------------------------------------------------------------------------------------------------------------------------------------------------------------
Is it the way I am passing the parameters?

I tried changing the 1st line in the .sh file to both:

Code:
#!/usr/bin/sh

Code:
#!/usr/bin/ksh

with the same result,
Code:
ksh: 1: parameter not set

---------- Post updated at 11:41 AM ---------- Previous update was at 11:34 AM ----------

I need to use the sh environment.

So can we continue to make it work using

Code:
#!/usr/bin/sh

as the first line?

---------- Post updated at 12:23 PM ---------- Previous update was at 11:41 AM ----------

The most progress I have made so far is changing what I enter on the UNIX Command Line to:

Code:
./rem_dups.sh '$1' /home/usr/script dups_file.txt

Code:
# - I added single quotes because I 
# want the AWK script to literally receive the value $1 (that is - the $ sign and the number 1) so that it looks at the 1st field of the infile.

I get this error on the command line:

Code:
awk: Field $$1) is not correct
 The input line number is 1.
 The source line number is 4.

Last edited by script_op2a; 11-03-2010 at 01:34 PM..
# 4  
Old 11-03-2010
Code:
./rem_dups.sh 1 /home/usr/script dups_file.txt

This User Gave Thanks to vgersh99 For This Post:
script_op2a
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Pass Parameters to awk command

I need to pass values at runtime for the below awk command where l is the length and partial.txt is the file name. awk -v l=285 '{s="%-"l"s\n";printf(s,$0);}' partial.txt > temp1.txt; (5 Replies)
Discussion started by: Amrutha24
5 Replies

2. Shell Programming and Scripting

Bash- Command run from script does not pass full parameters with spaces inside

There's a JavaScript file that I call from command line (there's a framework) like so: ./RunDiag.js param1:'string one here' param2:'string two here' I have a shell script where I invoke the above command. I can run it in a script as simple as this #!/bin/bash stuff="./RunDiag.js... (4 Replies)
Discussion started by: AcerAspirant
4 Replies

3. Shell Programming and Scripting

Need to pass parameters to an "at" script...

but the man page seems to not have that feature. Am I missing something? This works: $ foo.sh bar abcThis doesn't: $ at now -f foo.sh bar abcTIA, Ron (2 Replies)
Discussion started by: RonJohn
2 Replies

4. Emergency UNIX and Linux Support

Pass two parameters

Hi I have a batch file aaa.exe which needs two input parameters: Usually the command's format likes aaa 555 10000 But I want to use parameters to do it. aaa $1 $2 These two parameters come from a text file list.txt 41800497 41801375 41814783 41816135 41814930 41816135 41819987 41820843... (4 Replies)
Discussion started by: zhshqzyc
4 Replies

5. Shell Programming and Scripting

pass shell parameters to awk does not work

Why does this work for myfile in `find . -name "R*VER" -mtime +1` do SHELLVAR=`grep ^err $myfile || echo "No error"` ECHO $SHELLVAR done and outputs No error err ->BIST Login Fail 3922 err No error err ->IR Remote Key 1 3310 err But... (2 Replies)
Discussion started by: alan
2 Replies

6. Shell Programming and Scripting

call another shell script and pass parameters to that shell script

Hi, I basically have 2 shell scripts. One is a shell script will get the variable value from the user. The variable is nothing but the IP of the remote system. Another shell script is a script that does the job of connecting to the remote system using ssh. This uses a expect utility in turn. ... (2 Replies)
Discussion started by: sunrexstar
2 Replies

7. Shell Programming and Scripting

want to pass parameters to awk script from shell script

Hello, I have this awk script that I want to execute by passing parameters through a shell script. I'm a little confused. This awk script removes duplicates from an input file. Ok, so I have a .sh file called rem_dups.sh #!/usr/bin/sh... (4 Replies)
Discussion started by: script_op2a
4 Replies

8. Shell Programming and Scripting

How to pass parameters transparently into a sub script

Hi, I am trying to write a script like this: #!/bin/ksh #script name: msgflow #The awk commands for Solaris and Linux are incompatible if ] then msgflow-solaris $* elif ] then msgflow-linux $* fi This script is shared by a file system which is visible to both... (3 Replies)
Discussion started by: danielnpu
3 Replies

9. Shell Programming and Scripting

How to pass parameters to an awk file?

I have an awk file where I need to pass a filename and a value as a parameter from a sh script. I need to know how to pass those values in the sh script and how to use the same in the awk file. Thanks in advance!!! Geetha (3 Replies)
Discussion started by: iamgeethuj
3 Replies

10. UNIX for Dummies Questions & Answers

How to pass two or more parameters to the main in shell script

Hey Guys from the below script what I understood is we are sending the the first parameter as input to the main (){} file main > $LOGFILE 2>&1 but can we send two or three parameter as input to this main file as main > $LOGFILE 2>&1 2>&2 like this Can any one plz help I need to writ a... (0 Replies)
Discussion started by: pinky
0 Replies
Login or Register to Ask a Question