Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Need help on awk script

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Need help on awk script
# 1  
Old 05-03-2013
HP Need help on awk script
Hi,

I am a new member and this is my very first post on this forum. I am naive to the world of Unix, so my apologies for any stupid questions Smilie

I am trying to manipulate a csv file, say input_file.csv. One of the columns in this file ($8 = Group No.) contains some codes (X codes). This is big file containing some 20,000 rows.

I have a list (say list name is no_x_code_list) of X codes (they are some 20-25 codes) that need not be processed, means they needs to be removed from input_file.csv.

I am thinking of an awk script that looks something like this:

Code:
  awk 'BEGIN{ FS=OFS=":"} $8 not in no_x_code_list {print $0}' input_file.csv

But I am not sure how to implement the bold portion of the script above. Any idea? Or any other way to achieve this?

regards,
juzz4fun
Last edited by Scott; 05-03-2013 at 03:27 PM.. Reason: Code tags
# 2  
Old 05-03-2013
Let us say if you have list of X codes in file: no_x_code_list

Assuming fields in your file: input_file.csv are separated by colon :, you could do something like:
Code:
awk -F':' 'NR==FNR{A[$1];next}!($8 in A)' no_x_code_list input_file.csv

This User Gave Thanks to Yoda For This Post:
juzz4fun
# 3  
Old 05-03-2013
Thank you very much, Yoda.
I forgot to mention that the list I am talking about is nothing but a Unix variable below. Sorry for that... Smilie
Your solution would work perfectly if X codes are stored in a separate file.

Code:
set no_x_code_list=("X6301" \
               "X6315" \
               "X6314" \
               "X6344" \
               "X6345" \
               "X7770" \
               "X8567" \
               "X8580" \
               "X8588" \
               "X9100" \
               "X9101" \
               "X9105" \
               "X9110" \
               "X9098")

And in the same script, I am writing the awk script.
Last edited by Scott; 05-03-2013 at 03:28 PM.. Reason: Please use code tags
# 4  
Old 05-03-2013
I guess you are using tcsh which I am not much familiar with.

I would suggest to create an associative array by visiting each elements in the variable using for loop and perform the comparison using awk

Or else you can keep X codes in a file and use the code I posted.
# 5  
Old 05-03-2013
echo "$no_x_code_list" is a list.
Not nice for awk. Convert to a file format and feed that to awk.
awk reads from stdin when the filename is '-'.
Code:
echo "$no_x_code_list" | tr ' ' '\n' | awk -F':' 'NR==FNR{A[$1];next}!($8 in A)' - input_file.csv

You need a Posix awk or GNU awk or nawk.
This User Gave Thanks to MadeInGermany For This Post:
juzz4fun
# 6  
Old 05-03-2013
Thanks MadeInGermany.
I tried to embed this logic in my current script, but it is giving me hard time Smilie

Used nawk instead of simple awk.

Code:
echo "$no_x_code_list" | tr ' ' '\n' | nawk -F':' 'NR==FNR{A[$1];next}!($8 in A) print $0' - input_file.csv > $output_file

Getting below error:

Code:
nawk: syntax error at source line 1
 context is
        NR==FNR{A[$1];next}!($8 in A) >>>  print <<<  $0
nawk: bailing out at source line 1

Will the syntax changes if I use nawk?
Can you explain me how it works what you said? I mean, how does awk handles two inputs - one from echo and other one from .csv file?
I am thinking to start reading about these stuffs soon.
Last edited by Scott; 05-04-2013 at 02:23 AM.. Reason: Please use code tags
# 7  
Old 05-03-2013
awk is a series of
condition {action code}
so it is
!($8 in A) {print $0}
not
!($8 in A) print $0!
At the end of the script-line a missing action code defaults to {print $0}.
A missing condition is "always true" and the action code can have if statements, so you can also write it as
{ if (!($8 in A)) print $0 }.
Last edited by MadeInGermany; 05-03-2013 at 10:11 PM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Shell script to call and sort awk script and output

I'm trying to create a shell script that takes a awk script that I wrote and a filename as an argument. I was able to get that done but I'm having trouble figuring out how to keep the header of the output at the top but sort the rest of the rows alphabetically. This is what I have now but it is... (1 Reply)
Discussion started by: Eric7giants
1 Replies

2. Shell Programming and Scripting

awk script to call another script based on second column entry

Hi I have a text file (Input.txt) with two column entries separated by tab as given below: aaa str1 bbb str2 cccccc str3 dddd str4 eee str3 ssss str2 sdf str3 hhh str1 fff str2 ccc str3 ..... ..... ..... (1 Reply)
Discussion started by: my_Perl
1 Replies

3. UNIX for Dummies Questions & Answers

Passing shell script parameter value to awk command in side the script

I have a shell script (.sh) and I want to pass a parameter value to the awk command but I am getting exception, please assist. diff=$1$2.diff id=$2 new=new_$diff echo "My id is $1" echo "I want to sync for user account $id" ##awk command I am using is as below cat $diff |... (1 Reply)
Discussion started by: Sarita Behera
1 Replies

4. Post Here to Contact Site Administrators and Moderators

Unable to pass shell script parameter value to awk command in side the same script

Variable I have in my shell script diff=$1$2.diff id=$2 new=new_$diff echo "My id is $1" echo "I want to sync for user account $id" ##awk command I am using is as below cat $diff | awk -F'~' ''$2 == "$id"' {print $0}' > $new I could see value of $id is not passing to the awk... (0 Replies)
Discussion started by: Ashunayak
0 Replies

5. Shell Programming and Scripting

Calling shell script within awk script throws error

I am getting the following error while passing parameter to a shell script called within awk script. Any idea what's causing this issue and how to ix it ? Thanks sh: -c: line 0: syntax error near unexpected token `newline' sh: -c: line 0: `./billdatecalc.sh ... (10 Replies)
Discussion started by: Sudhakar333
10 Replies

6. Shell Programming and Scripting

Passing awk variable argument to a script which is being called inside awk

consider the script below sh /opt/hqe/hqapi1-client-5.0.0/bin/hqapi.sh alert list --host=localhost --port=7443 --user=hqadmin --password=hqadmin --secure=true >/tmp/alerts.xml awk -F'' '{for(i=1;i<=NF;i++){ if($i=="Alert id") { if(id!="") if(dt!=""){ cmd="sh someScript.sh... (2 Replies)
Discussion started by: vivek d r
2 Replies

7. Shell Programming and Scripting

Help: How to convert this bash+awk script in awk script only?

This is the final first release of the dynamic menu generator for pekwm (WM). #!/bin/bash function param_val { awk "/^${1}=/{gsub(/^${1}="'/,""); print; exit}' $2 } echo "Dynamic {" for CF in `ls -c1 /usr/share/applications/*.desktop` do name=$(param_val Name $CF) ... (3 Replies)
Discussion started by: alexscript
3 Replies

8. Shell Programming and Scripting

Call shell script function from awk script

hi everyone i am trying to do this bash> cat abc.sh deepak() { echo Deepak } deepak bash>./abc.sh Deepak so it is giving me write simply i created a func and it worked now i modified it like this way bash> cat abc.sh (2 Replies)
Discussion started by: aishsimplesweet
2 Replies

9. Shell Programming and Scripting

want to pass parameters to awk script from shell script

Hello, I have this awk script that I want to execute by passing parameters through a shell script. I'm a little confused. This awk script removes duplicates from an input file. Ok, so I have a .sh file called rem_dups.sh #!/usr/bin/sh... (4 Replies)
Discussion started by: script_op2a
4 Replies

10. Shell Programming and Scripting

create a shell script that calls another script and and an awk script

Hi guys I have a shell script that executes sql statemets and sends the output to a file.the script takes in parameters executes sql and sends the result to an output file. #!/bin/sh echo " $2 $3 $4 $5 $6 $7 isql -w400 -U$2 -S$5 -P$3 << xxx use $4 go print"**Changes to the table... (0 Replies)
Discussion started by: magikminox
0 Replies
Login or Register to Ask a Question