Bash variable expansion in awk script


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Bash variable expansion in awk script
# 1  
Old 06-21-2019
Bash variable expansion in awk script

Hello,
I need to split a file into two of different locations by re-direction in awk.
Code:
cat infle
aaa     1 3
bbb     2 4
aaa     3 3
bbb     4 4
aaa     5 3
bbb     6 4

cat /storage/tmp/group_a.gtf
aaa     1 3
aaa     3 3
aaa     5 3

cat /storage/tmp/group_b.gtf
bbb     2 4
bbb     4 4
bbb     6 4

The path of the resulted files should be passed in with shell variable, and I have tried:
Code:
MY_DIR="/storage/tmp"
awk -v MY_DIR="${MY_DIR}" '{if ($1 == "aaa")  {print $0 >> "${MY_DIR}/group_a.gtf"} else if ($1 == "bbb") {print  $0 >> "${MY_DIR}/group_b.gtf"}}' infle  #A typo was $7=="bbb"
MY_DIR="/storage/tmp";  awk '{if ($1 == "aaa") {print $0 >>  ENVIRON[\"MY_DIR\"]/group_a.gtf"} else if ($1 == "bbb") {print $0  >> "ENVIRON[\"MY_DIR\"]/group_b.gtf"}}' infle
awk -v  awk_var=${MY_DIR} '{if ($1 == "aaa") {print $0 >>  "awk_var/group_a.gtf"} else if ($1 == "bbb") {print $0 >>  "awk_var/group_b.gtf"}}' infle

but none worked with error like: cmd. line:1: (FILENAME=aaa FNR=1) fatal: can't redirect to `awk_var/group_a.gtf' (No such file or directory)

I checked the threads in this forum:
How to pass a variable from shell to awk
Pass shell Variable to awk
but they did not help me out probably due to my expansion is related to file directory. However, the it worked if the path is hard-coded, which is not thru shell variable along with other part of the shell script.
Code:
awk  '{if ($1 == "aaa") {print $0 >> "/storage/tmp/group_a.gtf"} else  if ($1 == "bbb") {print $0 >> "/storeage/tmp/group_b.gtf"}}'  infle

Also It is interesting with following oneliner, in which the expansion worked at every other hit of ENTER. Could not understand why.
Code:
MY_DIR="/storage/tmp"; awk -v awk_var="$MY_DIR" '{print awk_var}'

/storage/tmp

/storage/tmp

/storage/tmp

 ......

1) What are the rules to expand shell (bash) variables (especialy related to directory) inside awk script?
2) Why my last oneliner worked every other time with ENTER? Thanks a lot!

Last edited by yifangt; 06-24-2019 at 12:22 PM..
This User Gave Thanks to yifangt For This Post:
# 2  
Old 06-21-2019
Code:
MY_DIR="/storage/tmp"
awk -v myDir="${MY_DIR}" '{if ($1 == "aaa")  out=(myDir "/group_a.gtf"); else if ($7 == "bbb") out=(myDir "/group_b.gtf"); print >> out;close(out)} ' infile

This User Gave Thanks to vgersh99 For This Post:
# 3  
Old 06-22-2019
Quote:
Originally Posted by yifangt
Hello,
I need to split a file into two of different locations by re-direction in awk.
Code:
cat infle
aaa     1 3
bbb     2 4
aaa     3 3
bbb     4 4
aaa     5 3
bbb     6 4

cat /storage/tmp/group_a.gtf
aaa     1 3
aaa     3 3
aaa     5 3

cat /storage/tmp/group_b.gtf
bbb     2 4
bbb     4 4
bbb     6 4

The path of the resulted files should be passed in with shell variable, and I have tried:
Code:
MY_DIR="/storage/tmp"
awk -v MY_DIR="${MY_DIR}" '{if ($1 == "aaa")  {print $0 >> "${MY_DIR}/group_a.gtf"} else if ($7 == "bbb") {print  $0 >> "${MY_DIR}/group_b.gtf"}}' infle
MY_DIR="/storage/tmp";  awk '{if ($1 == "aaa") {print $0 >>  ENVIRON[\"MY_DIR\"]/group_a.gtf"} else if ($7 == "bbb") {print $0  >> "ENVIRON[\"MY_DIR\"]/group_b.gtf"}}' infle
awk -v  awk_var=${MY_DIR} '{if ($1 == "aaa") {print $0 >>  "awk_var/group_a.gtf"} else if ($1 == "bbb") {print $0 >>  "awk_var/group_b.gtf"}}' infle

but none worked with error like: cmd. line:1: (FILENAME=aaa FNR=1) fatal: can't redirect to `awk_var/group_a.gtf' (No such file or directory)

I checked the threads in this forum:
How to pass a variable from shell to awk
Pass shell Variable to awk
but they did not help me out probably due to my expansion is related to file directory. However, the it worked if the path is hard-coded, which is not thru shell variable along with other part of the shell script.
Code:
awk  '{if ($1 == "aaa") {print $0 >> "/storage/tmp/group_a.gtf"} else  if ($7 == "bbb") {print $0 >> "/storeage/tmp/group_b.gtf"}}'  infle

Also It is interesting with following oneliner, in which the expansion worked at every other hit of ENTER. Could not understand why.
Code:
MY_DIR="/storage/tmp"; awk -v awk_var="$MY_DIR" '{print awk_var}'

/storage/tmp

/storage/tmp

/storage/tmp

 ......

1) What are the rules to expand shell (bash) variables (especialy related to directory) inside awk script?
2) Why my last oneliner worked every other time with ENTER? Thanks a lot!
1) It looks like vgersh99 has already shown you how to expand shell variables for use in awk. Note, however, that his code will append to your two output files; not replace them if they already existed. It also looks like he had a typo looking for "bbb" in field 7 instead of in field 1. And, if you have some value other than "aaa" or "bbb" on an input line, that line will be written to one of those two output files depending on what was on the previous input line or cause awk to error output if another value appeared on the first input line. If that isn't what you want, you might want to try:
Code:
MY_DIR="/storage/tmp"
awk -v myDir="${MY_DIR}" '
$1 == "aaa"{ print > (myDir "/group_a.gtf")}
$1 == "bbb"{ print > (myDir "/group_b.gtf")}' infile

The above form should work with any standard awk as long as you don't specify more than 9 output files in your awk script.



2) Your one-liner worked every time you hit ENTER. The blank lines you see in the output are the echo of your input lines and the printing of the contents of your variable is the output from the print statement in your awk script.

This might be more obvious if you used:
Code:
MY_DIR="/storage/tmp"; printf '\ntext to be ignored by awk\n\n' | awk -v awk_var="$MY_DIR" '{print awk_var}'
/storage/tmp
/storage/tmp
/storage/tmp
$ 

These 3 Users Gave Thanks to Don Cragun For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Use parameter expansion over a parameter expansion in bash.

Hello All, Could you please do help me here as I would like to perform parameter expansion in shell over a parameter expansion. Let's say I have following variable. path="/var/talend/nat/cdc" Now to get only nat I could do following. path1="${path%/*}" path1="${path1##*/}" Here... (8 Replies)
Discussion started by: RavinderSingh13
8 Replies

2. Shell Programming and Scripting

Bash Parameter Expansion

#!/bin/bash SNMPW='/usr/bin/snmpwalk' while read h i do loc=$($SNMPW -v3 -u 'Myusername' -l authPriv -a SHA -A 'Password1' -x AES -X 'Password2' $i sysLocation.0 2>/dev/null) loc=${loc:-" is not snmpable."} loc=${loc##*: } loc=${loc//,/} echo "$i,$h,$loc" done < $1 My question is ... ... (1 Reply)
Discussion started by: sumguy
1 Replies

3. Shell Programming and Scripting

Bash Parameter Expansion

I have made the following examples that print various parameter expansions text: iv-hhz-sac/hpac/hhz.d/iv.hpac..hhz.d.2016.250.070018.sac (text%.*): iv-hhz-sac/hpac/hhz.d/iv.hpac..hhz.d.2016.250.070018 (text%%.*): iv-hhz-sac/hpac/hhz (text#*.): d/iv.hpac..hhz.d.2016.250.070018.sac... (2 Replies)
Discussion started by: kristinu
2 Replies

4. Shell Programming and Scripting

Bash variable expansion

Hello. The file /etc/fstab contains UUID=957c3295-9944-1593-82e2-2b90dede4312 / ext4 noatime,discard,acl,user_xattr 1 1 I fill a variable SOME_LINE=$( cat /etc/fstab | grep \/\..*ext4 | grep noatime,discard )echo $SOME_LINE... (3 Replies)
Discussion started by: jcdole
3 Replies

5. Shell Programming and Scripting

Bash expansion

Hello. I cannot write a command without using eval. Any help is welcome Note 1 : What does the function SOMETHING has no importance. Note 2 : What does the command find has no importance. It is an expansion variable problem : where to put or or or anythings else What works (FILTRE_1... (8 Replies)
Discussion started by: jcdole
8 Replies

6. Shell Programming and Scripting

Shell variable expansion in awk

I want to split one file input.tab into two separate ones, odd lines to input_reads1.txt, even lines to input_reads2.txt for a serial of files with similar name pattern. Also I want to "match" input/output file names to keep consistency of file name: CSEL_02.0_input.tab CSEL_03.4_input.tab... (2 Replies)
Discussion started by: yifangt
2 Replies

7. Shell Programming and Scripting

Bash shell expansion help

This is what I have in my directory. $ ls test1.txt test2.txt test3.txt test4.txt test5.txt test_script.sh This is what my shellscript looks like. #!/bin/bash for filename in /shell_expansion/*.txt; do for ((i=0; i<=3; i++)); do echo "$filename" ... (5 Replies)
Discussion started by: cokedude
5 Replies

8. Shell Programming and Scripting

Bash variable delayed expansion?

i write a batch file , here is the content. dirname='date +%Y-%m-%d' mkdir dirname but it doen's work, it just create a folder named date and +%Y-%m-%d. i have tried run the command seperately in the bash prompt. after the first sentence executed , i use $dirname to watch the value of... (4 Replies)
Discussion started by: premotheus
4 Replies

9. Shell Programming and Scripting

bash - delay expansion of variable

Hello - I have a bash script which does some logging, and I'd like to include the line number of the echo statement that pipes into $LOGGER: MYPID=$$ MYNAME=`basename $0` LOGGER="/usr/bin/logger -t $MYNAME($LINENO) -p daemon.error" ... echo 'this is an entry into the log file' | $LOGGER ... (3 Replies)
Discussion started by: scandora
3 Replies

10. Shell Programming and Scripting

globbing, $# is too high after wildcard expansion in bash script

How can I pass in an argument such as "*.k" to a bash script without having to double-quote *.k and not having *.k `glob` to match all files in the pattern? I tried using noglob in my script but this didn't work the way I thought it would.. expansion is still occuring, $# is higher than I... (3 Replies)
Discussion started by: zoo591
3 Replies
Login or Register to Ask a Question