Visit Our UNIX and Linux User Community


Tip: alternative for NR==FNR in awk


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Tip: alternative for NR==FNR in awk
# 1  
Old 08-01-2014
Tip: alternative for NR==FNR in awk

Example:
Code:
$ cat file1
2
3

Code:
$ cat file2
1
2
3
4
5
6

The following awk script works like a charm, NR==FNR is true for file1, the remainder runs for file2:
Code:
awk '
NR==FNR {A[$1]; next}
($1 in A)
' file1 file2
2
3

Now have an empty file1:
Code:
>file1

and run the awk script again.
The result is empty as expected.
However, this time it did the NR==FNR action for file2!
Check with
Code:
awk '
NR==FNR {A[$1]; print FILENAME,$1; next}
($1 in A)
' file1 file2

Here the result was good - by good luck.
In some other cases this can lead to misbehavior.
The following fix is available:
Code:
awk '
FILENAME=="file1" {A[$1]; print FILENAME,$1; next}
($1 in A)
' file1 file2

But not always applicable, for example if you have a wild card file*.
So here is a better fix:
Code:
awk '
F==0 {A[$1]; print FILENAME,$1; next}
($1 in A) {print}
' file1 F=1 file2

Now F is undefined=0 in file1 and set to 1 before file2 is opened.
You can even continue like this: file1 F=1 file2 F=2 file3
then you can even distinguish between file2 and file3.
The ultra-short-code-hackers can even use !F.
These 3 Users Gave Thanks to MadeInGermany For This Post:
# 2  
Old 08-01-2014
Another option would be to just check the filename:
Code:
awk '
BEGIN {f=FILENAME}
FILENAME==f {A[$1]; f=FILENAME; next}
($1 in A)
' file1 file2

The ultra-short-code-hackers can even use:
Code:
awk '
FILENAME==f {A[$1]; f=FILENAME; next}
($1 in A)
' f=file1 file1 file2


Last edited by jethrow; 08-01-2014 at 08:20 AM..
# 3  
Old 08-01-2014
Hmm, what is the f=FILENAME in the main loop for?
Then, in your first example, the BEGIN {f=FILENAME} only works with nawk and derived awk's.
# 4  
Old 08-01-2014
I get the impression that's why that feature exists, so you can process different files with their own default values of some sort.
# 5  
Old 08-01-2014
Quote:
Originally Posted by Corona688
I get the impression that's why that feature exists, so you can process different files with their own default values of some sort.
Sure.
Most useful is FS, like file1 FS="," file2
Then one can as well test with FS!=","
# 6  
Old 08-01-2014
You can also let awk directly examine the arguments given to it:
Code:
awk '
BEGIN {	printf("ARGV[0]=%s\n", ARGV[0])
	for(i = 1; i < ARGC; i++)
		if(ARGV[i] ~ /=/)
			printf("ARGV[%d]=%s: assignment\n", i, ARGV[i])
		else {	printf("ARGV[%d]=%s: file operand\n", i, ARGV[i])
			if(!f1)	f1 = ARGV[i]
		}
	print ""
}
FILENAME == f1 {
	# Process lines from 1st file here...
	printf("From 1st file(%s); %s\n", f1, $0)
	next
}
{	# Process remaining files here...
	printf("From subsequent file(%s): %s\n", FILENAME, $0)
}' FS=, empty_file OFS='|' file1 FS='|' file2

If empty_file is an empty file, file1 contains:
Code:
f1 line1
f1 line2

and file2 contains:
Code:
f2 line1
f2 line2

it produces the output:
Code:
ARGV[0]=awk
ARGV[1]=FS=,: assignment
ARGV[2]=empty_file: file operand
ARGV[3]=OFS=|: assignment
ARGV[4]=file1: file operand
ARGV[5]=FS=|: assignment
ARGV[6]=file2: file operand

From subsequent file(file1): f1 line1
From subsequent file(file1): f1 line2
From subsequent file(file2): f2 line1
From subsequent file(file2): f2 line2

and if the last line of the script is changed to:
Code:
}' FS=, OFS='|' file1 FS='|' file2

it produces the output:
Code:
ARGV[0]=awk
ARGV[1]=FS=,: assignment
ARGV[2]=OFS=|: assignment
ARGV[3]=file1: file operand
ARGV[4]=FS=|: assignment
ARGV[5]=file2: file operand

From 1st file(file1); f1 line1
From 1st file(file1); f1 line2
From subsequent file(file2): f2 line1
From subsequent file(file2): f2 line2

This User Gave Thanks to Don Cragun For This Post:
# 7  
Old 08-01-2014
Quote:
Originally Posted by jethrow
Another option would be to just check the filename:
Code:
awk '
BEGIN {f=FILENAME}
FILENAME==f {A[$1]; f=FILENAME; next}
($1 in A)
' file1 file2

This might work on some systems, but the standards say that the value of FILENAME in a BEGIN clause is undefined.

In awk on OS X, FILENAME expands to an empty string (or 0 depending on context) in a BEGIN action.

Previous Thread | Next Thread
Test Your Knowledge in Computers #715
Difficulty: Easy
A stored-program computer is a computer that stores program instructions on external hard drives..
True or False?

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Awk: Assigning a variable to be the value of FNR at a certain line

Sorry for the probably strangely worded title but I don't really know how else to put it. Background context: Post processing LAMMPS simulation data. tl;dr: I'm making two spheres collide, every defined timestep the simulation outputs a bunch of data including total energy of the particles,... (10 Replies)
Discussion started by: ThomasP
10 Replies

2. Shell Programming and Scripting

Explanation of FNR in this awk script

To merge mutiple *.tab files as: file1.tab rs1 A A rs2 A A rs3 C C rs4 C Cfile2.ind rs1 T T rs2 T T rs3 G G rs4 G Gand file3.tab rs1 B B rs2 B B rs3 L L rs4 L LOutput: file1.tab file2.tab file3.tab AA TT BB AA TT BB CC GG LL CC GG ... (4 Replies)
Discussion started by: yifangt
4 Replies

3. Shell Programming and Scripting

awk --> selective printout with FNR

Hi everybody! need some awk-support. i want a line-selective printout of a file. wat i normally will do with ... awk ' FNR==8' sample.txt But now i need the data from line 8, 10 and the following data from line13 to 250 wich is not end of the file. I tried allready to combine it but without... (2 Replies)
Discussion started by: IMPe
2 Replies

4. Shell Programming and Scripting

Help with Alternative for NR==FNR

Hi, I have an issue with the below script nawk 'NR==FNR{a=$4" "$5}NR>FNR{print NF?$0:a"\n";if(/^cn:/) x=$0}' FS="" in1.txt in2.txt > out1.txt It is taking too long to get a string from in1.text, search for the string in in2.txt and create a new file out1.txt. Is there any alternative way we... (1 Reply)
Discussion started by: Samingla
1 Replies

5. Shell Programming and Scripting

Quick awk tip :)

how do i "awk" the date after the from only to compare it on a if statement later . filename example: server1-ips-ultranoob-ok_From_2012_21_12-23:40:23_To_2012_21_12-23:49:45.zip what i want o do is compare only the date from the string in "From_2012_21_12" in this case i only want the... (4 Replies)
Discussion started by: drd0spt
4 Replies

6. Shell Programming and Scripting

Awk FNR==NR question

awk -F'' 'FNR==NR {a=$2; next} {$1=a} 1' $useralias ${entries} >> ${entries}_2 Hi, Is there anyway to alter this command so that if it does not find a match it will just leave the line alone instead of replacing what it doesn't find with a blank space? (4 Replies)
Discussion started by: Jazmania
4 Replies

7. UNIX for Dummies Questions & Answers

awk NR==FNR output control

Hi Guys, I have two files: f1: A B C D E F G H f2: A X Y Z f1 has 48000 lines, and f2 has 68. I have been matching f1 $3 to f2 $1, and getting f3: A A B C D E F G I would like f3 too look like this: A X Y Z A B C D E F G (2 Replies)
Discussion started by: heecha
2 Replies

8. UNIX for Dummies Questions & Answers

Multiple Column print after lookup using NR==FNR (awk)

foo.txt FAMID IID AFF SEX Group AgeCat Dis1 Dis2 Dis3 Dis4 Dis5 Dis6 Dis6 AMD0001 Mayo_49542 1 2 AMD 8 1 1 1 1 1 1 1 AMD0002 Mayo_49606 1 1 AMD 3 1 1 1 1 ... (7 Replies)
Discussion started by: genehunter
7 Replies

9. Shell Programming and Scripting

awk NR==FNR compare 2 files produce a 3rd

hi, i have two files, both with 3 columns, the 3rd column has common values between the two files and i want to produce a 3rd file with 4 columns. file 1 a, ,b c file 2 a, b ,d I want to compare the 3rd value and if a match print to file 3 with the 3 columns from the first file... (11 Replies)
Discussion started by: borderblaster
11 Replies

10. Shell Programming and Scripting

Awk: different between NR and FNR

As I know: FNR: The ordinal number of the current record in the current file. NR: The ordinal number of the current record from the start of input. I don't understand really differency between NR and FNR. Who can explain it for me? And give me an example. Thanks (1 Reply)
Discussion started by: anhtt
1 Replies

Featured Tech Videos