Sponsored Content
Full Discussion: awk NR==FNR output control
Top Forums UNIX for Dummies Questions & Answers awk NR==FNR output control Post 302532027 by agama on Sunday 19th of June 2011 11:06:04 AM
Old 06-19-2011
The short answer is to process f2 first, then process f1. This will reduce your memory footprint as you'll only save 68 things in a[] rather than 48K things.

The long answer is to be a bit more clever which might also help speed things up. Your programme will loop through the entire contents of file f1 for each record in f2 (48,000 * 68) testing to see if there's a match. Instead, use the hash capabilities of awk to your advantage.

This example assumes that the 'key' (field 1 in file 2) can occur multiple times and so we must do a bit of looping for each f1 record, but the only looping needed when reading limited to the number of duplicate 'keys' that existed in f2 for the current f1 record. If f2 will not have duplicates, then the code can be simplified more, but not knowing you exact data, this general case will work for either. We also don't need to make an explicit check to see if the key in the current record matches the one saved from f2.

Code:
awk -v f2=f2 '
    BEGIN {
        while( (getline<f2) > 0 )   # read and collect records from f2
        {
            key = $1;
            ki = kidx[key]++;        # track number of duplicate keys (0 based)
            k2rec[key,ki] = $0;      # save unique record by key and dup count
        }
        close( f2 );
    }

    {
        key = $3;
        for( i = 0; i < kidx[key]; i++ )          # for each duplicate of key
            printf( "%s\t%s\n", k2rec[key,i], $0 );   # print f2 record, followed by current f1 record
    }
' <f1 >f3

Hope this makes sense.

Last edited by agama; 06-19-2011 at 12:07 PM.. Reason: Corrected printf to output f2 then f1
This User Gave Thanks to agama For This Post:
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Awk: different between NR and FNR

As I know: FNR: The ordinal number of the current record in the current file. NR: The ordinal number of the current record from the start of input. I don't understand really differency between NR and FNR. Who can explain it for me? And give me an example. Thanks (1 Reply)
Discussion started by: anhtt
1 Replies

2. Shell Programming and Scripting

awk NR==FNR compare 2 files produce a 3rd

hi, i have two files, both with 3 columns, the 3rd column has common values between the two files and i want to produce a 3rd file with 4 columns. file 1 a, ,b c file 2 a, b ,d I want to compare the 3rd value and if a match print to file 3 with the 3 columns from the first file... (11 Replies)
Discussion started by: borderblaster
11 Replies

3. Shell Programming and Scripting

error "awk: (FILENAME=- FNR=23) fatal: division by zero attempted"

Hi , I have file : after i run this command : there are error can we print blank line if output error ?? thanks.. ^^ (4 Replies)
Discussion started by: justbow
4 Replies

4. UNIX for Dummies Questions & Answers

Multiple Column print after lookup using NR==FNR (awk)

foo.txt FAMID IID AFF SEX Group AgeCat Dis1 Dis2 Dis3 Dis4 Dis5 Dis6 Dis6 AMD0001 Mayo_49542 1 2 AMD 8 1 1 1 1 1 1 1 AMD0002 Mayo_49606 1 1 AMD 3 1 1 1 1 ... (7 Replies)
Discussion started by: genehunter
7 Replies

5. Shell Programming and Scripting

Awk FNR==NR question

awk -F'' 'FNR==NR {a=$2; next} {$1=a} 1' $useralias ${entries} >> ${entries}_2 Hi, Is there anyway to alter this command so that if it does not find a match it will just leave the line alone instead of replacing what it doesn't find with a blank space? (4 Replies)
Discussion started by: Jazmania
4 Replies

6. Shell Programming and Scripting

How to control a null output in EMC storage?

I dont want to print the output in a EMC VMAX storage if it says "The specified device was not found", however it is not letting me do it. I am trying to run this command: symaccess -sid xxxx list -type storage -devs 1234 output: The specified device was not found I just want the script... (1 Reply)
Discussion started by: prodigy06
1 Replies

7. Shell Programming and Scripting

Tip: alternative for NR==FNR in awk

Example: $ cat file1 2 3$ cat file2 1 2 3 4 5 6The following awk script works like a charm, NR==FNR is true for file1, the remainder runs for file2: awk ' NR==FNR {A; next} ($1 in A) ' file1 file2 2 3Now have an empty file1: >file1and run the awk script again. The result is empty... (8 Replies)
Discussion started by: MadeInGermany
8 Replies

8. Shell Programming and Scripting

awk --> selective printout with FNR

Hi everybody! need some awk-support. i want a line-selective printout of a file. wat i normally will do with ... awk ' FNR==8' sample.txt But now i need the data from line 8, 10 and the following data from line13 to 250 wich is not end of the file. I tried allready to combine it but without... (2 Replies)
Discussion started by: IMPe
2 Replies

9. Shell Programming and Scripting

Explanation of FNR in this awk script

To merge mutiple *.tab files as: file1.tab rs1 A A rs2 A A rs3 C C rs4 C Cfile2.ind rs1 T T rs2 T T rs3 G G rs4 G Gand file3.tab rs1 B B rs2 B B rs3 L L rs4 L LOutput: file1.tab file2.tab file3.tab AA TT BB AA TT BB CC GG LL CC GG ... (4 Replies)
Discussion started by: yifangt
4 Replies

10. Shell Programming and Scripting

Awk: Assigning a variable to be the value of FNR at a certain line

Sorry for the probably strangely worded title but I don't really know how else to put it. Background context: Post processing LAMMPS simulation data. tl;dr: I'm making two spheres collide, every defined timestep the simulation outputs a bunch of data including total energy of the particles,... (10 Replies)
Discussion started by: ThomasP
10 Replies
makedbm(1M)															       makedbm(1M)

NAME
makedbm - make a Network Information System database SYNOPSIS
nis_input_file] nis_output_name] nis_domain_name] nis_master_name] infile outfile database_name Remarks The Network Information Service (NIS) was formerly known as Yellow Pages (yp). Although the name has changed, the functionality of the service remains the same. DESCRIPTION
generates databases (maps) for the Network Information System (NIS) from infile. A database created by consists of two files: and A data- base contains records called dbm records composed of key-value pairs. Each line of infile is converted to a single dbm record; all characters up to the first tab or space form the key, and the remainder of the line is the value. If a value read from infile ends with the value for that record is continued onto the next line. The NIS clients must interpret the character (which means that does not treat the as if it precedes a comment). If infile is a hyphen reads standard input. always generates a special dbm record with the key whose value is the time of last modification of infile (or the current time, if infile is This value is also known as the order number of a map, and prints it for a specified NIS map (see yppoll(1M)). Another special dbm record created by has the key Its value is usually the host name retrieved by however, the option can be used to spec- ify a different value (see gethostname(2)). If the option is used, another special dbm record with the key is created. When this key exists in the NIS host.by* maps or ipnodes.by* maps and the NIS host name resolution fails, the process will query the Internet domain name server, to provide the host name resolution. Before using the option, it is recommended that the name services switch, be set to allow NIS host name resolution first. (Note that, since the process only checks hosts.by* and ipnodes.by* for the existence of the key, using the option on any other NIS map will have no effect. Also, the option should be used on both the *.byname and *.byaddr maps, not one exclusively.) If the option is used, another special dbm record created is the key. If this key exists in an NIS map, will only allow privileged pro- cesses (applications that can create reserved ports) to access the data within the map. Options recognizes the following options and command-line arguments. Create a special dbm record with the key This key, which is in the hosts.byname, hosts.byaddr, ipnodes.byname, and ipnodes.byaddr maps, allows the process to query the Internet domain name server (see named(1M)). Convert the keys of the given map to lowercase. This command option allows host name matches to work independent of character-case distinctions. Accept connections from secure NIS networks only. Create a special dbm record with the key and the value If the option is used, another special dbm record created is the key. If this key exists in an NIS map, will only allow privileged processes to access the data within the map (that is, applications that can create reserved ports). Create a special dbm record with the key and the value nis_output_name. Create a special dbm record with the key and the value nis_domain_name. Replace the value of the special dbm record whose key is with nis_master_name. Undo the database_name (that is, write the contents of database_name to the standard output) one dbm record per line. A single space sepa- rates each key from its value. EXAMPLES
Shell scripts can be written to convert ASCII files such as to the key-value form used by For example, #!/usr/bin/sh /usr/bin/awk 'BEGIN { FS = ":" } { print $1, $0 }' /etc/netgroup | makedbm - netgroup converts the file to a form that is read by to make the NIS map The keys in the database are names, and the values are the remainders of the lines in the file. AUTHOR
was developed by Sun Microsystems, Inc. SEE ALSO
domainname(1), named(1M), ypinit(1M), ypmake(1M), yppoll(1M), gethostname(2), netgroup(4), ypfiles(4). makedbm(1M)
All times are GMT -4. The time now is 02:34 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy