Can awk do lookups to other files and process results Post: 302250540

Sponsored Content

Top Forums Shell Programming and Scripting Can awk do lookups to other files and process results Post 302250540 by radoulov on Thursday 23rd of October 2008 04:15:13 PM

10-23-2008

Registered User

Edit: actually you should jump to the second example. The first assumes that posision 39 is always empty.

The code below sets 1 for Linda, because she's not present in the example file_cd1:

Code:

awk 'NR == FNR { cd1[$1]; next }
f { cd2[$1] = $2; next }
!f && / $/ { 
  if (!(substr($0, 7, 6) in cd1)) sub(/. $/, "1 ") 
  if ((substr($0, 29, 4) in cd2) && cd2[substr($0, 29, 4)] != "abc") 
    sub(/  $/, "2 ") 
  }1' file_cd1 f=1 file_cd2 f=0 file_in

An example:

Code:

% awk 'NR == FNR { cd1[$1]; next }
f { cd2[$1] = $2; next }
!f && / $/ {
  if (!(substr($0, 7, 6) in cd1)) sub(/. $/, "1 ")
  if ((substr($0, 29, 4) in cd2) && cd2[substr($0, 29, 4)] != "abc")
    sub(/  $/, "2 ")
  }1' file_cd1 f=1 file_cd2 f=0 file_in
1234567890123456789012345678901234567890
Joe   123456  30 Main St    1234    F 1 
Jim   101362  1492 Hugh     0101    P   
Kerry 040419  6091 Lost St  0101    F   
Linda 123456  50 High Way   1235      1 
Matt  242424  48 Speedway Dr4343    F 2 
Kerrin180118  99 Skaters Way2012    P  *

If you want the second test to have precedence:

Code:

awk 'NR == FNR { cd1[$1]; next }
f { cd2[$1] = $2; next }
!f && / $/ { 
  if (!(substr($0, 7, 6) in cd1)) sub(/. $/, "1 ") 
  if ((substr($0, 29, 4) in cd2) && cd2[substr($0, 29, 4)] != "abc") 
    sub(/. $/, "2 ") 
  }1' file_cd1 f=1 file_cd2 f=0 file_in

For example:

Code:

% awk 'NR == FNR { cd1[$1]; next }
quote> f { cd2[$1] = $2; next }
quote> !f && / $/ { 
quote>   if (!(substr($0, 7, 6) in cd1)) sub(/. $/, "1 ") 
quote>   if ((substr($0, 29, 4) in cd2) && cd2[substr($0, 29, 4)] != "abc") 
quote>     sub(/. $/, "2 ") 
quote>   }1' file_cd1 f=1 file_cd2 f=0 file_in
1234567890123456789012345678901234567890
Joe   123456  30 Main St    1234    F 1 
Jim   101362  1492 Hugh     0101    P   
Kerry 040419  6091 Lost St  0101    F   
Linda 123456  50 High Way   1235      2 
Matt  242424  48 Speedway Dr4343    F 2 
Kerrin180118  99 Skaters Way2012    P  *

Last edited by radoulov; 10-24-2008 at 06:20 AM.. Reason: correction

radoulov

View Public Profile for radoulov

Find all posts by radoulov

10 More Discussions You Might Find Interesting

1. HP-UX

TWS 8.4 on HP-UX - lookups?

Just wondering if anyone else is using IBM's TWS on HP-UX 11.11i. Seeing some very strange name-lookup issues when it comes to using various utilities on the system. The same software works fine o0n AIX, Linux, Solaris, etc, but on HP-UX there is noticeable time lags in issuing commands - at the...

2. Shell Programming and Scripting

awk - Matching columns between 2 files and reordering results

I am trying to match 4 colums (first_name,last_name,dob,ssn) between 2 files and when there is an exact match I need to write out these matches to a new file with a combination of fields from file1 and file2. I've managed to come up with a way to match these 2 files based on the columns (see below)...

3. Shell Programming and Scripting

awk script to parse results from TWO files

I am trying to parse two files and get data that does not match in one of the columns ( column 3 in my case ) Data for two files are as follows A.txt ===== abc 10 5 0 1 16 xyz 16 1 1 0 18 efg 30 8 0 2 40 ijk 22 2 0 1 25 B.txt ===== abc...

4. Shell Programming and Scripting

awk help: Match data fields from 2 files & output results from both into 1 file

I need to take 2 input files and create 1 output based on matches from each file. I am looking to match field #1 in both files (Userid) and create an output file that will be a combination of fields from both file1 and file2 if there are any differences in the fields 2,3,4,5,or 6. Below is an...

5. Shell Programming and Scripting

Bash-awk to process thousands of files

Hi to all, I have thousand of files in a folder with names with format "FILE-YYYY-MM-DD-HHMM" for what I want to send the following AWK command awk '/Code.*/' FILE-2014* I'd like to separate all files that have the same date to a folder named with the corresponding date. For example, if I...

6. BSD

Process remians in Running state causing other similar process to sleep and results to system hang

Hi Experts, I am facing one problem here which is one process always stuck in running state which causes the other similar process to sleep state . This causes my system in hanged state. On doing cat /proc/<pid>wchan showing the "__init_begin" in the output. Can you please help me here...

7. UNIX for Advanced & Expert Users

Process remians in Running state causing other similar process to sleep and results to system hang

8. UNIX for Advanced & Expert Users

Process remians in Running state causing other similar process to sleep and results to system hang

9. Shell Programming and Scripting

Process multiple large files with awk

Hi there, I'm camor and I'm trying to process huge files with bash scripting and awk. I've got a dataset folder with 10 files (16 millions of row each one - 600MB), and I've got a sorted file with all keys inside. For example: a sample_1 200 a.b sample_2 10 a sample_3 10 a sample_1 10 a...

10. UNIX for Beginners Questions & Answers

Comparing 2 files using awk , not getting any results - C shell

I am using c shell and trying to compare 2 files using awk . But the below awk statement doesnt give any result. Pls. advise why am not getting the desired o/p with the corrected awk script. Need to acheive this solution in awk using C shell. awk 'FNR==NR{a++;next} {for(i in a) {if ( a=$0...

LEARN ABOUT HPUX

mailq

mailq(1)						      General Commands Manual							  mailq(1)

NAME

       mailq - prints the mail queue

SYNOPSIS

DESCRIPTION

       prints a summary of the mail messages queued for future delivery.

       The  first line printed for each message shows the internal identifier used on this host for the message, the size of the message in bytes,
       the date and time the message was accepted into the queue, and the envelope sender of the message.  The second line shows the error message
       that  caused  this  message  to be retained in the queue; it will not be present if the message is being processed for the first time.  The
       status characters are:

	      to indicate that the job is being processed

	      to indicate that the load is too high to process the job

	      to indicate that the job is too new in the queue to process.

       The output lines that follow the second line show the message recipients, one per line.

       is identical to

   Options
       The supported options are:

       Show the mail submission queue specified in the
		       file instead of the MTA queue specified in the file.

       Show the lost items in the mail queue instead of normal queue items.

       Show the quarantined items in the mail queue instead
		       of the normal queue items.

       Limit processed jobs to those containing
		       substr as a substring of the queue ID or not when is specified.

       Limit processed jobs to quarantined jobs containing
		       substr as a substring of the quarantine reason or not when is specified.

       Limit processed jobs to those containing
		       substr as a substring of one of the recipients or not when is specified.

       Limit processed jobs to those containing
		       substr as a substring of the sender or not when is specified.

       Print verbose information.
		       This adds the priority of the message and a single character indicator or blank) indicating whether a warning  message  has
		       been  sent on the first line of the message.  In addition, extra lines may be intermixed with the recipients indicating the
		       `controlling user' information.	This shows who owns the programs that are executed on behalf of this message and the  name
		       of the alias this command expanded from, if any.

RETURN VALUE

       The utility exits with 0 on success, and >0 if an error occurs.

AUTHOR

       was developed by the University of California, Berkeley, and originally appeared in 4.0BSD.

FILES

       mail queue files for

SEE ALSO

       sendmail(1M).

																	  mailq(1)

10 More Discussions You Might Find Interesting

1. HP-UX

TWS 8.4 on HP-UX - lookups?

Discussion started by: rnbwkat

2. Shell Programming and Scripting

awk - Matching columns between 2 files and reordering results

Discussion started by: ambroze

3. Shell Programming and Scripting

awk script to parse results from TWO files

Discussion started by: roger67

4. Shell Programming and Scripting

awk help: Match data fields from 2 files & output results from both into 1 file

Discussion started by: ambroze

5. Shell Programming and Scripting

Bash-awk to process thousands of files

Discussion started by: Ophiuchus

6. BSD

Process remians in Running state causing other similar process to sleep and results to system hang

Discussion started by: naveeng

7. UNIX for Advanced & Expert Users

Process remians in Running state causing other similar process to sleep and results to system hang

Discussion started by: naveeng

8. UNIX for Advanced & Expert Users

Process remians in Running state causing other similar process to sleep and results to system hang

Discussion started by: naveeng

9. Shell Programming and Scripting

Process multiple large files with awk

Discussion started by: camor

10. UNIX for Beginners Questions & Answers

Comparing 2 files using awk , not getting any results - C shell

Discussion started by: reach2khan

LEARN ABOUT HPUX

mailq