Shell Programming and Scripting

I have a fixed width file. The records looks something similar to below:

Type ID SSN NAME .....AND SOME MORE FIELDS
A1 1234 .....
A1 1234 .....
B1 1234 .....
M2 4567 .....
M2 4567 .....
N2 4567 .....
N2 4567 .....
A1 9999
N2 9999


Now if A1 is present then B1 has to be present. Also if M2 is present then N2 has to be present.
So A1 and B1 record goes together. And M2 and N2 records goes together.

I am looking for TWO things:

1) I am looking for records where A1 record has missing B1 record and vise versa AND I am looking for records
where M2 record has missing N2 record and vice versa.
In other words, if the A1 and B1 combination is missing then throw that record in a file. If M2 and N2 combination is missing throw
that in a file

e.g: So in the above example it will be the records below as both of them has missing B1 and M2 records.

A1 9999
N2 9999

2) I am looking to find duplicate records and put it in a separate file. For Duplicate we look for A1 and 1234 in combination(Type and Id field together).
e.g: So I am looking to put A1 duplicates in one file, B1 duplicates in another file,M2 duplicates in another file and N2 duplicates in another.
so from above example A1 1234 in one file as it is duplicate and M2 4567 will go in one file as it is duplicate and N2 4567 in another file.

Really appreciate your help.

Thanks

Divide them into two files, A/M and (B/N only, where B and N are converted to A and M), sort each, comm -3, sed process them so any line beginning with a tab is a missing A/M else it indicates a missing B/N (convert on fly to B and N).

Code:

#!/bin/bash # or ksh where there is <().
 
comm -3 <(
  grep '^[AM]' infile |sort
 ) <(
  sed '
    s/^B/A/
    t
    s/^N/M/
    t
    d
   ' infile | sort 
 ) | sed '
    s/^\t//
    t
    s/^A/B/
    s/^M/N/
   '

Note: \t is typed in as a real tab.

Try this awk solution:

Code:

awk '
BEGIN {
   for(i=split("A1 B1 M2 N2", s);i;i-=2) {
      SAME[s[i]]=s[i-1]
      SAME[s[i-1]]=s[i]
   }
}
$1 SUBSEP $2 in K {
   print > "dup_"$1
   next
}
{
    H[NR]=$0
    K[$1,$2]=NR
}
END {
   for(i=1;i<=NR;i++) {
     if(i in H) {
        split(H[i], V)
        if (!((SAME[V[1]] SUBSEP V[2]) in K))
            print H[i] > "missing_"V[1]
        else print H[i]
     }
    }
}' infile