UNIX for Dummies Questions & Answers

All,

We receive a file with a large no of records (records can vary) and we have to split it into two files based on another file. e.g.

File1:

UHDR 2008112
"25187","00000022","00",21-APR-1991,"" ,"D",-000000519,+0000000000,"C", ,+000000000,+000000000,000000000,"2","" ,21-APR-1991
"8Y3H4","0000004H","00",16-APR-1992,"" ,"H",-001621119,+0000000000,"C", ,+000000000,+000000000,000000000,"2","" ,21-APR-1991
"95Y8U","02100971","00",03-MAR-1991,"" ,"H",-000004499,+0000000000,"" , ,+000000000,+000000000,000000000,"2","US",21-APR-1991
"24567","02100973","00",26-SEP-1991,"" ,"H",-000000362,+0000000000,"" , ,+000000000,+000000000,000000000,"2","US",21-APR-1991
--
--
--
UTRL 00144700


File2:
2518720080512
2456720080512
1256720080512
8WE7820080512
8Y3H020080512
8Y3H220080512
8Y3H420080512
8Y3H620080512
-
--
--
--

If the first 5 characters of file 2 matched with the chars 2-6 in file1, it should separate those records and put them into another file and rest of the records should be copied into a second file.

I tried cut command but as the file1 is quite large, it was taking a lot of time to put the values into a variable and then compare it.

Is there a way which can do the above task quite fast.

Please help as it is needed urgently.

Thanks in anticipation.

If your grep can read patterns from a file (like GNU grep), try something like this.

Code:

cut -c1-5 file2 >patterns
grep -f patterns file1 >matches
grep -f patterns -v file1 >nonmatches

There is some room for improvement, relating to making the patterns always match at beginning of line, etc. You could replace the first line with something like

Code:

sed 's/^\(.....\).*/^"\1",/' file2 >patterns

This adds the double quotes around the search string and adds a comma behind it and the special character "^" before it, which means match only at beginning of line.

With awk:

Code:

awk 'NR==FNR{a[substr($0,1,5)];next}
NF<3{next}
substr($1,2,5) in a {print > "file1";next}
{print > "file2"}' file2 file1

Regards

Thanks Franklin and era.

Franklin,
I fpossible, can you explain wha tthe cod eis doing. I'm an awk novice and this will help me in interpreting things better.
Also, what do I do if I don't want to modify the original files.

Thanks again.

Code:

awk 'NR==FNR{a[substr($0,1,5)];next}

Gets first 5 characters of the first file into array.

Code:

 NF<3{next}

Ignore lines with less then 3 fields (the first and last line)

Code:

 substr($1,2,5) in a {print > "file1";next}

If 5 characters after quotes in array print to file1..

Code:

 {print > "file2"}

... else print to file2


The original files should not change.

Regards

Franklin,

You rock.
But when I use this:

Code:

awk 'NR==FNR{a[substr($0,1,5)];next}
NF<3{next}
substr($1,2,5) in a {print > "matched.txt";next}
{print > "notmatched.txt"}' file2 file1

I only see matched.txt file but not "notmatched.txt". Am I missing something here?

With the given files it works fine for me.
Are the formats of the files you've posted different from the original files?
The first line of the script use the position 1-5 from the first file as key and it's compared with position 2-6 of the second file. Check the exact positions of your original files. You should be able to fix it now for .

Regards