Splitting a file based on record sin another file

er_ashu · #1 (**permalink**) 05-12-2008

All,

We receive a file with a large no of records (records can vary) and we have to split it into two files based on another file. e.g.

File1:

UHDR 2008112
"25187","00000022","00",21-APR-1991,"" ,"D",-000000519,+0000000000,"C", ,+000000000,+000000000,000000000,"2","" ,21-APR-1991
"8Y3H4","0000004H","00",16-APR-1992,"" ,"H",-001621119,+0000000000,"C", ,+000000000,+000000000,000000000,"2","" ,21-APR-1991
"95Y8U","02100971","00",03-MAR-1991,"" ,"H",-000004499,+0000000000,"" , ,+000000000,+000000000,000000000,"2","US",21-APR-1991
"24567","02100973","00",26-SEP-1991,"" ,"H",-000000362,+0000000000,"" , ,+000000000,+000000000,000000000,"2","US",21-APR-1991
--
--
--
UTRL 00144700

File2:
2518720080512
2456720080512
1256720080512
8WE7820080512
8Y3H020080512
8Y3H220080512
8Y3H420080512
8Y3H620080512
-
--
--
--

If the first 5 characters of file 2 matched with the chars 2-6 in file1, it should separate those records and put them into another file and rest of the records should be copied into a second file.

I tried cut command but as the file1 is quite large, it was taking a lot of time to put the values into a variable and then compare it.

Is there a way which can do the above task quite fast.

Please help as it is needed urgently.

Thanks in anticipation.

era · #2 (**permalink**) 05-13-2008

If your grep can read patterns from a file (like GNU grep), try something like this.

Code:

cut -c1-5 file2 >patterns
grep -f patterns file1 >matches
grep -f patterns -v file1 >nonmatches

There is some room for improvement, relating to making the patterns always match at beginning of line, etc. You could replace the first line with something like

Code:

sed 's/^\(.....\).*/^"\1",/' file2 >patterns

This adds the double quotes around the search string and adds a comma behind it and the special character "^" before it, which means match only at beginning of line.

Franklin52 · #3 (**permalink**) 05-13-2008

With awk:

Code:

awk 'NR==FNR{a[substr($0,1,5)];next}
NF<3{next}
substr($1,2,5) in a {print > "file1";next}
{print > "file2"}' file2 file1

Regards

er_ashu · #4 (**permalink**) 05-13-2008

Thanks Franklin and era.

Franklin,
I fpossible, can you explain wha tthe cod eis doing. I'm an awk novice and this will help me in interpreting things better.
Also, what do I do if I don't want to modify the original files.

Thanks again.

Franklin52 · #5 (**permalink**) 05-13-2008

Code:

awk 'NR==FNR{a[substr($0,1,5)];next}

Gets first 5 characters of the first file into array.

Code:

 NF<3{next}

Ignore lines with less then 3 fields (the first and last line)

Code:

 substr($1,2,5) in a {print > "file1";next}

If 5 characters after quotes in array print to file1..

Code:

 {print > "file2"}

... else print to file2

The original files should not change.

Regards

er_ashu · #6 (**permalink**) 05-13-2008

Franklin,

You rock.
But when I use this:

Code:

awk 'NR==FNR{a[substr($0,1,5)];next}
NF<3{next}
substr($1,2,5) in a {print > "matched.txt";next}
{print > "notmatched.txt"}' file2 file1

I only see matched.txt file but not "notmatched.txt". Am I missing something here?

Franklin52 · #7 (**permalink**) 05-13-2008

With the given files it works fine for me.
Are the formats of the files you've posted different from the original files?
The first line of the script use the position 1-5 from the first file as key and it's compared with position 2-6 of the second file. Check the exact positions of your original files. You should be able to fix it now for

.

Regards

More UNIX and Linux Forum Topics You Might Find Helpful
Thread	Thread Starter	Forum	Replies	Last Post
Splitting file based on number of rows	wahi80	Shell Programming and Scripting	2	06-03-2008 09:38 PM
Splitting a file based on the records in another file	er_ashu	Shell Programming and Scripting	2	05-12-2008 04:34 PM
splitting a record and adding a record to a file	rsolap	Shell Programming and Scripting	1	08-13-2007 01:58 PM
splitting files based on text in the file	matrix1067	Shell Programming and Scripting	1	01-30-2006 08:45 PM
Splitting a file based on some condition and naming them	srivsn	Shell Programming and Scripting	1	12-07-2005 11:27 AM

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes	Rate This Thread
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode	Rate This Thread: