Split Large Files Based On Row Pattern..


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Split Large Files Based On Row Pattern..
# 8  
Old 11-26-2013
Quote:
Originally Posted by aimy
Oh God, you are so kind Don.

Thank you so much!

But would you mind to explain a little bit about the code? Why FNR <=2 not FNR == 2?

Thanks.
It might be clearer to write:
Code:
FNR == 1 { next }
FNR == 2 { h = $0; next }

but:
Code:
FNR <= 2 { h = $0; next }

is less typing and accomplishes the same thing. Setting h to the contents of the 1st line is unnecessary, but it is reset to the contents of the 2nd line when the 2nd line is processed.

Last edited by Don Cragun; 11-26-2013 at 02:18 AM.. Reason: Add missing space.
This User Gave Thanks to Don Cragun For This Post:
# 9  
Old 11-26-2013
@ Don

I think you forgot to close ofile (close(ofile)), I suspect he might face problem if many files are open while handling big file
# 10  
Old 11-26-2013
Quote:
Originally Posted by Akshay Hegde
@ Don

I think you forgot to close ofile (close(ofile)), I suspect he might face problem if many files are open while handling big file
That was discussed in messages #2 and #3 in this thread.
# 11  
Old 11-26-2013
Yes.. Sorry I just seen it.
# 12  
Old 11-26-2013
Hi.

Sorry to bother again.

But when I execute against the real file, there's a such situation that the text itself contains comma ",", so I have to make it pipe "|" separated instead.

So, how do I split the file with "|" delimiter?

I've tried analyzing the script but could not find where could I tune in to replace the "," with "|".

Thanks.
# 13  
Old 11-26-2013
Code:
awk -F"|" ' {................}' file

OR
Code:
awk '{................}'  FS="|" file

# 14  
Old 11-26-2013
Thanks a lot Akshay.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Split files based on row delimiter count

I have a huge file (around 4-5 GB containing 20 million rows) which has text like: <EOFD>11<EOFD>22<EORD>2<EOFD>2222<EOFD>3333<EORD>3<EOFD>44<EOFD>55<EORD>66<EOFD>888<EOFD>9999<EORD> Actually above is an extracted file from a Sql Server with each field delimited by <EOFD> and each row ends... (8 Replies)
Discussion started by: amvip
8 Replies

2. UNIX for Advanced & Expert Users

Split one file to many based on pattern

Hello All, I have records in a file in a pattern A,B,B,B,B,K,A,B,B,K Is there any command or simple logic I can pull out records into multiple files based on A record? I want output as File1: A,B,B,B,B,K File2: A,B,B,K (9 Replies)
Discussion started by: deal1dealer
9 Replies

3. UNIX for Dummies Questions & Answers

Split a huge 7 GB File Based on Pattern into 4 files

Hi, I have a Huge 7 GB file which has around 1 million records, i want to split this file into 4 files to contain around 250k messages each. Please help me as Split command cannot work here as it might miss tags.. Format of the file is as below <!--###### ###### START-->... (6 Replies)
Discussion started by: KishM
6 Replies

4. Shell Programming and Scripting

Help needed - Split large file into smaller files based on pattern match

Help needed urgently please. I have a large file - a few hundred thousand lines. Sample CP START ACCOUNT 1234556 name 1 CP END ACCOUNT CP START ACCOUNT 2224444 name 1 CP END ACCOUNT CP START ACCOUNT 333344444 name 1 CP END ACCOUNT I need to split this file each time "CP START... (7 Replies)
Discussion started by: frustrated1
7 Replies

5. Shell Programming and Scripting

Problem with splitting large file based on pattern

Hi Experts, I have to split huge file based on the pattern to create smaller files. The pattern which is expected in the file is: Master..... First... second.... second... third.. third... Master... First.. second... third... Master... First... second.. second.. second..... (2 Replies)
Discussion started by: saisanthi
2 Replies

6. Shell Programming and Scripting

split XML file into multiple files based on pattern

Hello, I am using awk to split a file into multiple files using command: nawk '{ if ( $1 == "<process" ) { n=split($2, arr, "\""); file=arr } print > file }' processes.xml <process name="Process1.process"> ... (3 Replies)
Discussion started by: chiru_h
3 Replies

7. Shell Programming and Scripting

Splitting large file into multiple files in unix based on pattern

I need to write a shell script for below scenario My input file has data in format: qwerty0101TWE 12345 01022005 01022005 datainala alanfernanded 26 qwerty0101mXZ 12349 01022005 06022008 datainalb johngalilo 28 qwerty0101TWE 12342 01022005 07022009 datainalc hitalbert 43 qwerty0101CFG 12345... (19 Replies)
Discussion started by: jimmy12
19 Replies

8. Shell Programming and Scripting

Split a file into multiple files based on the input pattern

I have a file with lines something like. ...... 123_start ...... ....... 123_end .... ..... 456_start ...... ..... 456_end .... ..... 789_start .... .... 789_end (6 Replies)
Discussion started by: abinash
6 Replies

9. Shell Programming and Scripting

Split large file based on last digit from a column

Hello, What's the best way to split a large into multiple files based on the last digit in the first column. input file: f 2738483300000x0y03772748378831x1y13478378358383x2y23743878383802x3y33787828282820x4y43748838383881x5y5 Desired Output: f0 3738483300000x0y03787828282820x4y4 f1... (9 Replies)
Discussion started by: alain.kazan
9 Replies

10. Shell Programming and Scripting

split large file based on field criteria

I have a file containing date/time sorted data of the form ... 2009/06/10,20:59:59.950,XAG/USD,Q,1,1115, 14.3025,100,1,1 2009/06/10,20:59:59.950,XAG/USD,Q,1,1116, 14.3026,125,1,1 2009/06/10,20:59:59.950,XAG/USD,R,0,0, , 0,0,0 2009/06/10,20:59:59.950,XAG/USD,R,1,0, 14.1910,100,1,1... (6 Replies)
Discussion started by: asriva
6 Replies
Login or Register to Ask a Question