Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Splitting 12M line file

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Splitting 12M line file
# 1  
Old 05-05-2011
Splitting 12M line file
I have a 12M line file that I need to split into smaller files but I'm getting a

csplit: virtual memory exhausted error

How can I quickly split this file?

Code:
This is sze
-r-xr-xr-x+ 1 VERGE  Domain Users  1158288000 May  4 16:31 inputfile
 
This is the command I've tried... 
csplit -ks -f splitfile inputfile 1000000 {30}

Thank you!!!!
# 2  
Old 05-05-2011
How, exactly, are you trying to split the file? Fixed number of lines, break apart on pattern x, or what? The arguments '1000000 {30}' don't make much sense to me.

---------- Post updated at 03:21 PM ---------- Previous update was at 03:15 PM ----------

Also, what is a "line file"? Are the lines fixed sizes?
# 3  
Old 05-05-2011
Hi Corona688!

fixed number of lines ... splitting every 1M lines

I'm willing to split it a different way though

---------- Post updated at 02:32 PM ---------- Previous update was at 02:31 PM ----------

I meant there are 12 million lines in the file and the length of the lines arent fixed
# 4  
Old 05-05-2011
poor-man's split:
Code:
nawk -v chunk=1000000 '!((FNR-1)%chunk){close(f);f=FILENAME "_" ++c}{print >> f}' myHugeFile

# 5  
Old 05-05-2011
Code:
awk '{print > int(NR/1000000 )  ".txt" }' infile

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk issue splitting a fixed-width file containing line feed in data

Hi Forum. I have the following script that splits a large fixed-width file into smaller multiple fixed-width files based on input segment type. The main command in the script is: awk -v search_col_pos=$search_col_pos -v search_str_len=$search_str_len -v segment_type="$segment_type"... (8 Replies)
Discussion started by: pchang
8 Replies

2. Shell Programming and Scripting

Splitting XML file on basis of line number into multiple file

Hi All, I have more than half million lines of XML file , wanted to split in four files in a such a way that top 7 lines should be present in each file on top and bottom line of should be present in each file at bottom. from the 8th line actual record starts and each record contains 15 lines... (14 Replies)
Discussion started by: ajju
14 Replies

3. UNIX for Dummies Questions & Answers

Splitting the file basis of Line Number

Can u pls advise the unix command as I have a file which contain the records in the below format 333434 435435 435443 434543 343536 Now the total line count is 89380 , now i want to create a separate I am trying to split my large big file into small bits using the line... (2 Replies)
Discussion started by: punpun66
2 Replies

4. Shell Programming and Scripting

Splitting Single line into multiple line

Hi All, I am reading a line from a file and writing it to other file. Whenever I got a particular line then I want that line to be splited into 4 line and written it to new file. e.g My line is U_ABC connector3 pin24E connector4 pin25E connector5 pin26E connector6 pin27E connector7... (2 Replies)
Discussion started by: diehard
2 Replies

5. Shell Programming and Scripting

Splitting a file based on line number

Hi I have a file with over a million lines (rows) and I want to split everything from 500,000 to a million into another file (to make the file smaller). Is there a simple command for this? Thank you Phil (4 Replies)
Discussion started by: phil_heath
4 Replies

6. Shell Programming and Scripting

Splitting file based on line numbers

Hello friends, Is there any way to split file from n to n+6 into 1 file and (n+7) to (n+16) into other file etc. f.e I have source pipe delimated file with 20 lines and i need to split 1-6 in file1 and 7-16 in file2 and 17-20 in file 3 I need to split into fixed number of file like 4 files... (2 Replies)
Discussion started by: Rizzu155
2 Replies

7. Shell Programming and Scripting

splitting a huge line of file into multiple lines with fixed number of columns

Hi, I have a huge file with a single line. But I want to break that line into lines of with each line having five columns. My file is like this: code: "hi","there","how","are","you?","It","was","great","working","with","you.","hope","to","work","you." I want it like this: code:... (1 Reply)
Discussion started by: rajsharma
1 Replies

8. UNIX for Dummies Questions & Answers

Splitting a line

I have a series of .txt files, that contain lines of text separated by the following string ==================== In some of the .txt files, the string ends with the word Document, leaving the string ==================Document. I would like to be able to split any such line and move the word... (16 Replies)
Discussion started by: spindoctor
16 Replies

9. UNIX for Dummies Questions & Answers

Line Splitting

Hi, I want to make a new line between for eg: 0102030405 to make it look like: 01 02 03 04 05 by using sed commands. I'm just wondering how would I do this? p.s i'm new to this unix programming. (7 Replies)
Discussion started by: evoGage
7 Replies

10. UNIX for Dummies Questions & Answers

Splitting a line up

9600012301F TF02FT T03FFTF04TF 05T FF06TTTT I have to split this line up into this 96000123 01F TF 02FT T 03FFTF 04TF 05T FF 06TTTT Can anyone please Help?? (1 Reply)
Discussion started by: lilas
1 Replies
Login or Register to Ask a Question