Sponsored Content
Full Discussion: Splitting 12M line file
Top Forums Shell Programming and Scripting Splitting 12M line file Post 302520064 by verge on Thursday 5th of May 2011 04:57:44 PM
Old 05-05-2011
Splitting 12M line file

I have a 12M line file that I need to split into smaller files but I'm getting a

csplit: virtual memory exhausted error

How can I quickly split this file?

Code:
This is sze
-r-xr-xr-x+ 1 VERGE  Domain Users  1158288000 May  4 16:31 inputfile
 
This is the command I've tried... 
csplit -ks -f splitfile inputfile 1000000 {30}

Thank you!!!!
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Splitting a line up

9600012301F TF02FT T03FFTF04TF 05T FF06TTTT I have to split this line up into this 96000123 01F TF 02FT T 03FFTF 04TF 05T FF 06TTTT Can anyone please Help?? (1 Reply)
Discussion started by: lilas
1 Replies

2. UNIX for Dummies Questions & Answers

Line Splitting

Hi, I want to make a new line between for eg: 0102030405 to make it look like: 01 02 03 04 05 by using sed commands. I'm just wondering how would I do this? p.s i'm new to this unix programming. (7 Replies)
Discussion started by: evoGage
7 Replies

3. UNIX for Dummies Questions & Answers

Splitting a line

I have a series of .txt files, that contain lines of text separated by the following string ==================== In some of the .txt files, the string ends with the word Document, leaving the string ==================Document. I would like to be able to split any such line and move the word... (16 Replies)
Discussion started by: spindoctor
16 Replies

4. Shell Programming and Scripting

splitting a huge line of file into multiple lines with fixed number of columns

Hi, I have a huge file with a single line. But I want to break that line into lines of with each line having five columns. My file is like this: code: "hi","there","how","are","you?","It","was","great","working","with","you.","hope","to","work","you." I want it like this: code:... (1 Reply)
Discussion started by: rajsharma
1 Replies

5. Shell Programming and Scripting

Splitting file based on line numbers

Hello friends, Is there any way to split file from n to n+6 into 1 file and (n+7) to (n+16) into other file etc. f.e I have source pipe delimated file with 20 lines and i need to split 1-6 in file1 and 7-16 in file2 and 17-20 in file 3 I need to split into fixed number of file like 4 files... (2 Replies)
Discussion started by: Rizzu155
2 Replies

6. Shell Programming and Scripting

Splitting a file based on line number

Hi I have a file with over a million lines (rows) and I want to split everything from 500,000 to a million into another file (to make the file smaller). Is there a simple command for this? Thank you Phil (4 Replies)
Discussion started by: phil_heath
4 Replies

7. Shell Programming and Scripting

Splitting Single line into multiple line

Hi All, I am reading a line from a file and writing it to other file. Whenever I got a particular line then I want that line to be splited into 4 line and written it to new file. e.g My line is U_ABC connector3 pin24E connector4 pin25E connector5 pin26E connector6 pin27E connector7... (2 Replies)
Discussion started by: diehard
2 Replies

8. UNIX for Dummies Questions & Answers

Splitting the file basis of Line Number

Can u pls advise the unix command as I have a file which contain the records in the below format 333434 435435 435443 434543 343536 Now the total line count is 89380 , now i want to create a separate I am trying to split my large big file into small bits using the line... (2 Replies)
Discussion started by: punpun66
2 Replies

9. Shell Programming and Scripting

Splitting XML file on basis of line number into multiple file

Hi All, I have more than half million lines of XML file , wanted to split in four files in a such a way that top 7 lines should be present in each file on top and bottom line of should be present in each file at bottom. from the 8th line actual record starts and each record contains 15 lines... (14 Replies)
Discussion started by: ajju
14 Replies

10. Shell Programming and Scripting

awk issue splitting a fixed-width file containing line feed in data

Hi Forum. I have the following script that splits a large fixed-width file into smaller multiple fixed-width files based on input segment type. The main command in the script is: awk -v search_col_pos=$search_col_pos -v search_str_len=$search_str_len -v segment_type="$segment_type"... (8 Replies)
Discussion started by: pchang
8 Replies
csplit(1)							   User Commands							 csplit(1)

NAME
csplit - split files based on context SYNOPSIS
csplit [-ks] [-f prefix] [-n number] file arg1... argn DESCRIPTION
The csplit utility reads the file named by the file operand, writes all or part of that file into other files as directed by the arg oper- ands, and writes the sizes of the files. OPTIONS
The following options are supported: -f prefix Names the created files prefix00, prefix01, ..., prefixn. The default is xx00 ... xxn. If the prefix argument would create a file name exceeding 14 bytes, an error results. In that case, csplit exits with a diagnostic message and no files are created. -k Leaves previously created files intact. By default, csplit removes created files if an error occurs. -n number Uses number decimal digits to form filenames for the file pieces. The default is 2. -s Suppresses the output of file size messages. OPERANDS
The following operands are supported: file The path name of a text file to be split. If file is -, the standard input will be used. The operands arg1 ... argn can be a combination of the following: /rexp/[offset] Create a file using the content of the lines from the current line up to, but not including, the line that results from the evaluation of the regular expression with offset, if any, applied. The regular expression rexp must follow the rules for basic regular expressions. Regular expressions can include the use of '/' and '\%'. These forms must be properly quoted with single quotes, since "" is special to the shell. The optional offset must be a positive or negative integer value representing a number of lines. The integer value must be preceded by + or -. If the selection of lines from an offset expression of this type would create a file with zero lines, or one with greater than the number of lines left in the input file, the results are unspecified. After the section is created, the current line will be set to the line that results from the evaluation of the regular expression with any offset applied. The pattern match of rexp always is applied from the current line to the end of the file. %rexp%[offset] This operand is the same as /rexp/[offset], except that no file will be created for the selected section of the input file. line_no Create a file from the current line up to (but not including) the line number line_no. Lines in the file will be numbered starting at one. The current line becomes line_no. {num} Repeat operand. This operand can follow any of the operands described previously. If it follows a rexp type operand, that operand will be applied num more times. If it follows a line_no operand, the file will be split every line_no lines, num times, from that point. An error will be reported if an operand does not reference a line between the current position and the end of the file. USAGE
See largefile(5) for the description of the behavior of csplit when encountering files greater than or equal to 2 Gbyte (2^31 bytes). EXAMPLES
Example 1 Splitting and combining files This example creates four files, cobol00...cobol03. example% csplit -f cobol filename '/procedure division/' /par5./ /par16./ After editing the split files, they can be recombined as follows: example% cat cobol0[0-3] > filename This example overwrites the original file. Example 2 Splitting a file into equal parts This example splits the file at every 100 lines, up to 10,000 lines. The -k option causes the created files to be retained if there are less than 10,000 lines; however, an error message would still be printed. example% csplit -k filename 100 {99} Example 3 Creating a file for separate C routines If prog.c follows the normal C coding convention (the last line of a routine consists only of a } in the first character position), this example creates a file for each separate C routine (up to 21) in prog.c. example% csplit -k prog.c '%main(%' '/^}/+1' {20} ENVIRONMENT VARIABLES
See environ(5) for descriptions of the following environment variables that affect the execution of csplit: LANG, LC_ALL, LC_COLLATE, LC_CTYPE, LC_MESSAGES, and NLSPATH. EXIT STATUS
The following exit values are returned: 0 Successful completion. >0 An error occurred. ATTRIBUTES
See attributes(5) for descriptions of the following attributes: +-----------------------------+-----------------------------+ | ATTRIBUTE TYPE | ATTRIBUTE VALUE | +-----------------------------+-----------------------------+ |Availability |SUNWesu | +-----------------------------+-----------------------------+ |CSI |Enabled | +-----------------------------+-----------------------------+ |Interface Stability |Standard | +-----------------------------+-----------------------------+ SEE ALSO
sed(1), split(1), attributes(5), environ(5), largefile(5), standards(5) DIAGNOSTICS
The diagnostic messages are self-explanatory, except for the following: arg - out of range The given argument did not reference a line between the current position and the end of the file. SunOS 5.11 4 Dec 2003 csplit(1)
All times are GMT -4. The time now is 08:10 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy