Split large file into smaller files without disturbing the entry chunks
Dears,
Need you help with the below file manipulation. I want to split the file into 8 smaller files but without cutting/disturbing the entries (meaning every small file should start with a entry and end with an empty line). It will be helpful if you can provide a one liner command for this and also help me understand the command u give. I'm not very good at this.
Input_file:- (With more than 10 Million such chunks separated by empty line)
small_file1:- small_file2:-
Thank you in advance.
Kamesh G
Last edited by RudiC; 04-28-2018 at 02:45 AM..
Reason: Added CODE tags for file data
hi all
im new to this forum..excuse me if anythng wrong.
I have a file containing 600 MB data in that. when i do parse the data in perl program im getting out of memory error.
so iam planning to split the file into smaller files and process one by one.
can any one tell me what is the code... (1 Reply)
I have a file with a simple list of ids. 750,000 rows. I have to break it down into multiple 50,000 row files to submit in a batch process.. Is there an easy script I could write to accomplish this task? (2 Replies)
hi Guys
i need some help here..
i have a file which has > 800,000 lines in it. I need to split this file into smaller files with 25000 lines each.
please help
thanks (1 Reply)
Hi,
I have a big text file with m columns and n rows. The format is like:
STF123450001000200030004STF123450005000600070008STF123450009001000110012
STF234560345002208330154STF234590705620600070080STF234567804094562357688
STF356780001000200030004STF356780005000600070080STF356780800094562657687... (2 Replies)
Help needed urgently please.
I have a large file - a few hundred thousand lines.
Sample
CP START ACCOUNT
1234556
name 1
CP END ACCOUNT
CP START ACCOUNT
2224444
name 1
CP END ACCOUNT
CP START ACCOUNT
333344444
name 1
CP END ACCOUNT
I need to split this file each time "CP START... (7 Replies)
Hi,
I need to split a large array "@sharedArray" into 10 small arrays.
The arrays should be like @sharedArray1,@sharedArray2,@sharedArray3...so on..
Can anyone help me with the logic to do so :(:confused: (6 Replies)
I will simplify the explaination a bit, I need to parse through a 87m file -
I have a single text file in the form of :
<NAME>house........
SOMETEXT
SOMETEXT
SOMETEXT
.
.
.
.
</script>
MORETEXT
MORETEXT
.
.
. (6 Replies)
I have a perl script which splits a large file into chunks.The script is given below
use strict;
use warnings;
open (FH, "<monolingual.txt") or die "Could not open source file. $!";
my $i = 0;
while (1) {
my $chunk;
print "process part $i\n";
open(OUT, ">part$i.log") or die "Could... (4 Replies)
Discussion started by: gimley
4 Replies
LEARN ABOUT OPENDARWIN
split
SPLIT(1) BSD General Commands Manual SPLIT(1)NAME
split -- split a file into pieces
SYNOPSIS
split [-a suffix_length] [-b byte_count[k|m]] [-l line_count] [-p pattern] [file [name]]
DESCRIPTION
The split utility reads the given file and breaks it up into files of 1000 lines each. If file is a single dash ('-') or absent, split reads
from the standard input.
The options are as follows:
-a Use suffix_length letters to form the suffix of the file name.
-b Create smaller files byte_count bytes in length. If ``k'' is appended to the number, the file is split into byte_count kilobyte
pieces. If ``m'' is appended to the number, the file is split into byte_count megabyte pieces.
-l Create smaller files n lines in length.
-p pattern
The file is split whenever an input line matches pattern, which is interpreted as an extended regular expression. The matching line
will be the first line of the next output file. This option is incompatible with the -b and -l options.
If additional arguments are specified, the first is used as the name of the input file which is to be split. If a second additional argument
is specified, it is used as a prefix for the names of the files into which the file is split. In this case, each file into which the file is
split is named by the prefix followed by a lexically ordered suffix using suffix_length characters in the range ``a-z''. If -a is not speci-
fied, two letters are used as the suffix.
If the name argument is not specified, the file is split into lexically ordered files named with prefixes in the range of ``x-z'' and with
suffixes as above.
SEE ALSO csplit(1), re_format(7)STANDARDS
The split utility conforms to IEEE Std 1003.1-2001 (``POSIX.1'').
HISTORY
A split command appeared in Version 3 AT&T UNIX.
BUGS
For historical reasons, if you specify name, split can only create 676 separate files. The default naming convention allows 2028 separate
files. The -a option can be used to work around this limitation.
The maximum line length for matching patterns is 65536.
BSD April 16, 1994 BSD