11-16-2008
splitting the large file into smaller files
hi all
im new to this forum..excuse me if anythng wrong.
I have a file containing 600 MB data in that. when i do parse the data in perl program im getting out of memory error.
so iam planning to split the file into smaller files and process one by one.
can any one tell me what is the code will do for spliting the bigger file.
what is the better way to achieve this ?
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Hi,
I need to split a large file into small files based on a string.
At different palces in the large I have the string ^Job.
I need to split the file into different files starting from ^Job to the last character before the next ^Job.
Also all the small files should be automatically named.... (4 Replies)
Discussion started by: dncs
4 Replies
2. Shell Programming and Scripting
Hi Everyone,
I am using a centos 5.2 server as an sflow log collector on my network. Currently I am using inmons free sflowtool to collect the packets sent by my switches. I have a bash script running on an infinate loop to stop and start the log collection at set intervals - currently one... (2 Replies)
Discussion started by: lord_butler
2 Replies
3. UNIX for Dummies Questions & Answers
I have a file with a simple list of ids. 750,000 rows. I have to break it down into multiple 50,000 row files to submit in a batch process.. Is there an easy script I could write to accomplish this task? (2 Replies)
Discussion started by: rtroscianecki
2 Replies
4. Shell Programming and Scripting
Hello everyone!
I have 2 types of files in the following format:
1) *.fa
>1234
...some text...
>2345
...some text...
>3456
...some text...
.
.
.
.
2) *.info
>1234 (7 Replies)
Discussion started by: ad23
7 Replies
5. Shell Programming and Scripting
I need to write a shell script for below scenario
My input file has data in format:
qwerty0101TWE 12345 01022005 01022005 datainala alanfernanded 26
qwerty0101mXZ 12349 01022005 06022008 datainalb johngalilo 28
qwerty0101TWE 12342 01022005 07022009 datainalc hitalbert 43
qwerty0101CFG 12345... (19 Replies)
Discussion started by: jimmy12
19 Replies
6. Shell Programming and Scripting
Hi,
I'm trying to split a large file into several smaller files
the script will have two input arguments argument1=filename and argument2=no of files to be split.
In my large input file I have a header followed by 100009 records
The first line is a header; I want this header in all my... (9 Replies)
Discussion started by: ramky79
9 Replies
7. Shell Programming and Scripting
Help needed urgently please.
I have a large file - a few hundred thousand lines.
Sample
CP START ACCOUNT
1234556
name 1
CP END ACCOUNT
CP START ACCOUNT
2224444
name 1
CP END ACCOUNT
CP START ACCOUNT
333344444
name 1
CP END ACCOUNT
I need to split this file each time "CP START... (7 Replies)
Discussion started by: frustrated1
7 Replies
8. Shell Programming and Scripting
I will simplify the explaination a bit, I need to parse through a 87m file -
I have a single text file in the form of :
<NAME>house........
SOMETEXT
SOMETEXT
SOMETEXT
.
.
.
.
</script>
MORETEXT
MORETEXT
.
.
. (6 Replies)
Discussion started by: sumguy
6 Replies
9. UNIX for Beginners Questions & Answers
Dears,
Need you help with the below file manipulation. I want to split the file into 8 smaller files but without cutting/disturbing the entries (meaning every small file should start with a entry and end with an empty line). It will be helpful if you can provide a one liner command for this... (12 Replies)
Discussion started by: Kamesh G
12 Replies
10. Shell Programming and Scripting
Hello,
I have some large text files that look like,
putrescine
Mrv1583 01041713302D
6 5 0 0 0 0 999 V2000
2.0928 -0.2063 0.0000 N 0 0 0 0 0 0 0 0 0 0 0 0
5.6650 0.2063 0.0000 N 0 0 0 0 0 0 0 0 0 0 0 0
3.5217 ... (3 Replies)
Discussion started by: LMHmedchem
3 Replies
LEARN ABOUT MOJAVE
splitmail
splitmail(1) General Commands Manual splitmail(1)
NAME
splitmail - Split a large mail message into MIME-compliant partial messages
SYNOPSIS
splitmail [-d] [-v] [-s splitsize] [-p prefix] [-i id-suffix] [file-name]
DESCRIPTION
The splitmail program will take an email message and break it up into smaller pieces using the "message/partial" type defined by MIME, the
proposed Internet standard for multimedia mail formats.
By default it will take the message either from standard input or the named file, and will produce a set of partial message files with
names like "/tmp/split.1" for the first part, and so on. The prefix "/tmp/split." can be overridden using the "-p" option.
If the -d option is specified, the mail will actually be delivered. If -v is specified, the verbose flag will be passed to sendmail.
The -i option can be used to make splitmail generate the pieces with similar (but not identical) message-id fields, in a format which
allows them to be easily correlated with one another and which end with the suffix provided on the command line after -i.
The default chunk size for spliting messages is 250000 at most sites, though this is also a compile-time option. This can be overriden
with the -s switch, or with the environment variable SPLITSIZE.
Messages smaller than the chunk size will not be turned into partial messages, but will be written to a single file or delivered as a sin-
gle message.
ENVIRONMENT
SPLITSIZE overrides the default chunk size. Setting SPLITSIZE to, say, 4000000 will effectively ensure that your messages are unlikely
ever to be split, but it may cause them to be rejected by some mail transport software.
SEE ALSO
mailto(1), metamail(1)
BUGS
If the size of the input is just on the fencepost, and if it is coming from a file rather than standard input, splitmail will sometimes
estimate the number of parts wrong and will have to write out an extra part. This is harmless but annoying. It is especially annoying if
the estimate was 2 but the real number was 1.
COPYRIGHT
Copyright (c) 1992 Bell Communications Research, Inc. (Bellcore)
Permission to use, copy, modify, and distribute this material for any purpose and without fee is hereby granted, provided that the above
copyright notice and this permission notice appear in all copies, and that the name of Bellcore not be used in advertising or publicity
pertaining to this material without the specific, prior written permission of an authorized representative of Bellcore. BELLCORE MAKES NO
REPRESENTATIONS ABOUT THE ACCURACY OR SUITABILITY OF THIS MATERIAL FOR ANY PURPOSE. IT IS PROVIDED "AS IS", WITHOUT ANY EXPRESS OR IMPLIED
WARRANTIES.
AUTHOR
Nathaniel S. Borenstein, Bellcore
Release 1 splitmail(1)