Sponsored Content
Top Forums UNIX for Dummies Questions & Answers splitting the large file into smaller files Post 302258933 by vsnreddy on Sunday 16th of November 2008 05:35:42 PM
Old 11-16-2008
splitting the large file into smaller files

hi all
im new to this forum..excuse me if anythng wrong.

I have a file containing 600 MB data in that. when i do parse the data in perl program im getting out of memory error.

so iam planning to split the file into smaller files and process one by one.

can any one tell me what is the code will do for spliting the bigger file.

what is the better way to achieve this ?
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Splitting large file into small files

Hi, I need to split a large file into small files based on a string. At different palces in the large I have the string ^Job. I need to split the file into different files starting from ^Job to the last character before the next ^Job. Also all the small files should be automatically named.... (4 Replies)
Discussion started by: dncs
4 Replies

2. Shell Programming and Scripting

Help with splitting a large text file into smaller ones

Hi Everyone, I am using a centos 5.2 server as an sflow log collector on my network. Currently I am using inmons free sflowtool to collect the packets sent by my switches. I have a bash script running on an infinate loop to stop and start the log collection at set intervals - currently one... (2 Replies)
Discussion started by: lord_butler
2 Replies

3. UNIX for Dummies Questions & Answers

multiple smaller files from one large file

I have a file with a simple list of ids. 750,000 rows. I have to break it down into multiple 50,000 row files to submit in a batch process.. Is there an easy script I could write to accomplish this task? (2 Replies)
Discussion started by: rtroscianecki
2 Replies

4. Shell Programming and Scripting

Divide large data files into smaller files

Hello everyone! I have 2 types of files in the following format: 1) *.fa >1234 ...some text... >2345 ...some text... >3456 ...some text... . . . . 2) *.info >1234 (7 Replies)
Discussion started by: ad23
7 Replies

5. Shell Programming and Scripting

Splitting large file into multiple files in unix based on pattern

I need to write a shell script for below scenario My input file has data in format: qwerty0101TWE 12345 01022005 01022005 datainala alanfernanded 26 qwerty0101mXZ 12349 01022005 06022008 datainalb johngalilo 28 qwerty0101TWE 12342 01022005 07022009 datainalc hitalbert 43 qwerty0101CFG 12345... (19 Replies)
Discussion started by: jimmy12
19 Replies

6. Shell Programming and Scripting

Splitting a file into several smaller files using perl

Hi, I'm trying to split a large file into several smaller files the script will have two input arguments argument1=filename and argument2=no of files to be split. In my large input file I have a header followed by 100009 records The first line is a header; I want this header in all my... (9 Replies)
Discussion started by: ramky79
9 Replies

7. Shell Programming and Scripting

Help needed - Split large file into smaller files based on pattern match

Help needed urgently please. I have a large file - a few hundred thousand lines. Sample CP START ACCOUNT 1234556 name 1 CP END ACCOUNT CP START ACCOUNT 2224444 name 1 CP END ACCOUNT CP START ACCOUNT 333344444 name 1 CP END ACCOUNT I need to split this file each time "CP START... (7 Replies)
Discussion started by: frustrated1
7 Replies

8. Shell Programming and Scripting

Sed: Splitting A large File into smaller files based on recursive Regular Expression match

I will simplify the explaination a bit, I need to parse through a 87m file - I have a single text file in the form of : <NAME>house........ SOMETEXT SOMETEXT SOMETEXT . . . . </script> MORETEXT MORETEXT . . . (6 Replies)
Discussion started by: sumguy
6 Replies

9. UNIX for Beginners Questions & Answers

Split large file into smaller files without disturbing the entry chunks

Dears, Need you help with the below file manipulation. I want to split the file into 8 smaller files but without cutting/disturbing the entries (meaning every small file should start with a entry and end with an empty line). It will be helpful if you can provide a one liner command for this... (12 Replies)
Discussion started by: Kamesh G
12 Replies

10. Shell Programming and Scripting

Splitting a text file into smaller files with awk, how to create a different name for each new file

Hello, I have some large text files that look like, putrescine Mrv1583 01041713302D 6 5 0 0 0 0 999 V2000 2.0928 -0.2063 0.0000 N 0 0 0 0 0 0 0 0 0 0 0 0 5.6650 0.2063 0.0000 N 0 0 0 0 0 0 0 0 0 0 0 0 3.5217 ... (3 Replies)
Discussion started by: LMHmedchem
3 Replies
splitmail(1)						      General Commands Manual						      splitmail(1)

NAME
splitmail - Split a large mail message into MIME-compliant partial messages SYNOPSIS
splitmail [-d] [-v] [-s splitsize] [-p prefix] [-i id-suffix] [file-name] DESCRIPTION
The splitmail program will take an email message and break it up into smaller pieces using the "message/partial" type defined by MIME, the proposed Internet standard for multimedia mail formats. By default it will take the message either from standard input or the named file, and will produce a set of partial message files with names like "/tmp/split.1" for the first part, and so on. The prefix "/tmp/split." can be overridden using the "-p" option. If the -d option is specified, the mail will actually be delivered. If -v is specified, the verbose flag will be passed to sendmail. The -i option can be used to make splitmail generate the pieces with similar (but not identical) message-id fields, in a format which allows them to be easily correlated with one another and which end with the suffix provided on the command line after -i. The default chunk size for spliting messages is 250000 at most sites, though this is also a compile-time option. This can be overriden with the -s switch, or with the environment variable SPLITSIZE. Messages smaller than the chunk size will not be turned into partial messages, but will be written to a single file or delivered as a sin- gle message. ENVIRONMENT
SPLITSIZE overrides the default chunk size. Setting SPLITSIZE to, say, 4000000 will effectively ensure that your messages are unlikely ever to be split, but it may cause them to be rejected by some mail transport software. SEE ALSO
mailto(1), metamail(1) BUGS
If the size of the input is just on the fencepost, and if it is coming from a file rather than standard input, splitmail will sometimes estimate the number of parts wrong and will have to write out an extra part. This is harmless but annoying. It is especially annoying if the estimate was 2 but the real number was 1. COPYRIGHT
Copyright (c) 1992 Bell Communications Research, Inc. (Bellcore) Permission to use, copy, modify, and distribute this material for any purpose and without fee is hereby granted, provided that the above copyright notice and this permission notice appear in all copies, and that the name of Bellcore not be used in advertising or publicity pertaining to this material without the specific, prior written permission of an authorized representative of Bellcore. BELLCORE MAKES NO REPRESENTATIONS ABOUT THE ACCURACY OR SUITABILITY OF THIS MATERIAL FOR ANY PURPOSE. IT IS PROVIDED "AS IS", WITHOUT ANY EXPRESS OR IMPLIED WARRANTIES. AUTHOR
Nathaniel S. Borenstein, Bellcore Release 1 splitmail(1)
All times are GMT -4. The time now is 06:37 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy