Sponsored Content
Top Forums Shell Programming and Scripting Breaking large file into small files Post 302937418 by rbatte1 on Thursday 5th of March 2015 11:34:36 AM
Old 03-05-2015
Well, gandolf989, that appears to be a poor suggestion given that split will do this all in a single operation. You are wasting IO and CPU resources so it will be quite slow. Also, how would you know when to stop writing code or running your loop? There is no point reinventing a process that works very well already and lets you customise the beginning and end of the output files if you need to.


If the splitting up dependant on any condition other than record count, then csplit may be the tool for you, however without some sample data and rules to follow it's impossible to really know what you need.

Please wrap all code, file, input & output/errors in CODE tags as it makes them far easier to read and preserves multiple spaces and long lines in case these are important.


Thanks, in advance,
Robin
This User Gave Thanks to rbatte1 For This Post:
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Splitting large file into small files

Hi, I need to split a large file into small files based on a string. At different palces in the large I have the string ^Job. I need to split the file into different files starting from ^Job to the last character before the next ^Job. Also all the small files should be automatically named.... (4 Replies)
Discussion started by: dncs
4 Replies

2. Shell Programming and Scripting

Split a file into 16 small files

Hi I want to split a file that has 'n' number of records into 16 small files. Can some one suggest me how to do this using Unix script? Thanks rrkk (10 Replies)
Discussion started by: rrkks
10 Replies

3. Shell Programming and Scripting

Split large file and add header and footer to each small files

I have one large file, after every 200 line i have to split the file and the add header and footer to each small file? It is possible to add different header and footer to each file? (7 Replies)
Discussion started by: ashish4422
7 Replies

4. Shell Programming and Scripting

script to splite large file to number of small files

Dear All, Could you please help me to split a file contain around 240,000,000 line to 4 files all equally likely , note that we need to maintain that the end of each file should started by start flage (MSISDN) and ended by end flag (End), also the number of the line between the... (10 Replies)
Discussion started by: ahmed.gad
10 Replies

5. Shell Programming and Scripting

Breaking one file into many files based on first column?

Hi, I have a file that looks like this (tab deliminited). MAT1 YKR2 3 MAT1 YMR1 2 MAT1 YFG2 2 MAT2 YLM4 4 MAT2 YHL2 1 BAR1 YKR2 3 BAR1 YFR1 4 BAR1 YMR1 1 What I want to do is break this file down into multiple files. So the result will look like this: File 1... (2 Replies)
Discussion started by: kylle345
2 Replies

6. Shell Programming and Scripting

Breaking the files as 10k recs. per file

Hi, I have a code as given below Set -A _Category="A\ B\ C" for _cat in ${_Category} do sed -e "s:<TABLE_NAME>:${_cat}:g" \ -e "s:<date>:${_dt}:g" \ ${_home}/skl/sq1.sql >> ${_dest}/del_${_dt}.sql fi ... (4 Replies)
Discussion started by: mr_manii
4 Replies

7. UNIX for Advanced & Expert Users

Splitting a file into small files

Hi Folks, Please help me in solving the problem. I want to write script in order to split a file into small pieces and send it automatically through mail. Ex. The file name is CALM*.txt . It is around 50 MB. I want to split the file into 20 MB 2-3 smaller files and send (like uuencode) it... (6 Replies)
Discussion started by: piyushbhashkar
6 Replies

8. UNIX for Dummies Questions & Answers

Breaking a fasta formatted file into multiple files containing each gene separately

Hey, I've been trying to break a massive fasta formatted file into files containing each gene separately. Could anyone help me? I've tried to use the following code but i've recieved errors every time: for i in *.rtf.out do awk '/^>/{f=++d".fasta"} {print > $i.out}' $i done (1 Reply)
Discussion started by: Ann Mc Cartney
1 Replies

9. Shell Programming and Scripting

Split a large array into small chunks

Hi, I need to split a large array "@sharedArray" into 10 small arrays. The arrays should be like @sharedArray1,@sharedArray2,@sharedArray3...so on.. Can anyone help me with the logic to do so :(:confused: (6 Replies)
Discussion started by: rkrish
6 Replies

10. UNIX for Beginners Questions & Answers

Split large file into 24 small files on one hour basis

I Have a large file with 24hrs log in the below format.i need to split the large file in to 24 small files on one hour based.i.e ex:from 09:55 to 10:55,10:55-11:55 can any one help me on this.! ... (20 Replies)
Discussion started by: Raghuram717
20 Replies
FMT(1)							    BSD General Commands Manual 						    FMT(1)

NAME
fmt -- simple text formatter SYNOPSIS
fmt [-cmnps] [-d chars] [-l num] [-t num] [goal [maximum] | -width | -w width] [file ...] DESCRIPTION
The fmt utility is a simple text formatter which reads the concatenation of input files (or standard input if none are given) and produces on standard output a version of its input with lines as close to the goal length as possible without exceeding the maximum. The goal length defaults to 65 and the maximum to 10 more than the goal length. Alternatively, a single width parameter can be specified either by prepend- ing a hyphen to it or by using -w. For example, ``fmt -w 72'', ``fmt -72'', and ``fmt 72 72'' all produce identical output. The spacing at the beginning of the input lines is preserved in the output, as are blank lines and interword spacing. Lines are joined or split only at white space; that is, words are never joined or hyphenated. The options are as follows: -c Center the text, line by line. In this case, most of the other options are ignored; no splitting or joining of lines is done. -m Try to format mail header lines contained in the input sensibly. -n Format lines beginning with a '.' (dot) character. Normally, fmt does not fill these lines, for compatibility with nroff(1). -p Allow indented paragraphs. Without the -p flag, any change in the amount of whitespace at the start of a line results in a new para- graph being begun. -s Collapse whitespace inside lines, so that multiple whitespace characters are turned into a single space. (Or, at the end of a sen- tence, a double space.) -d chars Treat the chars (and no others) as sentence-ending characters. By default the sentence-ending characters are full stop ('.'), ques- tion mark ('?') and exclamation mark ('!'). Remember that some characters may need to be escaped to protect them from your shell. -l number Replace multiple spaces with tabs at the start of each output line, if possible. Each number spaces will be replaced with one tab. The default is 8. If number is 0, spaces are preserved. -t number Assume that the input files' tabs assume number spaces per tab stop. The default is 8. The fmt utility is meant to format mail messages prior to sending, but may also be useful for other simple tasks. For instance, within vis- ual mode of the ex(1) editor (e.g., vi(1)) the command !}fmt will reformat a paragraph, evening the lines. SEE ALSO
mail(1), nroff(1) HISTORY
The fmt command appeared in 3BSD. The version described herein is a complete rewrite and appeared in FreeBSD 4.4. AUTHORS
Kurt Shoens Liz Allen (added goal length concept) Gareth McCaughan BUGS
The program was designed to be simple and fast - for more complex operations, the standard text processors are likely to be more appropriate. When the first line of an indented paragraph is very long (more than about twice the goal length), the indentation in the output can be wrong. The fmt utility is not infallible in guessing what lines are mail headers and what lines are not. BSD
June 25, 2000 BSD
All times are GMT -4. The time now is 01:00 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy