12-17-2007
Hi.
Quote:
Originally Posted by
radoulov
...
If you want the number of files/parts,
it's important if the total number of lines in the bigfile is known a priori
(otherwise it will be slower, because we have to read it twice: first to get
the number of lines, and then again to split it).
This does not appear to need absolute precision. You can read the first 100 (or few hundreds) of lines, then seek to near the end, read another 100 or so until the end, then calculate the mean length of the lines you saw. Obtaining the length of the file is cheap -- Linux has a
stat command, but one could cut the length out from "ls -l". Divide the length in bytes by the estimated mean length and you get an estimate of the number of lines.
If necessary, one could also read a section from the middle (or elsewhere) to increase the accuracy.
The key is using a seek for positioning, and stat for the length -- both are very fast ... cheers, drl
10 More Discussions You Might Find Interesting
1. UNIX for Dummies Questions & Answers
I have an excel file with more than 65K records... Since excel does not take more than 65K records i wan to split the file and send it as two excel files... Could some help me how to use the csplit by specifiying the no of records (7 Replies)
Discussion started by: savitha
7 Replies
2. Shell Programming and Scripting
Hi All,
Can someone please help me write a script for the following requirement in awk, grep, sed or perl.
Buuuu xxx bbb
Kmmmm rrr ssss uuuu
Kwwww zzzz ccc
Roooowwww eeee
Bxxxx jjjj dddd
Kuuuu eeeee nnnn
Rpppp cccc vvvv cccc
Rhhhhhhyyyy tttt
Lhhhh rrrrrssssss
Bffff mmmm iiiii
Ktttt... (5 Replies)
Discussion started by: kumarn
5 Replies
3. Shell Programming and Scripting
Dear all,
I have a large file which is composed of 8000 frames, what i would like to do is split the file into 8000 single files names file.pdb.1, file.pdb.2 etc etc
each frame in the large file is seperated by a "ENDMDL" flag so my thinking is to use this flag a a point to split the files... (4 Replies)
Discussion started by: Mish_99
4 Replies
4. Shell Programming and Scripting
Hello!
Have some problem with extract files from saved session.
File contains any kind of special/printable characters.
DATA NumberA DATA
DATA Begin
DATA1.1
DATA1.2 NumberB1 DATA1.3
DATA1.4
End DATA
DATA
DATA Begin
DATA2.1
DATA2.2 NumberB2 DATA2.3
DATA2.4
End DATA
DATA
... (4 Replies)
Discussion started by: vvild
4 Replies
5. Shell Programming and Scripting
Hi all,
I'm pretty new to Shell scripting and I need some help to split a source text file into multiple files. The source has a row with pattern where the file needs to be split, and the pattern row also contains the file name of the destination for that specific piece. Here is an example:
... (2 Replies)
Discussion started by: cul8er
2 Replies
6. Shell Programming and Scripting
Hello;
I have a file consists of 4 columns separated by tab. The problem is the third fields. Some of the them are very long but can be split by the vertical bar "|". Also some of them do not contain the string "UniProt", but I could ignore it at this moment, and sort the file afterwards. Here is... (5 Replies)
Discussion started by: yifangt
5 Replies
7. Shell Programming and Scripting
Hello, I have a large file (2GB) that I would like to split based on pattern and size.
I've used the following command to split the file (token is "HELLO")
awk '/HELLO/{i++}{print > "file"i}' input.txt
and the output is similar to the following (i included filesize in KB):
10 ... (2 Replies)
Discussion started by: jl487
2 Replies
8. Shell Programming and Scripting
Hello,
I want to split a big file into smaller ones with certain "counts". I am aware this type of job has been asked quite often, but I posted again when I came to csplit, which may be simpler to solve the problem.
Input file (fasta format):
>seq1
agtcagtc
agtcagtc
ag
>seq2
agtcagtcagtc... (8 Replies)
Discussion started by: yifangt
8 Replies
9. Shell Programming and Scripting
Hi ,
I have huge files around 400 mb, which has clob data and have diffeent scenarios:
I am trying to pass scenario number as parameter and and get required modified file based on the scenario number and criteria.
Scenario 1:
file name : scenario_1.txt
... (2 Replies)
Discussion started by: sol_nov
2 Replies
10. UNIX for Advanced & Expert Users
Hello All,
I have records in a file in a pattern A,B,B,B,B,K,A,B,B,K
Is there any command or simple logic I can pull out records into multiple files based on A record? I want output as
File1: A,B,B,B,B,K
File2: A,B,B,K (9 Replies)
Discussion started by: deal1dealer
9 Replies
fmt(1) General Commands Manual fmt(1)
NAME
fmt - Formats mail messages prior to sending
SYNOPSIS
fmt [-width] file...
DESCRIPTION
The fmt command reads the input file or files, or standard input if no files are specified, and writes to standard output a version of the
input with lines of a length as close as possible to width columns. (Because fmt is internationalized software, the number of display col-
umns is not necessarily equivalent to the number of bytes.)
The fmt command both joins and splits lines to achieve the desired width, but words are never joined or split; spaces are always preserved,
and lines are split at spaces only. In effect, fmt ignores newline characters in the input and wraps words to make lines a close as possi-
ble to width columns, resulting in individual lines of varying length but a consistent (new) text width overall. Because blank lines are
always preserved, fmt does not merge paragraphs separated by blank lines.
If you specify more than one file, the files are concatenated as input to fmt. If you do not specify -width, the default line length is 72
columns. Spacing at the beginning of input lines is always preserved in the output.
The fmt command is generally used to format mail messages to improve their appearance before they are sent. It may also be useful, how-
ever, for other simple formatting tasks. For example, when you are using vi, you can use the command :%!fmt -60 to reformat your text so
that all lines are approximately 60 columns long.
NOTES
The fmt command is a fast, simple formatting program. Standard text editing programs are more appropriate than fmt for complex formatting
operations. Do not use the fmt command if the message contains embedded messages or preformatted information from other files. This com-
mand formats the heading information in embedded messages and may change the format of preformatted information.
EXAMPLES
file1 contains these lines:
Australia is an island-continent, home to many very interesting plants and animals.
To reformat this text to a narrower width, enter: fmt -30 file1
This results in the following, displayed on your screen: Australia is an island-continent, home to many very interesting plants and
animals.
To make file1 wider, enter: fmt -60 file1
This results in: Australia is an island-continent, home to many very interesting plants and animals. To format a message you have
created with the mailx editor, at the left margin enter: ~|fmt
After you enter the command, your message is formatted, in this case to the default line length of 72 columns, and the word continue
is displayed to indicate that you can enter more information or send your message.
SEE ALSO
Commands: mail(1), mailx(1), vi(1)
fmt(1)