Sponsored Content
Top Forums UNIX for Dummies Questions & Answers Split a file with no pattern -- Split, Csplit, Awk Post 302151805 by drl on Monday 17th of December 2007 11:57:24 AM
Old 12-17-2007
Hi.
Quote:
Originally Posted by radoulov
...
If you want the number of files/parts,
it's important if the total number of lines in the bigfile is known a priori
(otherwise it will be slower, because we have to read it twice: first to get
the number of lines, and then again to split it).
This does not appear to need absolute precision. You can read the first 100 (or few hundreds) of lines, then seek to near the end, read another 100 or so until the end, then calculate the mean length of the lines you saw. Obtaining the length of the file is cheap -- Linux has a stat command, but one could cut the length out from "ls -l". Divide the length in bytes by the estimated mean length and you get an estimate of the number of lines.

If necessary, one could also read a section from the middle (or elsewhere) to increase the accuracy.

The key is using a seek for positioning, and stat for the length -- both are very fast ... cheers, drl
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Split files using Csplit

I have an excel file with more than 65K records... Since excel does not take more than 65K records i wan to split the file and send it as two excel files... Could some help me how to use the csplit by specifiying the no of records (7 Replies)
Discussion started by: savitha
7 Replies

2. Shell Programming and Scripting

Split a file based on pattern in awk, grep, sed or perl

Hi All, Can someone please help me write a script for the following requirement in awk, grep, sed or perl. Buuuu xxx bbb Kmmmm rrr ssss uuuu Kwwww zzzz ccc Roooowwww eeee Bxxxx jjjj dddd Kuuuu eeeee nnnn Rpppp cccc vvvv cccc Rhhhhhhyyyy tttt Lhhhh rrrrrssssss Bffff mmmm iiiii Ktttt... (5 Replies)
Discussion started by: kumarn
5 Replies

3. Shell Programming and Scripting

Split a file based on a pattern

Dear all, I have a large file which is composed of 8000 frames, what i would like to do is split the file into 8000 single files names file.pdb.1, file.pdb.2 etc etc each frame in the large file is seperated by a "ENDMDL" flag so my thinking is to use this flag a a point to split the files... (4 Replies)
Discussion started by: Mish_99
4 Replies

4. Shell Programming and Scripting

Split binary file with pattern

Hello! Have some problem with extract files from saved session. File contains any kind of special/printable characters. DATA NumberA DATA DATA Begin DATA1.1 DATA1.2 NumberB1 DATA1.3 DATA1.4 End DATA DATA DATA Begin DATA2.1 DATA2.2 NumberB2 DATA2.3 DATA2.4 End DATA DATA ... (4 Replies)
Discussion started by: vvild
4 Replies

5. Shell Programming and Scripting

Split File by Pattern with File Names in Source File... Awk?

Hi all, I'm pretty new to Shell scripting and I need some help to split a source text file into multiple files. The source has a row with pattern where the file needs to be split, and the pattern row also contains the file name of the destination for that specific piece. Here is an example: ... (2 Replies)
Discussion started by: cul8er
2 Replies

6. Shell Programming and Scripting

awk to split one field and print the last two fields within the split part.

Hello; I have a file consists of 4 columns separated by tab. The problem is the third fields. Some of the them are very long but can be split by the vertical bar "|". Also some of them do not contain the string "UniProt", but I could ignore it at this moment, and sort the file afterwards. Here is... (5 Replies)
Discussion started by: yifangt
5 Replies

7. Shell Programming and Scripting

Split a file based on pattern and size

Hello, I have a large file (2GB) that I would like to split based on pattern and size. I've used the following command to split the file (token is "HELLO") awk '/HELLO/{i++}{print > "file"i}' input.txt and the output is similar to the following (i included filesize in KB): 10 ... (2 Replies)
Discussion started by: jl487
2 Replies

8. Shell Programming and Scripting

split file by delimiter with csplit

Hello, I want to split a big file into smaller ones with certain "counts". I am aware this type of job has been asked quite often, but I posted again when I came to csplit, which may be simpler to solve the problem. Input file (fasta format): >seq1 agtcagtc agtcagtc ag >seq2 agtcagtcagtc... (8 Replies)
Discussion started by: yifangt
8 Replies

9. Shell Programming and Scripting

Split the file based on pattern

Hi , I have huge files around 400 mb, which has clob data and have diffeent scenarios: I am trying to pass scenario number as parameter and and get required modified file based on the scenario number and criteria. Scenario 1: file name : scenario_1.txt ... (2 Replies)
Discussion started by: sol_nov
2 Replies

10. UNIX for Advanced & Expert Users

Split one file to many based on pattern

Hello All, I have records in a file in a pattern A,B,B,B,B,K,A,B,B,K Is there any command or simple logic I can pull out records into multiple files based on A record? I want output as File1: A,B,B,B,B,K File2: A,B,B,K (9 Replies)
Discussion started by: deal1dealer
9 Replies
FMT(1)							    BSD General Commands Manual 						    FMT(1)

NAME
fmt -- simple text formatter SYNOPSIS
fmt [-cmnps] [-d chars] [-l num] [-t num] [goal [maximum] | -width | -w width] [file ...] DESCRIPTION
The fmt utility is a simple text formatter which reads the concatenation of input files (or standard input if none are given) and produces on standard output a version of its input with lines as close to the goal length as possible without exceeding the maximum. The goal length defaults to 65 and the maximum to 10 more than the goal length. Alternatively, a single width parameter can be specified either by prepend- ing a hyphen to it or by using -w. For example, ``fmt -w 72'', ``fmt -72'', and ``fmt 72 72'' all produce identical output. The spacing at the beginning of the input lines is preserved in the output, as are blank lines and interword spacing. Lines are joined or split only at white space; that is, words are never joined or hyphenated. The options are as follows: -c Center the text, line by line. In this case, most of the other options are ignored; no splitting or joining of lines is done. -m Try to format mail header lines contained in the input sensibly. -n Format lines beginning with a '.' (dot) character. Normally, fmt does not fill these lines, for compatibility with nroff(1). -p Allow indented paragraphs. Without the -p flag, any change in the amount of whitespace at the start of a line results in a new para- graph being begun. -s Collapse whitespace inside lines, so that multiple whitespace characters are turned into a single space. (Or, at the end of a sen- tence, a double space.) -d chars Treat the chars (and no others) as sentence-ending characters. By default the sentence-ending characters are full stop ('.'), ques- tion mark ('?') and exclamation mark ('!'). Remember that some characters may need to be escaped to protect them from your shell. -l number Replace multiple spaces with tabs at the start of each output line, if possible. Each number spaces will be replaced with one tab. The default is 8. If number is 0, spaces are preserved. -t number Assume that the input files' tabs assume number spaces per tab stop. The default is 8. The fmt utility is meant to format mail messages prior to sending, but may also be useful for other simple tasks. For instance, within vis- ual mode of the ex(1) editor (e.g., vi(1)) the command !}fmt will reformat a paragraph, evening the lines. ENVIRONMENT
The LANG, LC_ALL and LC_CTYPE environment variables affect the execution of fmt as described in environ(7). SEE ALSO
fold(1), mail(1), nroff(1) HISTORY
The fmt command appeared in 3BSD. The version described herein is a complete rewrite and appeared in FreeBSD 4.4. AUTHORS
Kurt Shoens Liz Allen (added goal length concept) Gareth McCaughan BUGS
The program was designed to be simple and fast - for more complex operations, the standard text processors are likely to be more appropriate. When the first line of an indented paragraph is very long (more than about twice the goal length), the indentation in the output can be wrong. The fmt utility is not infallible in guessing what lines are mail headers and what lines are not. BSD
August 2, 2004 BSD
All times are GMT -4. The time now is 03:44 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy