08-11-2010
Split a large file
I have a 3 GB text file that I would like to split. How can I do this?
It's a giant comma-separated list of numbers. I would like to make it into about 20 files of ~100 MB each, with a custom header and footer. The file can only be split on commas, but they're plentiful.
Something like this:
file:
1,2,3,4,...,12500000,12500001,...
to
file1:
v=[1,2,3,4,...,12500000];
file2:
v=[12500001,...];
...
file20:
v=[250000001,...];
where the header is "v=[" and the footer is "];". (The comma between, e.g., 12500000 and 12500001 is dropped.)
Last edited by CRGreathouse; 08-11-2010 at 10:43 PM..
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Hi,
I have a large file(csv format) that I need to split into 2 files. The file looks something like
Original_file.txt
first name, family name, address
a, b, c,
d, e, f,
and so on for over 100,00 lines
I need to create two files from this one file. The condition is i need to ensure... (4 Replies)
Discussion started by: nbvcxzdz
4 Replies
2. Shell Programming and Scripting
Hi,
I have a large file with a repeating pattern in it. Now i want the file split into the block of patterns with a specified no. of lines in each file.
i.e. The file is like
1...
2...
2...
3...
1...
2...
3...
1...
2...
2...
2...
2...
2...
3...
where 1 is the start of the block... (5 Replies)
Discussion started by: sudhamacs
5 Replies
3. Shell Programming and Scripting
HI,
i've to split a large file which inputs seems like :
Input file name_file.txt
00001|AAAA|MAIL|DATEOFBIRTHT|.......
00001|AAAA|MAIL|DATEOFBIRTHT|.......
00002|BBBB|MAIL|DATEOFBIRTHT|.......
00002|BBBB|MAIL|DATEOFBIRTHT|.......
00003|CCCC|MAIL|DATEOFBIRTHT|.......... (1 Reply)
Discussion started by: AMARA
1 Replies
4. Shell Programming and Scripting
I have a file containing date/time sorted data of the form
...
2009/06/10,20:59:59.950,XAG/USD,Q,1,1115, 14.3025,100,1,1
2009/06/10,20:59:59.950,XAG/USD,Q,1,1116, 14.3026,125,1,1
2009/06/10,20:59:59.950,XAG/USD,R,0,0, , 0,0,0
2009/06/10,20:59:59.950,XAG/USD,R,1,0, 14.1910,100,1,1... (6 Replies)
Discussion started by: asriva
6 Replies
5. Shell Programming and Scripting
Hello Everyone,
I have a large file that needs to be split into many seperate files, however the text in between the blank lines need to be intact. The file looks like
SomeText
SomeText
SomeText
SomeOtherText
SomeOtherText
....
Since the number of lines of text are different for... (3 Replies)
Discussion started by: jwillis0720
3 Replies
6. Shell Programming and Scripting
Hello,
What's the best way to split a large into multiple files based on the last digit in the first column.
input file:
f
2738483300000x0y03772748378831x1y13478378358383x2y23743878383802x3y33787828282820x4y43748838383881x5y5
Desired Output:
f0
3738483300000x0y03787828282820x4y4
f1... (9 Replies)
Discussion started by: alain.kazan
9 Replies
7. UNIX for Dummies Questions & Answers
hi ,
I have a requirement
input file:
1 1111111111111 108
1 1111111111111 109
1 1111111111111 109
1 1111111111111 110
1 1111111111111 111
1 1111111111111 111
1 1111111111111 111
1 1111111111111 112
1 1111111111111 112
1 1111111111111 112
The output should be, (19 Replies)
Discussion started by: mechvijays
19 Replies
8. UNIX for Beginners Questions & Answers
Dear Users,
Appreciate your help if you could help me with splitting a large file > 1 million lines with sed or awk. below is the text in the file
input file.txt
scaffold1 928 929 C/T +
scaffold1 942 943 G/C +
scaffold1 959 960 C/T +... (6 Replies)
Discussion started by: kapr0001
6 Replies
9. UNIX for Advanced & Expert Users
Hi,
I have received a file which is 20 GB. We would like to split the file into 4 equal parts and process it to avoid memory issues.
If the record delimiter is unix new line, I could use split command either with option l or b.
The problem is that the line terminator is |##|
How to use... (5 Replies)
Discussion started by: Ravi.K
5 Replies
10. UNIX for Beginners Questions & Answers
Trying to split a 35gb file into 1000mb parts. My research shows I should you this. split -b 1000m file.txt and my return is "split: cannot open 'crunch1.txt' for reading: No such file or directory" so I tried split -b 1000m Documents/Wordlists/file.txt and I get nothing other than the curser just... (3 Replies)
Discussion started by: sub terra
3 Replies
LEARN ABOUT REDHAT
sgml2html
SGML2HTML(1) General Commands Manual SGML2HTML(1)
NAME
sgml2html - create HTML output from a LinuxDoc DTD SGML source file
SYNOPSIS
sgml2html [generic-option...] [--split 0|1|2 ] [--toc 0|1|2 ] [--dosnames] [--imagebuttons] file[.sgml]
DESCRIPTION
sgml2html is an old and obsoleted form of the html converter command of LinuxDoc-Tools. It is recommended to switch the new form linuxdoc
-B html now. This converts a LinuxDoc DTD SGML source file to HTML output. Output will appear in the top level file file.html and file-
n.html for each section, where file is the name of the SGML source file and n is the section name.
The attribute/value pair "output=html" is set for conditionals.
OPTIONS
sgml2html accepts all the generic options described in linuxdoc(1), and the following specific options:
--split, -s
What level to split source documents. 0 = don't split, 1 = split by major sections, 2 = split by subsections.
--toc, -T
What level to generate toc.
0 = don't generate toc at all,
1 = includes major sections(/chapters/parts),
2 = includes subsections.
--dosnames, -h
Use ".htm" rather than ".html" as the extension of
--imagebuttons, -I
Use the "next", "previous", and "contents" arrow image icons included in /usr/share/linuxdoc-tools as navigation buttons.
--footer, -F
Use the specified file as the footer in each resulted html file. Default footer is just plain
</BODY>
</HTML>
--header, -H
Use the specified file as the top part of the header in each resulted html file. Note this is not the full part of the header.
(i.e. the title and the links (next,previous,contents) in the default header are retained. Default is
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<HTML>
<HEAD>
file The SGML source file, named either file or file.sgml.
FILES
Many files and executables in /usr/share/linuxdoc-tools and /usr/bin are used.
BUGS
None known.
AUTHOR
Originally written by Greg Hankins <greg.hankins@cc.gatech.edu>, and Cees de Groot <cg@cdegroot.com> for SGML-Tools (v1). Currently main-
tained by Taketoshi Sano <sano@debian.org> for Linuxdoc-Tools.
SEE ALSO
linuxdoc(1), sgml2info(1), sgml2latex(1), sgml2lyx(1), sgml2rtf(1), sgml2txt(1), sgmlcheck(1).
16 May 2000 SGML2HTML(1)