How to split a huge file into small pieces (per 2000 columns)? Post: 302560347

Sponsored Content

Top Forums UNIX for Dummies Questions & Answers How to split a huge file into small pieces (per 2000 columns)? Post 302560347 by Corona688 on Thursday 29th of September 2011 03:19:16 PM

09-29-2011

Registered User

how about uname -a?

---------- Post updated at 12:30 PM ---------- Previous update was at 11:26 AM ----------

Not knowing your system means I can't use the following 5-second solutions:

Shell math: 400,000 could be greater than the 65536 limit on some shells.
awk: many UNIX versions of awk have a linesize limit of 2000 chars.
sed: see awk.
cut: how can you script the number of columns without math?

which means I'm stuck with tr and can only use awk when I set RS to space. So far it's not working too well, I always end up off by one column. I'm about to give up and try perl.

Please post your system!

---------- Post updated at 01:19 PM ---------- Previous update was at 12:30 PM ----------

One way in perl:

Code:

#!/usr/bin/perl

my $COLS=2000, %FILES, @ARR;

while($LINE=<STDIN>)
{
        @ARR=split(/[ \n]/, "$LINE");

        for($F=0, $N=0; $N<($#ARR+1); $N+=$COLS, $F++)
        {
                $FN=sprintf("out%03d", $F);
                $FILES{$FN} || open($FILES{$FN}, ">$FN");
                print { $FILES{$FN} } join(" ", @ARR[$N..$N+$COLS-1])."\n";
        }
}

Code:

$ ./split.pl < filename

This User Gave Thanks to Corona688 For This Post:

Corona688

View Public Profile for Corona688

Visit Corona688's homepage!

Find all posts by Corona688

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Split a file into 16 small files

Hi I want to split a file that has 'n' number of records into 16 small files. Can some one suggest me how to do this using Unix script? Thanks rrkk

2. Shell Programming and Scripting

How do I split file into pieces with PERL?

How do I split file into pieces with PERL? IE file.txt head 1 2 3 4 end head 5 6 7 8 9 end n so on

3. Shell Programming and Scripting

Help- counting delimiter in a huge file and split data into 2 files

I’m new to Linux script and not sure how to filter out bad records from huge flat files (over 1.3GB each). The delimiter is a semi colon “;” Here is the sample of 5 lines in the file: Name1;phone1;address1;city1;state1;zipcode1 Name2;phone2;address2;city2;state2;zipcode2;comment...

4. Shell Programming and Scripting

how to split a huge file by every 100 lines

into small files. i need to add a head.txt and tail.txt into small files at the begin and end, and give a name as q1.xml q2.xml q3.xml .... thank you very much.

5. Shell Programming and Scripting

Split a 30GB XML file into 16 pieces

I have a 30 GB XMl file which looks like this: <page> <title>APRIL</title> .........(text contents that I need to extract and store in 1.dat including the <title> tag) </page> <page> <title>August</title> ....(text contents that I need to store in 2.dat including the <title> tag) </page>...

6. Solaris

How to split 10GB file into small Sizes

Hi Team I have one 10 Gb log file I want to split it into say 10 of 1-1Gb file pls share ur experiences how to do this? Thanks in advance,

7. UNIX for Dummies Questions & Answers

Split a huge 7 GB File Based on Pattern into 4 files

Hi, I have a Huge 7 GB file which has around 1 million records, i want to split this file into 4 files to contain around 250k messages each. Please help me as Split command cannot work here as it might miss tags.. Format of the file is as below ...

8. Shell Programming and Scripting

How to split this txt file into small files?

Dear shell experts, I would like to spilt a txt file into small ones. However, I did not know how to program use shell. If someone could help, it is greatly appreciated! Specifically, I supposed there is file named A.txt. The content of the file likes this: Subject run condtion ACC time...

9. UNIX for Beginners Questions & Answers

Split large file into 24 small files on one hour basis

I Have a large file with 24hrs log in the below format.i need to split the large file in to 24 small files on one hour based.i.e ex:from 09:55 to 10:55,10:55-11:55 can any one help me on this.! ...

10. Solaris

Split huge File System

Gents I have huge NAS File System as /sys with size 10 TB and I want to Split each 1TB in spirit File System to be mounted in the server. How to can I do that without changing anything in the source. Please your support.

LEARN ABOUT DEBIAN

deb-split

deb-split(5)							      Debian							      deb-split(5)

NAME

       deb-split - Debian multi-part binary package format

SYNOPSIS

       filename.deb

DESCRIPTION

       The multi-part .deb format is used to split big packages into smaller pieces to ease transport in small media.

FORMAT

       The file is an ar archive with a magic value of !<arch>.  The file names might contain a trailing slash (since dpkg 1.15.6).

       The first member is named debian-split and contains a series of lines, separated by newlines. Currently eight lines are present:

       o   The format version number, 2.1 at the time this manual page was written.

       o   The package name.

       o   The package version.

       o   The md5sum of the package.

       o   The total size of the package.

       o   The maximum part size.

       o   The current part number, followed by a slash and the total amount of parts (as in '1/10').

       o   The package architecture (since dpkg 1.16.1).

       Programs  which	read multi-part archives should be prepared for the minor format version number to be increased and additional lines to be
       present, and should ignore these if this is the case.

       If the major format version number has changed, an incompatible change has been made and the program should stop. If it has not,  then  the
       program	should	be  able  to  safely  continue, unless it encounters an unexpected member in the archive (except at the end), as described
       below.

       The second, last required member is named data.N, where N denotes the part number. It contains the raw part data.

       These members must occur in this exact order. Current implementations should ignore any additional members after data.N.   Further  members
       may be defined in the future, and (if possible) will be placed after these two.

SEE ALSO

       deb(5), dpkg-split(1).

Debian Project							    2012-04-09							      deb-split(5)

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Split a file into 16 small files

Discussion started by: rrkks

2. Shell Programming and Scripting

How do I split file into pieces with PERL?

Discussion started by: 3junior

3. Shell Programming and Scripting

Help- counting delimiter in a huge file and split data into 2 files

Discussion started by: lv99

4. Shell Programming and Scripting

how to split a huge file by every 100 lines

Discussion started by: dtdt