Login or Register to Ask a Question and Join Our Community


UNIX for Dummies Questions & Answers

How to split a huge file into small pieces (per 2000 columns)?

 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers How to split a huge file into small pieces (per 2000 columns)?
# 1  
Old 09-29-2011
How to split a huge file into small pieces (per 2000 columns)?
Dear all,

I have a big file:2879(rows)x400,170 (columns) like below. I 'd like to split the file into small pieces:2879(rows)x2000(columns) per file (the last small piece will be 2879x170.
So far, I only know how to create one samll piece at one time. But actually I need to repeat this work many times. I am wondering if there is a easy way I just run the code once and it will create all the samll pieces itself. Anyone can help? Thanks!!

file:
Code:
1 0 0.898 0.123 1 0 1 0 0.7987 0.124 1 0 ...
0.123 1 0 1 0 0.7987 0.124 1 0 1 0 1 0...
...

Thanks.
Moderator's Comments:
Mod Comment
Please use code tags when posting data and code samples!
Last edited by vgersh99; 09-29-2011 at 12:57 PM.. Reason: code tags, please!
# 2  
Old 09-29-2011
What's your system? The tools I'd usually use for this may not work with such long lines on most UNIX systems.
# 3  
Old 09-29-2011
just unix system. I use cut command to create the small pieces sucessfully one by one. Just want to know if there is a easy way I can create all the small pieces at once. Thanks.
# 4  
Old 09-29-2011
"what kind of system do you want to do x with?"
"unix"
"what model car do you need new brakepads for?"
"blue"

I'm writing a generic solution for you... It won't be the most efficient but should work most places.
# 5  
Old 09-29-2011
sorry, I am new to unix. I don't know how to answer your questions.

Thank you for working on it.
# 6  
Old 09-29-2011
how about uname -a?

---------- Post updated at 12:30 PM ---------- Previous update was at 11:26 AM ----------

Not knowing your system means I can't use the following 5-second solutions:
  1. Shell math: 400,000 could be greater than the 65536 limit on some shells.
  2. awk: many UNIX versions of awk have a linesize limit of 2000 chars.
  3. sed: see awk.
  4. cut: how can you script the number of columns without math?
which means I'm stuck with tr and can only use awk when I set RS to space. So far it's not working too well, I always end up off by one column. I'm about to give up and try perl.

Please post your system!

---------- Post updated at 01:19 PM ---------- Previous update was at 12:30 PM ----------

One way in perl:

Code:
#!/usr/bin/perl

my $COLS=2000, %FILES, @ARR;

while($LINE=<STDIN>)
{
        @ARR=split(/[ \n]/, "$LINE");

        for($F=0, $N=0; $N<($#ARR+1); $N+=$COLS, $F++)
        {
                $FN=sprintf("out%03d", $F);
                $FILES{$FN} || open($FILES{$FN}, ">$FN");
                print { $FILES{$FN} } join(" ", @ARR[$N..$N+$COLS-1])."\n";
        }
}

Code:
$ ./split.pl < filename

This User Gave Thanks to Corona688 For This Post:
forevertl
# 7  
Old 09-30-2011
Thanks so much for your help. I will try the perl script tomorrow.

Thanks again!
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Solaris

Split huge File System

Gents I have huge NAS File System as /sys with size 10 TB and I want to Split each 1TB in spirit File System to be mounted in the server. How to can I do that without changing anything in the source. Please your support. (1 Reply)
Discussion started by: AbuAliiiiiiiiii
1 Replies

2. UNIX for Beginners Questions & Answers

Split large file into 24 small files on one hour basis

I Have a large file with 24hrs log in the below format.i need to split the large file in to 24 small files on one hour based.i.e ex:from 09:55 to 10:55,10:55-11:55 can any one help me on this.! ... (20 Replies)
Discussion started by: Raghuram717
20 Replies

3. Shell Programming and Scripting

How to split this txt file into small files?

Dear shell experts, I would like to spilt a txt file into small ones. However, I did not know how to program use shell. If someone could help, it is greatly appreciated! Specifically, I supposed there is file named A.txt. The content of the file likes this: Subject run condtion ACC time... (3 Replies)
Discussion started by: psychmyluo
3 Replies

4. UNIX for Dummies Questions & Answers

Split a huge 7 GB File Based on Pattern into 4 files

Hi, I have a Huge 7 GB file which has around 1 million records, i want to split this file into 4 files to contain around 250k messages each. Please help me as Split command cannot work here as it might miss tags.. Format of the file is as below <!--###### ###### START-->... (6 Replies)
Discussion started by: KishM
6 Replies

5. Solaris

How to split 10GB file into small Sizes

Hi Team I have one 10 Gb log file I want to split it into say 10 of 1-1Gb file pls share ur experiences how to do this? Thanks in advance, (3 Replies)
Discussion started by: zimmyyash
3 Replies

6. Shell Programming and Scripting

Split a 30GB XML file into 16 pieces

I have a 30 GB XMl file which looks like this: <page> <title>APRIL</title> .........(text contents that I need to extract and store in 1.dat including the <title> tag) </page> <page> <title>August</title> ....(text contents that I need to store in 2.dat including the <title> tag) </page>... (13 Replies)
Discussion started by: shoaibjameel123
13 Replies

7. Shell Programming and Scripting

how to split a huge file by every 100 lines

into small files. i need to add a head.txt and tail.txt into small files at the begin and end, and give a name as q1.xml q2.xml q3.xml .... thank you very much. (2 Replies)
Discussion started by: dtdt
2 Replies

8. Shell Programming and Scripting

Help- counting delimiter in a huge file and split data into 2 files

I’m new to Linux script and not sure how to filter out bad records from huge flat files (over 1.3GB each). The delimiter is a semi colon “;” Here is the sample of 5 lines in the file: Name1;phone1;address1;city1;state1;zipcode1 Name2;phone2;address2;city2;state2;zipcode2;comment... (7 Replies)
Discussion started by: lv99
7 Replies

9. Shell Programming and Scripting

How do I split file into pieces with PERL?

How do I split file into pieces with PERL? IE file.txt head 1 2 3 4 end head 5 6 7 8 9 end n so on (7 Replies)
Discussion started by: 3junior
7 Replies

10. Shell Programming and Scripting

Split a file into 16 small files

Hi I want to split a file that has 'n' number of records into 16 small files. Can some one suggest me how to do this using Unix script? Thanks rrkk (10 Replies)
Discussion started by: rrkks
10 Replies
Login or Register to Ask a Question