Login or Register to Ask a Question and Join Our Community


UNIX for Dummies Questions & Answers

Split large file to smaller fastly

 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Split large file to smaller fastly
# 1  
Old 09-22-2014
Split large file to smaller fastly
hi ,

I have a requirement

input file:
Code:

1	1111111111111	108
1	1111111111111	109
1	1111111111111	109
1	1111111111111	110
1	1111111111111	111
1	1111111111111	111
1	1111111111111	111
1	1111111111111	112
1	1111111111111	112
1	1111111111111	112

The output should be,
Code:
108.txt:
======
1	1111111111111	108

109.txt:
======
1	1111111111111	109
1	1111111111111	109

110.txt:
======
1	1111111111111	110

111.txt:
======
1	1111111111111	111
1	1111111111111	111
1	1111111111111	111

112.txt:
======
1	1111111111111	112
1	1111111111111	112
1	1111111111111	112

the input file will be a huge file. based on the column i have split it by groups and filename should based the coulmn only.
if i use awk
it throwing an error:
Code:
awk: too many output files 10
 record number 11


Please give a command which will split the huge file to multiple files in less time.



Moderator's Comments:
Mod Comment Please use code tags next time for your code and data. Thanks
# 2  
Old 09-22-2014
Code:
$ awk '{f=sprintf("%s.txt",$NF); if(f in F){ print >>f }else { print >f ; F[f]} close(f) }' file

This User Gave Thanks to Akshay Hegde For This Post:
mechvijays
# 3  
Old 09-22-2014
try
Code:
awk '{print > $3".txt"}' filename

This User Gave Thanks to Makarand Dodmis For This Post:
mechvijays
# 4  
Old 09-22-2014
Quote:
Originally Posted by Makarand Dodmis
try
Code:
awk '{print > $3".txt"}' filename

if you are not closing you will get too many Too many open files error with large input file.
This User Gave Thanks to Akshay Hegde For This Post:
mechvijays
# 5  
Old 09-22-2014
then like this .. will it?
Code:
awk '{print > $3".txt"; close($3".txt")}' file

This User Gave Thanks to Makarand Dodmis For This Post:
mechvijays
# 6  
Old 09-22-2014
Quote:
Originally Posted by Makarand Dodmis
then like this .. will it?
Code:
awk '{print > $3".txt"; close($3".txt")}' file

This will overwrite file everytime, did you try ?
This User Gave Thanks to Akshay Hegde For This Post:
mechvijays
# 7  
Old 09-22-2014
please give commands other than awk,

awk commands showing the error

awk: too many output files 10
record number 11


os :

SunOS sasbsd27c1 5.10 Generic_150400-10 sun4u sparc SUNW,SPARC-Enterprise
Last edited by mechvijays; 09-22-2014 at 11:07 AM..
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Split large file into smaller files without disturbing the entry chunks

Dears, Need you help with the below file manipulation. I want to split the file into 8 smaller files but without cutting/disturbing the entries (meaning every small file should start with a entry and end with an empty line). It will be helpful if you can provide a one liner command for this... (12 Replies)
Discussion started by: Kamesh G
12 Replies

2. UNIX for Dummies Questions & Answers

Split files into smaller ones with 1000 hierarchies in a single file.

input file: AD,00,--,---,---,---,---,---,---,--,--,--- AM,000,---,---,---,---,---,--- AR, ,---,--,---,--- AA,---,---,---,--- AT,--- AU,---,---,--- AS,---,--- AP,---,---,--- AI,--- AD,00,---,---,---, ,---,---,---,---,---,--- AM,000,---,---,--- AR,... (6 Replies)
Discussion started by: kcdg859
6 Replies

3. Shell Programming and Scripting

Help needed - Split large file into smaller files based on pattern match

Help needed urgently please. I have a large file - a few hundred thousand lines. Sample CP START ACCOUNT 1234556 name 1 CP END ACCOUNT CP START ACCOUNT 2224444 name 1 CP END ACCOUNT CP START ACCOUNT 333344444 name 1 CP END ACCOUNT I need to split this file each time "CP START... (7 Replies)
Discussion started by: frustrated1
7 Replies

4. Shell Programming and Scripting

How to split a file into smaller files

Hi, I have a big text file with m columns and n rows. The format is like: STF123450001000200030004STF123450005000600070008STF123450009001000110012 STF234560345002208330154STF234590705620600070080STF234567804094562357688 STF356780001000200030004STF356780005000600070080STF356780800094562657687... (2 Replies)
Discussion started by: wintersnow2011
2 Replies

5. Shell Programming and Scripting

Split large file into smaller file

hi Guys i need some help here.. i have a file which has > 800,000 lines in it. I need to split this file into smaller files with 25000 lines each. please help thanks (1 Reply)
Discussion started by: sitaldip
1 Replies

6. UNIX for Dummies Questions & Answers

multiple smaller files from one large file

I have a file with a simple list of ids. 750,000 rows. I have to break it down into multiple 50,000 row files to submit in a batch process.. Is there an easy script I could write to accomplish this task? (2 Replies)
Discussion started by: rtroscianecki
2 Replies

7. Shell Programming and Scripting

Help with splitting a large text file into smaller ones

Hi Everyone, I am using a centos 5.2 server as an sflow log collector on my network. Currently I am using inmons free sflowtool to collect the packets sent by my switches. I have a bash script running on an infinate loop to stop and start the log collection at set intervals - currently one... (2 Replies)
Discussion started by: lord_butler
2 Replies

8. Shell Programming and Scripting

perl help to split big verilog file into smaller ones for each module

Hi I have a big verilog file with multiple modules. Each module begin with the code word 'module <module-name>(ports,...)' and end with the 'endmodule' keyword. Could you please suggest the best way to split each of these modules into multiple files? Thank you for the help. Example of... (7 Replies)
Discussion started by: return_user
7 Replies

9. UNIX for Dummies Questions & Answers

splitting the large file into smaller files

hi all im new to this forum..excuse me if anythng wrong. I have a file containing 600 MB data in that. when i do parse the data in perl program im getting out of memory error. so iam planning to split the file into smaller files and process one by one. can any one tell me what is the code... (1 Reply)
Discussion started by: vsnreddy
1 Replies

10. Shell Programming and Scripting

Cutting a large log file in to smaller ones

I have a very large (150 megs) IRC log file from 2000-2001 which I want to cut down to individual daily log files. I have a very basic knowledge of the cat, sed and grep commands. The log file is time stamped and each day in the large log file begins with a "Session Start" string like so: ... (11 Replies)
Discussion started by: MrTangent
11 Replies
Login or Register to Ask a Question