Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Splitting a file into several smaller files using perl

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Splitting a file into several smaller files using perl
# 1  
Old 03-29-2012
Splitting a file into several smaller files using perl
Hi,
I'm trying to split a large file into several smaller files
the script will have two input arguments argument1=filename and argument2=no of files to be split.

In my large input file I have a header followed by 100009 records
The first line is a header; I want this header in all my splitted files

Here is what I have done so far
Code:
#!/usr/bin/perl
use File::Basename;

$inputfile=@ARGV[0];
$nof=@ARGV[1];                              # nof - no of files to split
($filename,$dir,$ext) = fileparse($inputfile,'\..*');
$header=`cat $inputfile | head -1`;
$NOLIF=`cat $inputfile | wc -l`;         # NOLIF - no of lines in file
$NOARIF=$NOLIF-1;                         # NOARIF - no of actual records in file
$NORPF=$NOARIF/$not;                    # NORPF - no of records per file
$NNORPF=`printf "%1.f\n" $NORPF`;   # NNORPF - new no of records per file

$count=0;
$filenum=0;

while (<$inputfile>) {
if ( $count == 0 ) {
     $nfilename = $filename._.$filenum.$ext;
     open( FILE, ">> $nfilename" );
     print( FILE "$header\n" );
     print( FILE "$_" );
     $count++;
  #} elsif ( $count == $NUM_LINES ) {
  } elsif ( $count == $NNORPF ) {
     close( FILE );
     $count = 0;
     $file_num++;
  } else {
     # just write the line!
     print( FILE "$_" );
     $count++;
  }
}

Here is my challenge:
Say I'm splitting my large input file into 10 files
so the first 9 files should have 10001 records and last should have 10000 records.

how do i get this working.
# 2  
Old 03-29-2012
Instead of showing a program that doesn't do what you want which doesn't seem to have anything to do with your question, and hoping we can guess what you think it's supposed to do, why not just show the input data you have, and the output data you want? Less guessing that way.
# 3  
Old 03-29-2012
Here is my input file "file.dat"; the file has 19 records and i want to split them into 10 files

BASENAME STREETTYPE PREFIX SUFFIX HOUSENUMBER
1 jhj jgu gv 36
2 dut jhg hg 54
3 gkl jkl hv 67
4 fjh gfh hg 45
5 hgl hgk hg 73
6 hkj hg yg 79
1 jhj jgu gv 36
2 dut jhg hg 54
3 gkl jkl hv 67
4 fjh gfh hg 45
5 hgl hgk hg 73
6 hkj hg yg 79
1 jhj jgu gv 36
2 dut jhg hg 54
3 gkl jkl hv 67
4 fjh gfh hg 45
5 hgl hgk hg 73
6 hkj hg yg 79
5 hgl hgk hg 73

Using this command
Code:
splitfile.pl file.dat 10

I want my output to look like this

Code:
file_1.dat:
BASENAME        STREETTYPE      PREFIX  SUFFIX  HOUSENUMBER
1       jhj     jgu     gv      36
2       dut     jhg     hg      54

file_2.dat:
BASENAME        STREETTYPE      PREFIX  SUFFIX  HOUSENUMBER
3       gkl     jkl     hv      67
4       fjh     gfh     hg      45

file_3.dat:
BASENAME        STREETTYPE      PREFIX  SUFFIX  HOUSENUMBER
5       hgl     hgk     hg      73
6       hkj     hg      yg      79

file_4.dat:
BASENAME        STREETTYPE      PREFIX  SUFFIX  HOUSENUMBER
1       jhj     jgu     gv      36
2       dut     jhg     hg      54

file_5.dat:
BASENAME        STREETTYPE      PREFIX  SUFFIX  HOUSENUMBER
3       gkl     jkl     hv      67
4       fjh     gfh     hg      45

file_6.dat:
BASENAME        STREETTYPE      PREFIX  SUFFIX  HOUSENUMBER
5       hgl     hgk     hg      73
6       hkj     hg      yg      79

file_7.dat:
BASENAME        STREETTYPE      PREFIX  SUFFIX  HOUSENUMBER
1       jhj     jgu     gv      36
2       dut     jhg     hg      54

file_8.dat:
BASENAME        STREETTYPE      PREFIX  SUFFIX  HOUSENUMBER
3       gkl     jkl     hv      67
4       fjh     gfh     hg      45

file_9.dat:
BASENAME        STREETTYPE      PREFIX  SUFFIX  HOUSENUMBER
5       hgl     hgk     hg      73
6       hkj     hg      yg      79

file_10.dat:
BASENAME        STREETTYPE      PREFIX  SUFFIX  HOUSENUMBER
5       hgl     hgk     hg      73

Last edited by Corona688; 03-30-2012 at 12:23 PM.. Reason: Code tags for code and data, please.
# 4  
Old 03-30-2012
Code:
$ cat fsplit.sh

#!/bin/sh

if [ "$#" -lt 2 ] || [ ! -f "$1" ]
then
        echo "usage:  $0 inputfile numfiles" >&2
        exit 1
fi

awk -v NFILES="$2" -v FNAME="file_%d.dat" '
        # Do not print the first file -- just count lines
        NR==FNR { next }

        # First line of the second read through the file.
        FNR==1 {        HEADER=$0
                        MAXLINES=sprintf("%d", (NR-1)/NFILES);
                        LINES=MAXLINES
                        next    }

        # skip to the next file and print header if exceeded maxlines
        (LINES >= MAXLINES) {
                        LINES=0;        FILE++;
                        print HEADER > sprintf(FNAME,FILE);     }

        # Print all lines into the current file
        { print > sprintf(FNAME, FILE); LINES++ }

# Yes, we give awk the same file twice.  On the first read, it just counts
# lines.  On the second, it decides which lines go into what file.
' "$1" "$1"

$ cat data
BASENAME STREETTYPE PREFIX SUFFIX HOUSENUMBER
1 jhj jgu gv 36
2 dut jhg hg 54
3 gkl jkl hv 67
4 fjh gfh hg 45
5 hgl hgk hg 73
6 hkj hg yg 79
1 jhj jgu gv 36
2 dut jhg hg 54
3 gkl jkl hv 67
4 fjh gfh hg 45
5 hgl hgk hg 73
6 hkj hg yg 79
1 jhj jgu gv 36
2 dut jhg hg 54
3 gkl jkl hv 67
4 fjh gfh hg 45
5 hgl hgk hg 73
6 hkj hg yg 79
5 hgl hgk hg 73

$ tail *.dat
==> file_1.dat <==
BASENAME STREETTYPE PREFIX SUFFIX HOUSENUMBER
1 jhj jgu gv 36
2 dut jhg hg 54

==> file_10.dat <==
BASENAME STREETTYPE PREFIX SUFFIX HOUSENUMBER
5 hgl hgk hg 73

==> file_2.dat <==
BASENAME STREETTYPE PREFIX SUFFIX HOUSENUMBER
3 gkl jkl hv 67
4 fjh gfh hg 45

==> file_3.dat <==
BASENAME STREETTYPE PREFIX SUFFIX HOUSENUMBER
5 hgl hgk hg 73
6 hkj hg yg 79

==> file_4.dat <==
BASENAME STREETTYPE PREFIX SUFFIX HOUSENUMBER
1 jhj jgu gv 36
2 dut jhg hg 54

==> file_5.dat <==
BASENAME STREETTYPE PREFIX SUFFIX HOUSENUMBER
3 gkl jkl hv 67
4 fjh gfh hg 45

==> file_6.dat <==
BASENAME STREETTYPE PREFIX SUFFIX HOUSENUMBER
5 hgl hgk hg 73
6 hkj hg yg 79

==> file_7.dat <==
BASENAME STREETTYPE PREFIX SUFFIX HOUSENUMBER
1 jhj jgu gv 36
2 dut jhg hg 54

==> file_8.dat <==
BASENAME STREETTYPE PREFIX SUFFIX HOUSENUMBER
3 gkl jkl hv 67
4 fjh gfh hg 45

==> file_9.dat <==
BASENAME STREETTYPE PREFIX SUFFIX HOUSENUMBER
5 hgl hgk hg 73
6 hkj hg yg 79

$

Note the files match out of order because 10 doesn't sort alphabetically later than 9. Try %02d instead of %d to get numbers with leading zeroes that are always 2 digits.
# 5  
Old 03-30-2012
Hi,
This script works, but when i'm trying to split for a large file(~150000 records)
It is creating more than 10000 files...

I'm trying to split a large file into 10 files... here is what I'm giving
I have tweaked the code a little bit to customize my needs

The parameters I'm passing in
inputfile = NA.dat
numfiles =10
opdir = destination directory where i need my files
fileext = the file extension (dat)
region = the first part of my file name (NA)

Code:
#!/bin/sh

if [ "$#" -lt 5 ] || [ ! -f "$1" ]
then
        echo "usage:  $0 inputfile numfiles opdir fileext region" >&2
        exit 1
fi

awk -v NFILES="$2" -v FNAME="$3/$5_%d.$4" '
        # Do not print the first file -- just count lines
        NR==FNR { next }

        # First line of the second read through the file.
        FNR==1 {        HEADER=$0
                        MAXLINES=sprintf("%d", (NR-1)/NFILES);
                        LINES=MAXLINES
                        next    }

        # skip to the next file and print header if exceeded maxlines
        (LINES >= MAXLINES) {
                        LINES=0;        FILE++;
                        print HEADER > sprintf(FNAME,FILE);     }

        # Print all lines into the current file
        { print > sprintf(FNAME, FILE); LINES++ }

# Yes, we give awk the same file twice.  On the first read, it just counts
# lines.  On the second, it decides which lines go into what file.
' "$1" "$1"

I want 10 files to be created in my opdir (NA_1.dat .. NA_10.dat) with 15000 records in each of them
# 6  
Old 03-30-2012
There was a minor error in my code which caused the MAXLINES variable to be a string and not a number.

Unfortunately, your input data was so perfectly matched to my bug that it worked anyway, so I didn't notice! Smilie

All your code needs is two keystrokes +0. I'd also add a bit of error checking.

Code:
#!/bin/sh

if [ "$#" -lt 5 ] || [ ! -f "$1" ] || [ ! -d "$3" ]
then
        echo "usage:  $0 inputfile numfiles opdir fileext region" >&2
        exit 1
fi

awk -v NFILES="$2" -v FNAME="$3/$5_%d.$4" '
        # Do not print the first file -- just count lines
        NR==FNR { next }

        # First line of the second read through the file.
        FNR==1 {        HEADER=$0
                        MAXLINES=sprintf("%d", (NR-1)/NFILES)+0;
                        LINES=MAXLINES
                        next    }

        # skip to the next file and print header if exceeded maxlines
        (LINES >= MAXLINES) {
                        LINES=0;        FILE++;
                        print HEADER > sprintf(FNAME,FILE);     }

        # Print all lines into the current file
        { print > sprintf(FNAME, FILE); LINES++ }

# Yes, we give awk the same file twice.  On the first read, it just counts
# lines.  On the second, it decides which lines go into what file.
' "$1" "$1"

This time I tested it all-out, making files with 150,000 lines, 100,009 lines, etc, etc. It splits as you wanted.
This User Gave Thanks to Corona688 For This Post:
shellter
# 7  
Old 03-30-2012
Thanks corona688... this works as expected... but i made one little change instead of +0 i did +1, that way I'm splitting into 10 files instead of 11.

Thank you so much.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Splitting a text file into smaller files with awk, how to create a different name for each new file

Hello, I have some large text files that look like, putrescine Mrv1583 01041713302D 6 5 0 0 0 0 999 V2000 2.0928 -0.2063 0.0000 N 0 0 0 0 0 0 0 0 0 0 0 0 5.6650 0.2063 0.0000 N 0 0 0 0 0 0 0 0 0 0 0 0 3.5217 ... (3 Replies)
Discussion started by: LMHmedchem
3 Replies

2. Shell Programming and Scripting

Splitting xml file into several xml files using perl

Hi Everyone, I'm new here and I was checking this old post: /shell-programming-and-scripting/180669-splitting-file-into-several-smaller-files-using-perl.html (cannot paste link because of lack of points) I need to do something like this but understand very little of perl. I also check... (4 Replies)
Discussion started by: mcosta
4 Replies

3. Shell Programming and Scripting

Splitting a file and creating new files using Perl script

Hi All, I am new to Scripting language. I want to split a file and create several subfiles using Perl script. Example : File format : Sourcename ID Date Nbr SU IMYFDJ 9/17/2012 5552159976555 SU BWZMIG 9/14/2012 1952257857887 AR PEHQDF 11/26/2012 ... (13 Replies)
Discussion started by: Deepak9870
13 Replies

4. Shell Programming and Scripting

Sed: Splitting A large File into smaller files based on recursive Regular Expression match

I will simplify the explaination a bit, I need to parse through a 87m file - I have a single text file in the form of : <NAME>house........ SOMETEXT SOMETEXT SOMETEXT . . . . </script> MORETEXT MORETEXT . . . (6 Replies)
Discussion started by: sumguy
6 Replies

5. Shell Programming and Scripting

How to split a file into smaller files

Hi, I have a big text file with m columns and n rows. The format is like: STF123450001000200030004STF123450005000600070008STF123450009001000110012 STF234560345002208330154STF234590705620600070080STF234567804094562357688 STF356780001000200030004STF356780005000600070080STF356780800094562657687... (2 Replies)
Discussion started by: wintersnow2011
2 Replies

6. Shell Programming and Scripting

Help with splitting a large text file into smaller ones

Hi Everyone, I am using a centos 5.2 server as an sflow log collector on my network. Currently I am using inmons free sflowtool to collect the packets sent by my switches. I have a bash script running on an infinate loop to stop and start the log collection at set intervals - currently one... (2 Replies)
Discussion started by: lord_butler
2 Replies

7. Shell Programming and Scripting

perl help to split big verilog file into smaller ones for each module

Hi I have a big verilog file with multiple modules. Each module begin with the code word 'module <module-name>(ports,...)' and end with the 'endmodule' keyword. Could you please suggest the best way to split each of these modules into multiple files? Thank you for the help. Example of... (7 Replies)
Discussion started by: return_user
7 Replies

8. Shell Programming and Scripting

splitting text file into smaller ones

Hello We have a text file with 400,000 lines and need to split into multiple files each with 5000 lines ( will result in 80 files) Got an idea of using head and tail commands to do that with a loop but looked not efficient. Please advise the simple and yet effective way to do it. TIA... (3 Replies)
Discussion started by: prvnrk
3 Replies

9. UNIX for Dummies Questions & Answers

splitting the large file into smaller files

hi all im new to this forum..excuse me if anythng wrong. I have a file containing 600 MB data in that. when i do parse the data in perl program im getting out of memory error. so iam planning to split the file into smaller files and process one by one. can any one tell me what is the code... (1 Reply)
Discussion started by: vsnreddy
1 Replies

10. Shell Programming and Scripting

Splitting a Larger File Into Mutiple Smaller ones.

Hello.. Iam in need to urgent help with the below. Have data-file with 40,567 and need to split them into multiple files with smaller line-count. Iam aware of "split" command with -l option which allows you to specify the no of lines in smaller files ,with the target file-name pattern... (1 Reply)
Discussion started by: madhubt_1982
1 Replies
Login or Register to Ask a Question