Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Splitting a file into several smaller files using perl
Top Forums Shell Programming and Scripting Splitting a file into several smaller files using perl Post 302616033 by Corona688 on Friday 30th of March 2012 11:23:16 AM
Old 03-30-2012
Code:
$ cat fsplit.sh

#!/bin/sh

if [ "$#" -lt 2 ] || [ ! -f "$1" ]
then
        echo "usage:  $0 inputfile numfiles" >&2
        exit 1
fi

awk -v NFILES="$2" -v FNAME="file_%d.dat" '
        # Do not print the first file -- just count lines
        NR==FNR { next }

        # First line of the second read through the file.
        FNR==1 {        HEADER=$0
                        MAXLINES=sprintf("%d", (NR-1)/NFILES);
                        LINES=MAXLINES
                        next    }

        # skip to the next file and print header if exceeded maxlines
        (LINES >= MAXLINES) {
                        LINES=0;        FILE++;
                        print HEADER > sprintf(FNAME,FILE);     }

        # Print all lines into the current file
        { print > sprintf(FNAME, FILE); LINES++ }

# Yes, we give awk the same file twice.  On the first read, it just counts
# lines.  On the second, it decides which lines go into what file.
' "$1" "$1"

$ cat data
BASENAME STREETTYPE PREFIX SUFFIX HOUSENUMBER
1 jhj jgu gv 36
2 dut jhg hg 54
3 gkl jkl hv 67
4 fjh gfh hg 45
5 hgl hgk hg 73
6 hkj hg yg 79
1 jhj jgu gv 36
2 dut jhg hg 54
3 gkl jkl hv 67
4 fjh gfh hg 45
5 hgl hgk hg 73
6 hkj hg yg 79
1 jhj jgu gv 36
2 dut jhg hg 54
3 gkl jkl hv 67
4 fjh gfh hg 45
5 hgl hgk hg 73
6 hkj hg yg 79
5 hgl hgk hg 73

$ tail *.dat
==> file_1.dat <==
BASENAME STREETTYPE PREFIX SUFFIX HOUSENUMBER
1 jhj jgu gv 36
2 dut jhg hg 54

==> file_10.dat <==
BASENAME STREETTYPE PREFIX SUFFIX HOUSENUMBER
5 hgl hgk hg 73

==> file_2.dat <==
BASENAME STREETTYPE PREFIX SUFFIX HOUSENUMBER
3 gkl jkl hv 67
4 fjh gfh hg 45

==> file_3.dat <==
BASENAME STREETTYPE PREFIX SUFFIX HOUSENUMBER
5 hgl hgk hg 73
6 hkj hg yg 79

==> file_4.dat <==
BASENAME STREETTYPE PREFIX SUFFIX HOUSENUMBER
1 jhj jgu gv 36
2 dut jhg hg 54

==> file_5.dat <==
BASENAME STREETTYPE PREFIX SUFFIX HOUSENUMBER
3 gkl jkl hv 67
4 fjh gfh hg 45

==> file_6.dat <==
BASENAME STREETTYPE PREFIX SUFFIX HOUSENUMBER
5 hgl hgk hg 73
6 hkj hg yg 79

==> file_7.dat <==
BASENAME STREETTYPE PREFIX SUFFIX HOUSENUMBER
1 jhj jgu gv 36
2 dut jhg hg 54

==> file_8.dat <==
BASENAME STREETTYPE PREFIX SUFFIX HOUSENUMBER
3 gkl jkl hv 67
4 fjh gfh hg 45

==> file_9.dat <==
BASENAME STREETTYPE PREFIX SUFFIX HOUSENUMBER
5 hgl hgk hg 73
6 hkj hg yg 79

$

Note the files match out of order because 10 doesn't sort alphabetically later than 9. Try %02d instead of %d to get numbers with leading zeroes that are always 2 digits.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Splitting a Larger File Into Mutiple Smaller ones.

Hello.. Iam in need to urgent help with the below. Have data-file with 40,567 and need to split them into multiple files with smaller line-count. Iam aware of "split" command with -l option which allows you to specify the no of lines in smaller files ,with the target file-name pattern... (1 Reply)
Discussion started by: madhubt_1982
1 Replies

2. UNIX for Dummies Questions & Answers

splitting the large file into smaller files

hi all im new to this forum..excuse me if anythng wrong. I have a file containing 600 MB data in that. when i do parse the data in perl program im getting out of memory error. so iam planning to split the file into smaller files and process one by one. can any one tell me what is the code... (1 Reply)
Discussion started by: vsnreddy
1 Replies

3. Shell Programming and Scripting

splitting text file into smaller ones

Hello We have a text file with 400,000 lines and need to split into multiple files each with 5000 lines ( will result in 80 files) Got an idea of using head and tail commands to do that with a loop but looked not efficient. Please advise the simple and yet effective way to do it. TIA... (3 Replies)
Discussion started by: prvnrk
3 Replies

4. Shell Programming and Scripting

perl help to split big verilog file into smaller ones for each module

Hi I have a big verilog file with multiple modules. Each module begin with the code word 'module <module-name>(ports,...)' and end with the 'endmodule' keyword. Could you please suggest the best way to split each of these modules into multiple files? Thank you for the help. Example of... (7 Replies)
Discussion started by: return_user
7 Replies

5. Shell Programming and Scripting

Help with splitting a large text file into smaller ones

Hi Everyone, I am using a centos 5.2 server as an sflow log collector on my network. Currently I am using inmons free sflowtool to collect the packets sent by my switches. I have a bash script running on an infinate loop to stop and start the log collection at set intervals - currently one... (2 Replies)
Discussion started by: lord_butler
2 Replies

6. Shell Programming and Scripting

How to split a file into smaller files

Hi, I have a big text file with m columns and n rows. The format is like: STF123450001000200030004STF123450005000600070008STF123450009001000110012 STF234560345002208330154STF234590705620600070080STF234567804094562357688 STF356780001000200030004STF356780005000600070080STF356780800094562657687... (2 Replies)
Discussion started by: wintersnow2011
2 Replies

7. Shell Programming and Scripting

Sed: Splitting A large File into smaller files based on recursive Regular Expression match

I will simplify the explaination a bit, I need to parse through a 87m file - I have a single text file in the form of : <NAME>house........ SOMETEXT SOMETEXT SOMETEXT . . . . </script> MORETEXT MORETEXT . . . (6 Replies)
Discussion started by: sumguy
6 Replies

8. Shell Programming and Scripting

Splitting a file and creating new files using Perl script

Hi All, I am new to Scripting language. I want to split a file and create several subfiles using Perl script. Example : File format : Sourcename ID Date Nbr SU IMYFDJ 9/17/2012 5552159976555 SU BWZMIG 9/14/2012 1952257857887 AR PEHQDF 11/26/2012 ... (13 Replies)
Discussion started by: Deepak9870
13 Replies

9. Shell Programming and Scripting

Splitting xml file into several xml files using perl

Hi Everyone, I'm new here and I was checking this old post: /shell-programming-and-scripting/180669-splitting-file-into-several-smaller-files-using-perl.html (cannot paste link because of lack of points) I need to do something like this but understand very little of perl. I also check... (4 Replies)
Discussion started by: mcosta
4 Replies

10. Shell Programming and Scripting

Splitting a text file into smaller files with awk, how to create a different name for each new file

Hello, I have some large text files that look like, putrescine Mrv1583 01041713302D 6 5 0 0 0 0 999 V2000 2.0928 -0.2063 0.0000 N 0 0 0 0 0 0 0 0 0 0 0 0 5.6650 0.2063 0.0000 N 0 0 0 0 0 0 0 0 0 0 0 0 3.5217 ... (3 Replies)
Discussion started by: LMHmedchem
3 Replies

LEARN ABOUT MOJAVE

geom

geom(1) 																   geom(1)

NAME

       geom - analyzes a molecular geometry input in Cartesian coordinates.

DESCRIPTION

       The  program  geom  reads a set of Cartesian coordinates and determines from them the bond distances (Bohr and angstrom), bond angles, tor-
       sional angles, out-of-plane angles (optional), moments of inertia, and rotational constants.

FILES REQUIRED

	   input.dat	      - Input file
       and one of the following:
	   geom.dat	      - geometry file
	   file11.dat	      - PSI-format geometry/gradient file

FILES GENERATED

	   geom.out	      - file containing the analysis

INPUT OPTIONS

       The geom program has the following options:

       -h     Print help information (these options).

       -aces [filename]
	      Read the input in an ACES output format, with each line consisting of atomic symbol, atomic number, and  Cartesian  coordinates  (in
	      Bohr).  The optional argument is the name of the geometry input file, which defaults to geom.dat.

       -qchem [filename]
	      Read the input in an QCHEM output format, with each line consisting of the atom number, the atomic symbol, and Cartesian coordinates
	      (in Angstroms).  The optional argument is the name of the geometry input file, which defaults to geom.dat.

       -xyz [filename]
	      Read the input in an XYZ output format.  The first line contains the number of atoms and the second line is a comment line.   Subse-
	      quent lines each contain the atomic symbol and Cartesian X Y and Z coordinates (in Angstroms).  The optional argument is the name of
	      the geometry input file, which defaults to geom.dat.

       -oop   Print out-of-plane angles, where the angle a-b-c-d is defined as the angle formed by the vector a-d and the plane defined  by  atoms
	      b, c, and d.

       DO_OOP = boolean
	      Same as command-line switch -oop described above.

       ISOTOPES = matrix of reals/strings
	      Used to specify masses to be used in the calculation of the center of mass and rotational constants.  One array is provided for each
	      isotopomer to be calculated.  The arrays can mix floating point numbers with strings which designate isotopes.  If this array is not
	      given  and  file11.dat  is  present, then the masses will be obtained by converting the atomic numbers in file11.  If masses are not
	      given by ISOTOPES or by file11, then no mass-related quantities will be calculated.

       -g [filename]
	      Read Cartesian coordinates from a file other than file11.dat.  If a filename is not given, the default is geom.dat.  The alternative
	      file is assumed to be in a format similar to that of PSI's geom.dat.  Since such files do not contain atomic numbers, the moments of
	      inertia and rotational constants are not calculated if this option is used.

       READ_GEOM = boolean
	      This option in input.dat tells geom to read the geometry from geom.dat in the PSI geom.dat format.

       -a     Print parameters for all pair distances.

       PRINT_ALL_DIST = boolean
	      Same as -a flag described above.

       -d distance
	      Only print parameters involving pairs of atoms which are less than distance bohr apart (default value is 4.0 bohr).

       PRINT_DISTANCE = value
	      Same as -d flag described above.

       -angstrom
	      The input coordinates are in angstroms, not bohr.

       ANGSTROM = boolean
	      If TRUE, the input coordinates are in angstroms instead of bohr.	The default is FALSE.

	      The following example is for calculating the geometrical information for water with several different  isotopes.	 The  geometry	is
	      read from geom.dat rather than from file11.dat.

	      geom: (
		 read_geom = true
		 isotopes = (
		    (O H H)
		    (O D D)
		    (O18 1.007825 1.007825)
		  ) )

								   5 June, 1998 							   geom(1)
All times are GMT -4. The time now is 10:08 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy