Splitting data file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Splitting data file
# 1  
Old 01-06-2005
Question Splitting data file

Hello,
I'm trying to split a file by lines. I know that I can use the split command to do this, but the one problem I'm having is, each file created, the first line needs to be a header. I can use the split command the create another file with the header, then append the new split file to it. I do not like this approach. So I'm hoping someone may enlighten me.

thanks,
# 2  
Old 01-06-2005
There's probably an easier way to do it, but I'd do something like this with Perl...
Code:
#!/usr/bin/perl

# Just modify the following two variables
$NUM_LINES=10;
$HEADER_STRING="This is my header";

$count=0;
$file_num=0;

while (<>) {
  if ( $count == 0 ) {
     $filename = join("","file",$file_num);
     open( FILE, ">> $filename" );
     print( FILE "$HEADER_STRING\n" );
     print( FILE "$_" );
     $count++;
  } elsif ( $count == $NUM_LINES ) {
     close( FILE );
     $count = 0;
     $file_num++;
  } else {
     # just write the line!
     print( FILE "$_" );
     $count++;
  }
}

Then make it executable and call with
$ my_perl_script.pl my_large_file

This will split your large file into as many 10 data line files (i.e. 11 lines including the header) as required.

Just modify the $NUM_LINES and $HEADER_STRING variables to suit your needs.

Cheers
ZB
# 3  
Old 01-06-2005
Assuming that the header is the first line of the input file...
Code:
awk -v rows=3 -v filespec=outfile%03d '
    NR == 1 { 
        header = $0
        next
    }
    (row++ % rows) == 0 { 
        close(filename)
        filename = sprintf(filespec, ++filecount)
        print header > filename
    }
    {print $0 > filename}
' infile

Tested...
Code:
$ head infile outfile???
==> infile <==
h1
d1
d2
d3
d4
d5
d6
d7

==> outfile001 <==
h1
d1
d2
d3

==> outfile002 <==
h1
d4
d5
d6

==> outfile003 <==
h1
d7

# 4  
Old 01-07-2005
just a suggestion but this one is an easy one...
awk '{if (FNR==1}{print FILENAME ; print $0}else print $0}' file1 file2 file3 ........> inputfile
not very fancy but will place FILENAME of every file as the header of that file and append the next name of file as header and the file proceeding the header..
you can also use the -v option to set a variable for the header instead of using the FILENAME option.
moxxx68
# 5  
Old 01-08-2005
Quote:
Originally posted by moxxx68
just a suggestion but this one is an easy one...
awk '{if (FNR==1}{print FILENAME ; print $0}else print $0}' file1 file2 file3 ........> inputfile
This will concatenate the files, not split them as per the original posters request.

Also, your redirection to "inputfile" is misleading - the ">" redirection operator redirects STDOUT, therefore a more apt name for the resulting file would be "outputfile" Smilie

Cheers
ZB
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk issue splitting a fixed-width file containing line feed in data

Hi Forum. I have the following script that splits a large fixed-width file into smaller multiple fixed-width files based on input segment type. The main command in the script is: awk -v search_col_pos=$search_col_pos -v search_str_len=$search_str_len -v segment_type="$segment_type"... (8 Replies)
Discussion started by: pchang
8 Replies

2. Shell Programming and Scripting

Splitting the Data using awk

Hello All, I have a comma delimiter file with 10 columns. I took the desired data but from $4 I need to split into two columns as 3+7 bytes. awk -F"," -v OFS=',' '{print $2,$3,$4}' foo.txt 42366,11/10/2014,5012418769 42366,11/10/2014,2046955672 42366,11/10/2014,2076802951 ... (3 Replies)
Discussion started by: karumudi7
3 Replies

3. UNIX for Dummies Questions & Answers

Extracting data from one file, based on another file (splitting)

Dear All, I have two files but want to extract data from one based on another... can you please help me file 1 David Tom Ellen and file 2 David|0010|testnamez|resultsz David|0004|testnamex|resultsx Tom|0010|testnamez|resultsz Tom|0004|testnamex|resultsx Ellen|0010|testnamez|resultsz... (12 Replies)
Discussion started by: A-V
12 Replies

4. UNIX for Dummies Questions & Answers

Splitting data into new records

Hi, My file is seperated with ";" delimiter, after 13 delimiter i want to put the data in new line... eg: My current file:- a;b;c;d;e;f;g;h;e;f;h;s;t;a;i;o;q;t;q;r;yu;f;sz;f;t;r........... i want o/p as:- a;b;c;d;e;f;g;h;e;f;h;s;t a;i;o;q;t;q;r;yu;f;sz;f;t;r How to achieve ths,... (2 Replies)
Discussion started by: gnnsprapa
2 Replies

5. Shell Programming and Scripting

Help me pls : splitting single file in unix into different files based on data

I have a file in unix with sample data as follows : -------------------------------------------------------------- -------------------------------------------------------------- {30001002|XXparameter|Layout|$ I want this file to be splitted into different files and corresponding to the sample... (54 Replies)
Discussion started by: Ravindra Swan
54 Replies

6. Shell Programming and Scripting

perl : splitting the data into 2 different variables

I have a perl variable which contains the below value. $var1 = "2% / 51%" Now I would like to split the data into 2 different variables. For example $part1 = 2 $part2 = 51 Could anyone please help me in this regard ? Regards, GS (4 Replies)
Discussion started by: giridhar276
4 Replies

7. Shell Programming and Scripting

Splitting the data in a column into several columns

Hi, I have the following input file 32895901-d17f-414c-ac93-3e7e0f5ec240 AND @GDF_INPUT 73b129e1-1fa9-4c0d-b95b-4682e5389612 AUS @GDF_INPUT 40f82e88-d1ff-4ce2-9b8e-d827ddb39447 BEL @GDF_INPUT 36e9c3f1-042a-43a4-a80e-4a3bc2513d01 BGR @GDF_INPUT I want to split column 3 into two columns:... (1 Reply)
Discussion started by: ramky79
1 Replies

8. UNIX for Dummies Questions & Answers

Splitting Data in File

I have a file with the below Data 1,nj@ny@pa@caa 2,ct 3,ca@vaa@txI want the output to be 1,nj 1,ny 1,pa 1,caa 2,ct 3,ca 3,vaa 3,tx I need to split the second column based on @ as delimiter The number of delimiters is unknown (4 Replies)
Discussion started by: traininfa
4 Replies

9. Shell Programming and Scripting

Data Splitting into two files from one file

I have a file as: I/P File: Ground Car 2009 Lib 2008 Lib 2003 Ground Car 2009 Ground Car 2003 Car 2005 Car 2003 Car 2005 Sita 2900 2006 Car 2007 I have to split the file into two: - one for names and second for years. O/p1 (Names): Ground Car (3 Replies)
Discussion started by: karumudi7
3 Replies

10. Shell Programming and Scripting

Splitting the data and storing it into 2 variables

Hi, I have a file in the given format: |aaa |hhh |fat hat chat |make sure I need to get store in a variable,say 'error' the value of the string 'fat hat chat' In short, echo $error should give the result fat hat chat Is this possible using awk? Also, can this be split.For... (10 Replies)
Discussion started by: jisha
10 Replies
Login or Register to Ask a Question