Split file using awk - Shell Programming and Scripting

pvar · #1 07-20-2004

I am trying to read a file and split the file into multiple files. I need to create new files with different set of lines from the original file. ie, the first output file may contain 10 lines and the second 100 lines and so on. The criteria is to get the lines between two lines starting with some characters(variable)

eg. A sample input file

DOC100
aaaaaa
bbbbbb
cccccccc
END
DOC200
ddddddd
eeeeeee
END

I need to send the lines between DOC100 and END to out.1
and lines between DOC200 and the next END to out.2

I thought I could do this using awk and I tried the following command but it didn't work.

k=grep DOC $filename|cut -c4-6
awk -v cn=$k '/DOC$cn/,/END/ { print $0 }' $filename >$outfile

Can anyone help me on this? Thanks

zazzybob · #2 07-20-2004

A script such as this will name the files out.100, out.200 where the number corresponds to whatever follows "DOC"

Code:

#!/bin/sh

while read line
do
   if echo $line | grep DOC; then
       line_no=`echo $line | sed 's/DOC\([0-9]*\)/\1/g'`
   elif echo $line | grep END; then
       : #ignore
   else
       echo $line >> out.$line_no
   fi
done < file.txt

Where file.txt contains your data.

This is a quick script and there's room for improvement, but you get the idea.

Cheers
ZB

Ygor · #3 07-21-2004

Using awk..

Code:

awk '/^DOC/{f=$1} f{print > f} /^END/{close f}' infile

pvar · #4 07-21-2004

Thanks zazzybob and Ygor for your help. But I see some problem with each of your scripts.

zazzybob's script - First of all I have a fixed length data file that I need to split. The script creates seperate files with the intended data but it ignores the spaces in the fields, means it concatenates the fields leaving out the spaces in them.

Ygor's script - Your script creates only one file named DOC with the data between the last DOC/END. But it puts the data like what it is in the input file, that is a good sign.

zazzybob · #5 07-21-2004

To solve the problem with the script I posted, just put quotes around "$line", thusly

Code:

echo "$line" >> out.$line_no

That will preserve the whitespace

Cheers
ZB

Ygor · #6 07-21-2004

I didn't read the bit about filenames like "doc.1" before. Try this instead...

Code:

awk '/^DOC/{f="doc."++d} f{print > f} /^END/{close f; f=""}' infile

$ head -1000000 doc.*
==> doc.1 <==
DOC100
aaaaaa
bbbbbb
cccccccc
END

==> doc.2 <==
DOC200
ddddddd
eeeeeee
END

pvar · #7 07-21-2004

zazzybob's script works fine now.

Ygor's script still have an issue. It creates seperate output files now but it puts lines starting with DOC & END also in the output files. I just need the data between those lines.

Thanks agin for the help.

More UNIX and Linux Forum Topics You Might Find Helpful
Thread	Thread Starter	Forum	Replies	Last Post
Split large file and add header and footer to each file	ashish4422	Shell Programming and Scripting	1	04-15-2008 03:12 AM
Split a file with no pattern -- Split, Csplit, Awk	madhunk	UNIX for Dummies Questions & Answers	10	12-17-2007 09:57 AM
split the file name	rinku	Shell Programming and Scripting	1	05-27-2007 10:47 PM
Need to split the file	getdpg	Shell Programming and Scripting	11	09-04-2006 12:08 AM
How to split file by tags inside file?	spookyrtd99	Shell Programming and Scripting	1	07-30-2006 09:50 PM

Thread Tools	Search this Thread
Show Printable Version Email this Page	Search this Thread: Advanced Search
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode