Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Splitting a complex file using awk

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Splitting a complex file using awk
# 1  
Old 10-07-2010
Splitting a complex file using awk
I have a file that contains the following format

Code:
delete from table1;
delete from table2;
insert into table1 (col1, col2) values (value1, value2)@
insert into table1 (col1, col2) values(value3, value4)@
insert into table2(col1, col2,col3) values(value1, value2, value3)@
etc
etc

This is in a single file, too big to edit.

The INSERT statements can be over multiple lines, and the terminator is always '@' sign.

There is a fixed number of tables and I know the name of these.

I have tried with awk using the following

Code:
awk '/^table1/{p=1}/^table2/{p=0;print}p' inputfile > outfile
awk '/^table2/{p=1}/^table3/{p=0;print}p' inputfile > outfile


This sort of works for anything apart from the first statement.

for the second statement it puts into the outfile

Code:
delete from table2
insert into table2(col1, col2,col3) values(value1, value2, value3)@
insert into table3........

But at least I can edit the file and remove table 3.

But there is probably a way of using awk to do this.

Thanks

Chris
Last edited by Scott; 10-07-2010 at 07:53 AM.. Reason: Code tags, please...
# 2  
Old 10-07-2010
Post desired output, that you want to get from this sample file.
# 3  
Old 10-07-2010
file1
Code:
insert into table1 (col1, col2) values (value1, value2)@
insert into table1 (col1, col2) values(value3, value4)@

file2
Code:
insert into table2 (col1, col2) values (value1, value2)@
insert into table2 (col1, col2) values(value3, value4)@

file3
Code:
etc etc

So we have 1 file containing all the inserts for 1 table, 1 file for all the inserts for table 2 etc

Also I have just noticed that some of the INSERTS are over multiple lines.

eg.

Code:
INSERT INTO
table3 (col1, col2) values(val1 etc)

Last edited by Scott; 10-07-2010 at 07:53 AM.. Reason: Code tags, please...
# 4  
Old 10-07-2010
Try:
Code:
awk '/^insert/,/@/{if (/^insert/){n=gensub("^insert into ([^ (]+).*","\\1",1)};print > n}' big_file

It will create files with table names as filenames.
# 5  
Old 10-07-2010
Code:
awk '/^insert/,/@$/{printf (NF>2)?$0 RS:$0 FS}' infile |awk '{print >$3}'

# 6  
Old 10-07-2010
Thanks for the replies.
bartus11 .. when running yours I am getting an error Function gensub is not defined
I changed the insert to INSERT as in uppercase.

rdcwayx I am getting There are not enough parameters in printf statement when running yours.

here is a full insert statement that I have for one table (the others are similar but could be more or less columns)
Code:
INSERT INTO
polc.LITExpandata2   (OBJID,PARID,COMPANY,RECIND,NAME,SHORTNAME,RPTNAME,KEYVALUE,FLAG1,FLAG2,FLAG3,FLAG4,FLAG5,FLAG6,FLAG7,FLAG8,FLAG9,FLAG10,LANGUAGE) VALUES (-1,-1,4402,100000004,'0-2 YEARS OF EXPERIENCE                                     ','                    ','          ','B        ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ','en')@
INSERT INTO
polc.LITExpandata2   (OBJID,PARID,COMPANY,RECIND,NAME,SHORTNAME,RPTNAME,KEYVALUE,FLAG1,FLAG2,FLAG3,FLAG4,FLAG5,FLAG6,FLAG7,FLAG8,FLAG9,FLAG10,LANGUAGE) VALUES (-1,-1,4402,100000004,'NOT APPLICABLE                                              ','                    ','          ','C        ',' ',' ',' ',' ',' ',' ',' ',' ',' ',' ','en')@

Last edited by Franklin52; 10-09-2010 at 10:44 AM.. Reason: Please use code tags
# 7  
Old 10-07-2010
Can you transfer that file to some Linux box and run my command there? Smilie
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

awk solution for Splitting a file.

Hi I have a csv file with as below sdg-catalog-00000001 sdg-sku-00000317 sdg-sku-00000318 sdg-sku-00000319 sdg-sku-00000320 sdg-catalog-00000002 sdg-sku-00000321 sdg-sku-00000322 sdg-sku-00000323 sdg-sku-00000324 sdg-sku-00000325 sdg-catalog-00000003 sdg-sku-00000326... (3 Replies)
Discussion started by: Raghuram717
3 Replies

2. Shell Programming and Scripting

Splitting a text file into smaller files with awk, how to create a different name for each new file

Hello, I have some large text files that look like, putrescine Mrv1583 01041713302D 6 5 0 0 0 0 999 V2000 2.0928 -0.2063 0.0000 N 0 0 0 0 0 0 0 0 0 0 0 0 5.6650 0.2063 0.0000 N 0 0 0 0 0 0 0 0 0 0 0 0 3.5217 ... (3 Replies)
Discussion started by: LMHmedchem
3 Replies

3. Programming

How to replace the complex strings from a file using sed or awk?

Dear All, I am having a requirement to find the difference between 2 files and generate a discrepancy report out of it as an html page. I prefer using diff -y file1 file2 since it gives user friendly layout to know any discrepancy in the record and unique records among the 2 file. Here's how it... (12 Replies)
Discussion started by: Badhrish
12 Replies

4. Shell Programming and Scripting

Splitting the file using awk

Hi, I have a requirement in which I am going to receive one file and should be splitted to 9 different files based on one distinguisher called TYPE. I heard that this can be done using awk or sed. Can any one advise regardint the logic and simpler way other than using awk or sed is also... (15 Replies)
Discussion started by: sagar.cumar
15 Replies

5. Shell Programming and Scripting

Splitting file using awk

I have file with below content FG1620000|20000 FG1623000|23000 FG1625000|25000 FG1643894|43894 FG1643895|43895 FG1643896|43896 FG1643897|43897 FG1643898|43898 My aim is to split the above file into two files based on the value in the second field. If the value in second field is... (2 Replies)
Discussion started by: anijan
2 Replies

6. Shell Programming and Scripting

echoing complex awk command into file fails

Using hp-ux's shell, I'm trying to echo a complex awk command into a script file for later use. But it fails on a newline character and splits the rest of the command onto the next line. echo ' printf("%s: TOTAL = %18.0lf\n", FILENAME, TOTAL) >> "TOTAL.TXT";' >>awk.script Looks... (3 Replies)
Discussion started by: Scottie1954
3 Replies

7. Shell Programming and Scripting

awk for splitting file in constant chunks

Hi gurus, I wanted to split main file in 20 files with 2500 lines in each file. My main file conatins total 2500*20 lines. Following awk I made, but it is breaking with error. awk '{ for (i = 1; i <= 20; i++) { starts=2500*$i-1; ends=2500*$i; NR>=starts && NR<=ends {f=My$i".txt"; print >> f;... (10 Replies)
Discussion started by: mukesh.lalwani
10 Replies

8. Shell Programming and Scripting

Sorting complex file with awk

i have a file ddd.txt its delimiter is : but has , and "" within each column as below and also each line ends with ; I_EP,"29":I_US,"120":I_P_ID,"2020":I_NEW,"600":I_OLD,"400":I_POW,"4.5":I_NAME,"TOM";... (9 Replies)
Discussion started by: blackzinga80
9 Replies

9. Shell Programming and Scripting

splitting tab-delimited file with awk

Hi all, I need help to split a tab-delimited list into separate files by the filename-field. The list is already sorted ascendingly by filename, an example list would look like this; filename001 word1 word2 filename001 word3 word4 filename002 word1 word2 filename002 word3 word4... (4 Replies)
Discussion started by: perkele
4 Replies

10. UNIX for Advanced & Expert Users

Help with splitting lines in a file using awk

I have a file which is one big long line of text about 10Kb long. Can someone provide a way using awk to introduce carriage returns every 40 chars in this file. Any other solutions would also be welcome. Thank you in advance. (5 Replies)
Discussion started by: martinbarretto
5 Replies
Login or Register to Ask a Question