Wrap lines with awk to create SQL script


Login or Register for Dates, Times and to Reply

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Wrap lines with awk to create SQL script
# 1  
Network Wrap lines with awk to create SQL script

Greetings!

Some of my files list hardware errors (we test electronic components), some have none. If the file name has no errors, I still want to display a message like "No error", else I display the error from the file itself.

I came up with this (with help)

Code:
for myfile in `find . -name "R*VER" -mtime +1`
do
     somestr=`grep -H ^err $myfile || echo -e "$myfile\tNo error"`
     echo "$somestr" |sed 's:./::;s:tmp/::;s/:/\t/;'
     echo $somestr | awk -F~ '{print $1"\t"$2"\t"$7"\t"$8"\t"$9"\t"$10}' | sed 's:./::;s:tmp/::;'
done

It works but outputs on two lines i.e.

Code:
RRR1~COS~COSMETICS~99537~jgmdtv132~1~P~R22-200~029053252648~20110607~094718.VER    No error
RRR1    COS    P    R22-200    029053252648    20110607
RRR1~COS~COSMETICS~ETT03~jgm14652.~1~F~R16-300~000894980523~20110607~084053.VER    err    ->IR Remote Key 1    3310    err   
RRR1    COS    F    R16-300    000894980523    20110607

I am looking to generate dynamically a SQL script with insert statements like so:

Code:
INSERT INTO MYTABLE (COL1,COL2,COL3,COL4,COL5,COL6,COL7,COL8)
VALUES ('RRR1~COS~COSMETICS~99537~jgmdtv132~1~P~R22-200~029053252648~20110607~094718.VER','No error','RRR1','COS','P','R22-200','029053252648','20110607');
INSERT INTO MYTABLE (COL1,COL2,COL3,COL4,COL5,COL6,COL7,COL8)
VALUES  ('RRR1~COS~COSMETICS~ETT03~jgm14652.~1~F~R16-300~000894980523~20110607~084053.VER',' err    ->IR Remote Key 1    3310    err','RRR1','COS','P','R16-300','000894980523','20110607');

All I would need to do is wrap both lines but I am not clear how to do it (still learning awk/sed). Is it possible?

Thanks for your help.

Last edited by alan; 06-15-2011 at 09:53 PM..
# 2  
Yep, it's possible. Here is one way:

Code:
#!/usr/bin/env ksh

# if you have a lot of these files, this method will
# prevent arg length errors to the for
find . -name "R*VER" -mtime +1 | while read fname
do
    ( grep "^err" $fname || echo "No error" ) | awk  -v base=${fname##*/} '
        {
            split( base, a, "~" );          # split filename into components
            printf( "%s\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n", base, $0, a[1], a[2], a[7], a[8], a[9], a[10] );
        }'
done

With some minor changes, it will generate your SQL (based on what you provided; I don't do SQL so I don't know if it's exactly what you need, but you'll get the drift):

Code:
#!/usr/bin/env ksh

# if you have a lot of these files, this method will
# prevent arg length errors to the for
find . -name "R*VER" -mtime +1 | while read fname
do
    ( grep "^err" $fname || echo "No error" ) | awk -v squote="'" -v base=${fname##*/} '
        {
            split( base, a, "~" );          # split filename into components
            printf( "INSERT INTO MYTABLE (COL1,COL2,COL3,COL4,COL5,COL6,COL7,COL8)\n" );
            str = sprintf( "VALUES (\"%s\",\"%s\",\"%s\",\"%s\",\"%s\",\"%s\",\"%s\",\"%s\")", base, $0, a[1], a[2], a[7], a[8], a[9], a[10] );
            gsub( "\"", squote, str );          # replace double quotes with singles
            print str;
        }'
done

The trick with the single quote, and substitution into str gets round the difficulty of using a single quote inside of an awk programme. It's possible, very messy, and makes for code that is difficult to maintain. There are other ways to do the quoting that are more efficient, but I think this is the most readable.

Other things that might not be too obvious:
${fname##*/} strips the leading path (./tmp/xxx/) from the filename returned by find.

The parens round this statement( grep "^err" $fname || echo "No error" ) cause the statement to be executed in a subshell. This is necessary to cause output from the grep and the echo to pipe properly into the awk.

Hope this helps.

Last edited by agama; 06-16-2011 at 12:37 AM.. Reason: clarification
This User Gave Thanks to agama For This Post:
# 3  
Thanks for taking the time. With a couple of friends, this is what we got

Code:
find . -name "R*VER" -exec sh -c 'grep -H ^err "{}" || echo "{}:No error"' \; |
        awk -F: '
                BEGIN { q="\047" }
                {
                        file=$1
                        result=$2
                        sub(/^.*\//,"",file)
                        split(file,a,/~/)
                        print "INSERT INTO MYTABLE (COL1,COL2,COL3,COL4,COL5,COL6,COL7,COL8) VALUES (" q file q "," q result q "," q a[1] q "," q a[2] q "," q a[7] q "," q a[8] q "," q a[9] q "," q a[10] q ")"
                }
        '

Works great. Thanks again.
Login or Register for Dates, Times and to Reply

Previous Thread | Next Thread
Thread Tools Search this Thread
Search this Thread:
Advanced Search

Test Your Knowledge in Computers #574
Difficulty: Medium
The process of allocating and deallocating memory is generally referred to as memory swapping.
True or False?

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to create SQL statement out of data using shell script?

Table TAB1 contains following example data (its a tree sitting in table data format & its driven based CHILD & PARENT column pick the RULE condition to generate the below SQL: CHILD PARENT SS MID MNM VNM RULE FLG 1 ? S1 ? ? V1 rule004 I 2 1 S1 ? ? V1 0 Z 3 1 S1 ? ? V1 1 Z ... (6 Replies)
Discussion started by: gksenthilkumar
6 Replies

2. Shell Programming and Scripting

Perl one liner to wrap comment lines

Greetings fellow scripters. I find myself editing multiple files, sometimes with the same bits of information. My bash script, a changelog, and a plist file (OS X). Once I realized this, I thought why not script part of this process (and so it begins). In any case, I've solved several of the... (1 Reply)
Discussion started by: reid
1 Replies

3. Shell Programming and Scripting

Using awk and sed to modify a create sql script

Hi, I have a file which contains the following data claim_src|clm_id,typ_id pat_src|pat_id prov_src|prov_id,clm_id,prov_name The first field is table name and second field is primary keys of the table Now I have three files which contain ddl of each table. clam_src.sql... (4 Replies)
Discussion started by: wahi80
4 Replies

4. Shell Programming and Scripting

Please help, need to create script to remove lines by date in file

Please Help (novice to PERL and SHELL scripting)…. Need to create a script which removes all lines in $filename = "cycle_calendar_ftp_out" older than current date – a variable which will be a number of days passed to script. For Ex it will look at the end date which is the last field (4) and... (2 Replies)
Discussion started by: m3pwr
2 Replies

5. UNIX for Dummies Questions & Answers

Bash does not wrap long lines correctly

Ksh is my default shell, but I want use the bash shell since its convenient to me. When I type a long command line in a terminal, it does not wrap to the next line when I reach the end of the line and it wraps onto the same line, overwriting my prompt and the rest of what I typed. $... (5 Replies)
Discussion started by: senthil.ak
5 Replies

6. UNIX for Advanced & Expert Users

SQL script with 86000 lines: new files with only 10000 lines (per file)

Hi this is my SQL script $ wc -l insert_into_customers.sql 85601 insert_into_customers.sqlI wish to cut this file into 9 files each 10000 lines (the last one less) $ wc -l insert_into_customers_00*.sql 10000 insert_into_customers_001.sql 10000 insert_into_customers_002.sql ... (1 Reply)
Discussion started by: slashdotweenie
1 Replies

7. Shell Programming and Scripting

Create SQL DML insert statements from file using AWK or similar

Hi all. This is my first post on this forum. I've previously found great help in the huge knowledgebase that is here, but this time I have not been able to find a solution to my problem. I have a large text file that looks like this: typedef struct ABC_struct_nbr1_ { char attr1; /*... (0 Replies)
Discussion started by: Yagi Uda
0 Replies

8. Programming

create a spool file based on values passed from korn shell to sql script

this is my issue. 4 parameters are passed from korn shell to sql script. parameter_1= varchar2 datatype or no value entered my user. parameter_2= number datatype or no value entered my user. parameter_3= number datatype or no value entered my user. parameter_4= number datatype or no... (5 Replies)
Discussion started by: megha2525
5 Replies

9. Shell Programming and Scripting

Wrap Interactive Script

Does anyone know of a program to wrap an interactive script into an application..I tried using platypus but i want a utility to allow interactive scripts to be run in a stand-alone window to avoid .profile settings on multiple computers...platypus provides the option of a text window output but... (0 Replies)
Discussion started by: meskue
0 Replies

10. Shell Programming and Scripting

Should I use sed/ grep/awk for wrap file?

Hi, This is my first time post a new thread. I have been trying to work on this for the past 2 days and could not find any good solution. I have 1 long long line ( EDI wrapped file) like below: NEW*SR*04411763447*279*278*Q~*ZR*AAV*SR*04511763460*SQ*21B37F04~HL*305*304*Q~K~SN1*1*1*SR*05511763461*... (6 Replies)
Discussion started by: vanda_25
6 Replies

Featured Tech Videos