looping and awk/sed help


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting looping and awk/sed help
# 1  
Old 06-28-2005
looping and awk/sed help

I am pretty new to this, but imagine what I am trying to do is possible

iI am trying to make an automated DB comparison tool that selects all columns in all tables and compares them to the same thing in another DB.

anyway I have created 2 files to help with this

the first file is a distinct list of table_names (used in the first for loop)

the second file contains all the column_names for all tables
table_name column_name
table_1 column_1
table_1 column_2
table_2 column_1
...etc

Code:
#! /bin/sh

#for each table in tables_list get all column_names
for table in `cat tables_list`
do
        ctr=0
        echo "\ntable name is $table"
        for col in `grep $table columns_list | awk {'print $2'}`
        do
                ((ctr=ctr+1))
                echo "column $ctr name is $col"
        done
done

I am stuck here trying to get all the column_names in one long string so I can insert them into an SQL statement.

like this "col_1 , col_2, col_3 ..."

can anyone help me save the column_names like above?

another potential problem is that a few of the tables have over 300 columns!!

Thanks,
Zelp
# 2  
Old 06-28-2005
use tr

get all the columns in a file and use this :-

cat column_list_file | tr '\n' ','
# 3  
Old 06-28-2005
nawk -f selp.awk table_namesFile column_namesFile

here's zelp.awk:
Code:
BEGIN { SEPlist="," }
FNR==NR { tbl[$1]; next}
{ 
  if ( $1 in tbl )
     out[$1] = ($1 in out) ? out[$1] SEPlist $2 : $2
}
END {
   for ( i in out)
      print i, out[i]
}

# 4  
Old 06-28-2005
okay that gets the output I need, but I am not sure how to use it the way I need to.

I have a template file I created that contains the sql statements and placeholders for columns_names and table_names

select * from #@connection1
minus
select * from #@connection2;

select * from #@connection1
minus
select * from #@connection3;
...etc for 25 diff DB's

i want to insert the column_name list for all the *'s in the file
and I want to replace the table_name for all the #'s in the file

then I want to save 1 set of sql's for each table so I can run them and create a log of any diffs form all DB's

sorry for so many questions, I am still very new to awk/sed wasn't sure which to use or how, thanks so much.
Zelp
# 5  
Old 06-28-2005
how do you map a 'tableName' to the 'connectionNumber'?
can you put that mapping in your 'table_namesFile' file like so:
Code:
table_1 connection1
table_2 connection2
table_3 connection3

Code:
BEGIN { SEPlist="," }
FNR==NR { tbl[$1]; next}
{ 
  if ( $1 in tbl )
     out[$1] = ($1 in out) ? out[$1] SEPlist $2 : $2
}
END {
   for ( i in out)
      printf("select %s from %s@connection1\n",  out[i], i)
}

# 6  
Old 06-28-2005
I guess there is more explanation needed.
We have 25 DB's that have identical schema (the same table exists in all of them) but the data is different.

we are hoping to combine all of them into 1 DB, first I am trying to find all the differences between each table in each DB's to investigate them before we just put them all together.

so you see the template I have already contains all the connections in the SQLs, I just want to replace all the *'s and #'s and save that as a new sql named after the table all the sql's are for. So now the new sql created will select the same columns from the same tables in all 25 DB's for the comparison.

am I making any sense anymore?

maybe in the second portion instead of a print, I can do a replace of the table_names and column_names into a new file based on my template?
# 7  
Old 06-28-2005
something like this?
Code:
BEGIN { 
    SEPlist="," 
    schemasN=split("connection1 connection2 connection3 connection4", schemasA, " ")
}
FNR==NR { tbl[$1]; next}
{ 
  if ( $1 in tbl )
     out[$1] = ($1 in out) ? out[$1] SEPlist $2 : $2
}
END {
   for(s=2; s <= schemasN; s++) 
     for ( i in out)
        printf("select %s from %s@%s\nminus\nselect %s from %s@%s\n",  out[i], i, schemasA[1], out[i], i, schemasA[s])
}


Last edited by vgersh99; 06-28-2005 at 03:07 PM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk nested looping?

I am trying to parse a text file and send its output to another file but I am having trouble conceptualizing how I am supposed to do this in awk. The text file has a organization like so: Name Date Status Location (city, state, zip fields) Where each of these is on a separate line in... (1 Reply)
Discussion started by: kellyanneghj
1 Replies

2. UNIX for Beginners Questions & Answers

Looping and sed

Hi, I've got text files (say file1 and file2) in a directory (say directory1) with several columns like this: a b c d a b c de f g h e f g hI need to add a header to the files in directory1 and the names of the columns must contain an identifier from another text file (say file3) like this:... (4 Replies)
Discussion started by: zajtat
4 Replies

3. Shell Programming and Scripting

Help on looping using awk

I have the data like this: PONUMBER,SUPPLIER,LINEITEM,SPLITLINE,LINEAMOUNT,CURRENCY IR5555,Supplier1,1,1,83.1,USD IR5555,Supplier1,1,3,40.4,USD IR5555,Supplier1,1,6,54.1,USD IR5555,Supplier1,1,8,75.1,USD IR5556,Supplier2,1,1,41.1,USD IR5556,Supplier2,1,3,43.1,USD ... (3 Replies)
Discussion started by: jeffreybsu
3 Replies

4. Shell Programming and Scripting

AWK looping over 2 variables

I would like to loop over variables i and j consecutively, { a = -6.7 b = 7.0 c =0.1 { for (i = 0; i<=(b-a)/c; i++) for (j = 1; j<=(b-a)/c; j++) '$1<=(a+j*c)&&$1>=(a+i*c)' FILENAME > output_j '{print $2}' output_j > output_j_f } I essentially want to print the range of $1... (9 Replies)
Discussion started by: chrisjorg
9 Replies

5. Shell Programming and Scripting

looping in awk

How do I remove last comma? echo "xx yy zz" | awk 'BEGIN{FS=" "}{for (i=1; i<=NF; i++) printf "%s,", $i}'output: xx,yy,zz, required output: xx,yy,zz or (ideally!): xx, yy & zz many thanks in advance! (4 Replies)
Discussion started by: euval
4 Replies

6. Shell Programming and Scripting

Sed pattern space/looping conundrum

Although my sed skills are gradually developing, thanks in large part to this forum, I'm having a hard time dealing with pattern space and looping, which I suspect is what I'll need a better handle on to figure out my current issue, which is converting a multi line file like this: ... (4 Replies)
Discussion started by: tiggyboo
4 Replies

7. UNIX for Dummies Questions & Answers

Help with AWK looping

I'm trying to parse a configuration text file using awk. The following is a sample from the file I'm searching. I can retrieve the formula and recipe names easily but now I want to take it one step farther. In addition to the formula name, I would like to also get the value of the attribute... (6 Replies)
Discussion started by: new2awk
6 Replies

8. Shell Programming and Scripting

Looping and using Sed

Hi guys I having som problem trying to use sed to get a file and insert inside another one. I'll try to explain better. I have a base.txt and using sed(having a array variables) I'm chaging some strings inside this file and saving as base1.txt. until here okay. Then, I have to get this... (4 Replies)
Discussion started by: digobh
4 Replies

9. Shell Programming and Scripting

Awk: looping problem!

I am having a problem with awk when I run it with a loop. It works perfectly when I echo a single line from the commandline. For example: echo 'MFG009 9153852832' | awk '$2 ~ /^0-9]$/{print $2}' The Awk command above will print field 2 if field 2 matches 10 digits, but when I run the loop... (5 Replies)
Discussion started by: cstovall
5 Replies

10. UNIX for Advanced & Expert Users

Looping in awk

Can somebody give me a cleaner way of writing the following script. I was thinking that I could use a loop in the awk statement. It works fine the way it is but I just want the script to be cleaner. #!/usr/bin/sh for r in 0 1 2 3 4 5 6 do DAY=`gdate --date="${r} days ago" +%m\/%d\/%y`... (3 Replies)
Discussion started by: keelba
3 Replies
Login or Register to Ask a Question