Visit Our UNIX and Linux User Community


Dynamically checking rules in database tables using SQL


 
Thread Tools Search this Thread
Top Forums Programming Dynamically checking rules in database tables using SQL
# 1  
Old 08-23-2013
Dynamically checking rules in database tables using SQL

I want to check for rows in a table where all values (except the key) is empty. I am using MySQL 5.5.
I plan to do this mechanically, so the approach should work for any table in my database schema.

Suppose for illustration purposes I start with the following table:
Code:
CREATE TABLE `sources` (
  `source` varchar(3) NOT NULL,
  `description` varchar(24) DEFAULT NULL,
  `pricedata` varchar(24) DEFAULT NULL,
  `tablename` varchar(24) DEFAULT NULL,
  `reference` varchar(64) DEFAULT NULL,
  PRIMARY KEY (`source`)
) DEFAULT CHARSET=latin1;

Then my initial steps are:
Code:
USE information_schema;
SET @myTable = 'sources';
PREPARE statement FROM "SELECT GROUP_CONCAT(column_name SEPARATOR ', ') AS columns FROM columns WHERE table_name = @myTable AND column_key = ' '";
EXECUTE statement;

This yields:
Code:
+----------------------------------------------+
| columns                                      |
+----------------------------------------------+
| description, pricedata, tablename, reference |
+----------------------------------------------+

How do I use this result to be able to construct and execute a statement as follows?:
Code:
SELECT * FROM sources WHERE [description] IS NULL AND [pricedata] IS NULL AND [tablename] IS NULL AND [reference] IS NULL;

That's obviously pseudo-code for illustration purposes, but the idea is that the approach should work for all tables.

I tried something as follows:
Code:
SELECT CONCAT(SELECT * FROM @myTable WHERE ', GROUP_CONCAT(column_name SEPARATOR ' IS NULL AND '), ' IS NULL;) FROM columns WHERE table_name = @myTable;

but that was a little too clever and resulted in an error message. Within the CONCAT a nested prepare-statement might work, but the syntax evades me.

So, how can I dynamically check for rows having all nulls?
# 2  
Old 08-24-2013
Quote:
Originally Posted by figaro
...This yields:
Code:
+----------------------------------------------+
| columns                                      |
+----------------------------------------------+
| description, pricedata, tablename, reference |
+----------------------------------------------+

How do I use this result to be able to construct and execute a statement as follows?:
Code:
SELECT * FROM sources WHERE [description] IS NULL AND [pricedata] IS NULL AND [tablename] IS NULL AND [reference] IS NULL;

...
Code:
mysql>
mysql> --
mysql> select column_name
    ->   from columns
    ->  where table_name = 'sources'
    ->    and column_key = '';
+-------------+
| column_name |
+-------------+
| description |
| pricedata   |
| tablename   |
| reference   |
+-------------+
4 rows in set (0.04 sec)

mysql>
mysql>
mysql> --
mysql> select concat ('select * from sources where ', substring(group_concat(concat('AND ',column_name,' IS NULL') separator ' '),5), ';') as x
    ->   from columns
    ->  where table_name = 'sources'
    ->    and column_key = '';
+--------------------------------------------------------------------------------------------------------------------+
| x                                                                                                                  |
+--------------------------------------------------------------------------------------------------------------------+
| select * from sources where description IS NULL AND pricedata IS NULL AND tablename IS NULL AND reference IS NULL; |
+--------------------------------------------------------------------------------------------------------------------+
1 row in set (0.01 sec)

mysql>
mysql>
mysql>
mysql> --
mysql> -- Use a variable instead of hard-coding the table name
mysql> --
mysql> set @x = 'sources';
Query OK, 0 rows affected (0.00 sec)

mysql>
mysql>
mysql> --
mysql> select concat ('select * from ', @x, ' where ', substring(group_concat(concat('AND ',column_name,' IS NULL') separator ' '),5), ';') as x
    ->   from columns
    ->  where table_name = @x
    ->    and column_key = '';
+--------------------------------------------------------------------------------------------------------------------+
| x                                                                                                                  |
+--------------------------------------------------------------------------------------------------------------------+
| select * from sources where description IS NULL AND pricedata IS NULL AND tablename IS NULL AND reference IS NULL; |
+--------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

mysql>
mysql>

---------- Post updated at 12:24 AM ---------- Previous update was at 12:15 AM ----------

Or another one:

Code:
mysql>
mysql> --
mysql> select concat ('select * from ', @x, ' where ', group_concat(column_name separator ' is null and '), ' is null;') as x
    ->   from columns
    ->  where table_name = @x
    ->    and column_key = '';
+--------------------------------------------------------------------------------------------------------------------+
| x                                                                                                                  |
+--------------------------------------------------------------------------------------------------------------------+
| select * from sources where description is null and pricedata is null and tablename is null and reference is null; |
+--------------------------------------------------------------------------------------------------------------------+
1 row in set (0.04 sec)

mysql>
mysql>

---------- Post updated at 12:53 AM ---------- Previous update was at 12:24 AM ----------

If you want to do something similar for a bunch of tables, then you could assign the table list to a variable and use it in your query.

Over here, I've generated relevant queries for a few tables of the Sakila database:

Code:
mysql>
mysql> --
mysql> set @set_of_tables = 'actor,city,country';
Query OK, 0 rows affected (0.00 sec)

mysql> set @schema = 'sakila';
Query OK, 0 rows affected (0.00 sec)

mysql>
mysql> --
mysql> select table_name, column_name, column_key
    ->   from columns
    ->  where find_in_set (table_name, @set_of_tables)
    ->    and table_schema = @schema;
+------------+-------------+------------+
| table_name | column_name | column_key |
+------------+-------------+------------+
| actor      | actor_id    | PRI        |
| actor      | first_name  |            |
| actor      | last_name   | MUL        |
| actor      | last_update |            |
| city       | city_id     | PRI        |
| city       | city        |            |
| city       | country_id  | MUL        |
| city       | last_update |            |
| country    | country_id  | PRI        |
| country    | country     |            |
| country    | last_update |            |
+------------+-------------+------------+
11 rows in set (0.04 sec)

mysql>
mysql> --
mysql> select concat ('select * from ', table_name, ' where ', group_concat(column_name separator ' is null and '), ' is null;') as x
    ->   from columns
    ->  where find_in_set (table_name, @set_of_tables)
    ->    and table_schema = @schema
    ->    and column_key != 'PRI'
    ->  group by table_name;
+---------------------------------------------------------------------------------------------+
| x                                                                                           |
+---------------------------------------------------------------------------------------------+
| select * from actor where first_name is null and last_name is null and last_update is null; |
| select * from city where city is null and country_id is null and last_update is null;       |
| select * from country where country is null and last_update is null;                        |
+---------------------------------------------------------------------------------------------+
3 rows in set (0.18 sec)

mysql>
mysql>

# 3  
Old 08-24-2013
Thank you for your response. So how do I then execute the generated SELECT statements?
# 4  
Old 08-24-2013
Quote:
Originally Posted by figaro
... So how do I then execute the generated SELECT statements?
(1) Create a script to generate such queries.
(2) Pass the script to the mysql command and redirect the output to a file.
(3) And then pass the queries.sql containing the generated queries to the mysql command.
# 5  
Old 08-24-2013
Yes, I was contemplating the export and then an import also. I see that more of a last resort option, because it involves writing to a file together with all its shortcomings.

The other solution we are investigating is through the use of cursors. This looks roughly as follows:
- create a list of all tables in the schema
- loop through these tables to create a list of all columns per table which are not keys
- loop through these columns to pick up rows where all values are NULL

Disadvantage here is that it requires the use of a procedural language as well as being less portable to other database systems.

Are you saying it is impossible to do entirely within MySQL in (possibly nested) SQL statements?

Previous Thread | Next Thread
Test Your Knowledge in Computers #792
Difficulty: Easy
In mathematics and digital electronics, a binary number is a number expressed in the base-2 numeral system or binary numeral system.
True or False?

8 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Identify tables from Oracle sql scripts

Hi, Please let me know if you have any thoughts on how to read a table that has all the oracle sql files or shell scripts at the job and step level to identify all the tables that does merge, update, delete, insert, create, truncate, alter table (ALTER TABLE XYZ RENAME TO ABC) and call them out... (1 Reply)
Discussion started by: techmoris
1 Replies

2. Shell Programming and Scripting

Help with database checking script

I haven't coded in ksh in forever and this script is proving me not a great coder. I keep getting unexpected end of file when I try to execute this...Line 94 is the last line of the code.. #!/usr/bin/ksh . /home/oraprod/.bash_profile DBSID=prod # Database sid CONNECTSTR=apps/xxxxxxxx@prod #... (6 Replies)
Discussion started by: kirkladb
6 Replies

3. Programming

Query to join two tables in SQL

Hi, I have 2 tables. Table1 name map_code data_code id aaa 2732C 3333B 10 bbb 1223F 2545v 11 ccc 12 Table2 name map_code data_code id text_description aaa 2732C 3333B 10 description 1 bbb 1223F 2545v 11 ... (3 Replies)
Discussion started by: vanitham
3 Replies

4. Shell Programming and Scripting

Passing filename dynamically in SPOOL of SQL*PLUS in shell script

Hi all, I am executing shell script in which I am using SQLLDR In this SQLLDR I am passing text file having PL/SQL script. This script will produce some formated output, this output I have to spool in another text file. Currently I have given this in script file as following Spool... (2 Replies)
Discussion started by: shekharjchandra
2 Replies

5. Programming

SQL Add to Multiple Tables

I'm pretty new to the database world and I've run into a mental block of sorts. I've been unable to find the answer anywhere. Here's my problem: I have several tables and everything is as normalized as possible (as I've been lead to understand normalization.) Normalization has lead to some... (1 Reply)
Discussion started by: flakblas
1 Replies

6. Shell Programming and Scripting

Tables to query to find users for database from shell script

I am coding shell script. I need to connect to different databases like DB2, Oracle and Sybase. I would then need to query tables where it has all the groups, users for that database. I would also need who has what kind of permissions. EG: I know for DB2 some TABAUTH table needs to be... (0 Replies)
Discussion started by: pinnacle
0 Replies

7. Shell Programming and Scripting

Automating back up of database tables

Hi all, i would like to set a cronjob to back up my db tables. I have written the par file, i would like to know how can i execute a exp command in a shell script. my par file userid=admin/password file=backup.dmp log=backupLOG.log tables= (Tables 1, Tables 2)My cron job script will be... (1 Reply)
Discussion started by: new2ss
1 Replies

8. UNIX for Advanced & Expert Users

writing database tables into a file in unix

I want to collect the database tables(Ex: Emp,Sal,Bonus...etc) in a file & give this file at the command prompt, when prompted for the tables list at command prompt. How can i do that ? Ex:- Import jason/jason1 tables=emp,sal,bonus log=j1.log i want to change this into Ex:- Import... (3 Replies)
Discussion started by: dreams5617
3 Replies

Featured Tech Videos