Oracle 11g script read from file in where clause (RHEL 5.7)


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Oracle 11g script read from file in where clause (RHEL 5.7)
# 1  
Old 01-09-2014
Hammer & Screwdriver Oracle 11g script read from file in where clause (RHEL 5.7)

Hi everyone,
Simple question.
I have a comma-delimited file, aux.txt, with the following contents (all in one line):
Code:
'value1','value2','value3',...,'valueN'

I would like to know if there's a way I can use that file inside a sql script in a where clause, like so:
Code:
select myfield1 from mytable
where myfield2 IN ('value1','value2','value3',...,'valueN')

This works if I hard code the values 'value1','value2','value3',...,'valueN' into the script. But the contents of the file aux.txt will be changing over time, so I'm looking for a way to read its contents directly into the script.
I read here that doing this would do, but it didn't work:
Code:
select myfield1 from mytable
where myfield2 IN 
(
@aux.txt
)

I have also tried to include the absolute path to aux.txt, but to no use.
When I run this query, I get the following error: ORA-00936: Missing Expression.
Any ideas or hints will be more than appreciated.
# 2  
Old 01-09-2014
You could do something like this:
Code:
args="$( cat aux.txt )"
sqlplus -s user/pass@inst << EOF
select myfield1 from mytable
where myfield2 IN ( $args );
exit
EOF

This User Gave Thanks to Yoda For This Post:
# 3  
Old 01-10-2014
Quote:
Originally Posted by Yoda
You could do something like this:
Code:
args="$( cat aux.txt )"
sqlplus -s user/pass@inst << EOF
select myfield1 from mytable
where myfield2 IN ( $args );
exit
EOF

Yoda,
Thank you so very much for taking the time to read and reply to my post.
It ALMOST worked. I had forgotten to mention that the aux.txt file is 66k in size (that is the current size, but it can be up to twice as big) and contains up to 12,000 values separated by commas, like 'value1','value2','value3',...,'value12000', which seems to be a little too much for sqlplus and Oracle 11g:
Code:
USER SQL>  2  Input truncated to 7499 characters
SP2-0027: Input is too long (> 2499 characters) - line ignored
  2  Input truncated to 7499 characters
SP2-0027: Input is too long (> 2499 characters) - line ignored
  2  Input truncated to 7499 characters
SP2-0027: Input is too long (> 2499 characters) - line ignored
  2  Input truncated to 7499 characters
SP2-0027: Input is too long (> 2499 characters) - line ignored
  2  SP2-0027: Input is too long (> 2499 characters) - line ignored
  2  Input truncated to 7499 characters
SP2-0027: Input is too long (> 2499 characters) - line ignored
  2  Input truncated to 7499 characters
SP2-0027: Input is too long (> 2499 characters) - line ignored
  2  Input truncated to 7499 characters
SP2-0027: Input is too long (> 2499 characters) - line ignored
  2  Input truncated to 7499 characters
SP2-0027: Input is too long (> 2499 characters) - line ignored
  2  SP2-0027: Input is too long (> 2499 characters) - line ignored
  2    3  Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

I also tried to supply the whole file as a argument to the sql script while calling it but the same limitation appears.
I'll mention the original issue in case you or someone else can suggest another workaround.
The contents of the aux.txt file are originally the results of another SQL query spooled to a text file, which are ordered by a date field.
My first idea was to do this:
Code:
select myfield1 from mytable
where myfield2 IN 
(
[SQL query that returns only one field (myfield2 in this example) - the contents of the aux.txt file in table format.
)

I also thought about writing a shell script that would query the database for each value in the aux.txt file, but as I mentioned earlier, this file can contain up to 12,000 records and you'll understand that I don't want to run such a number of queries.
That being said, all suggestions and hints will be more than welcome.
# 4  
Old 01-10-2014
You may be better to load the values into a temporary table and then use that in you query like this:-
Code:
....
WHERE myfield in (SELECT * from temp_table)
....

Even so, if you are hitting an Oracle limitation, then you are a bit stuck. You might have to split the input up and loop.


I hope that this helps,
Robin
This User Gave Thanks to rbatte1 For This Post:
# 5  
Old 01-10-2014
[SOLVED] Oracle 11g script read from file in where clause (RHEL 5.7)

Thanks Robin and Yoda!
I also tried the temporary table workaround but my user does not have sufficient privileges.
In the meanwhile, the DBA (who was supposed to provide me a sql query in the first place) found a query that we can use and I just tried it - and it worked.
That being said, I believe this thread will serve as a reference for other users in the future. I will mark it as solved and will click the Thanks button in each of your posts.
# 6  
Old 01-10-2014
Looks like you've hit a limitation of SQL*Plus. The documentation of SQL*Plus shipped with Oracle 11g R2 at this link:

SQL*Plus Limits

mentions that limit for the command-line length is 2500 characters. That explains the "SP2-0027" errors you encountered.

In your case, since the data was simply a list of comma-delimited values, you could split and align them so that each line is less than 2500 characters. A short Perl program to do that is provided below:

Code:
$
$ cat -n align.pl
     1  #!/usr/bin/perl -w
     2  use strict;
     3
     4  my ($file, $line_size) = @ARGV;                         # read input parameters
     5  open (FH, "<", $file) or die "Can't open $file: $!";
     6  chomp(my $str = <FH>);                                  # read the comma-delimited line into a variable
     7  close (FH) or die "Can't close $file: $!";
     8
     9  my $from_index = 0;                                     # set the from and to indices at the beginning
    10  my $to_index = index($str, ",", $from_index);           # of the string
    11  my $next_index;
    12  my $done = 0;
    13
    14  while ($to_index != -1) {                               # while we still have commas in the string
    15      $next_index = index($str, ",", $to_index + 1);      # determine the position of the next comma
    16      if ($next_index > $line_size - 1) {                 # and if it is higher than the line size,
    17          print substr($str, 0, $to_index + 1), "\n";     # cut the string and print it
    18          $str = substr($str, $to_index + 1);
    19          $from_index = 0;
    20      } elsif ($next_index == -1) {                       # if the next position of the comma is not found
    21          if (length($str) > $line_size) {                # and length of string is greater than the line size
    22            print substr($str, 0, $to_index + 1), "\n";   # then print till the current position of comma
    23          } else {
    24            $done = 1;                                    # otherwise print the string and we are done
    25            print $str, "\n";
    26            last;
    27          }
    28          $str = substr($str, $to_index + 1);             # reset the string so that the printed part is
    29          $from_index = 0;                                # discarded, and reset the from index as well
    30      } else {
    31          $from_index = $to_index + 1;                    # the comma position is within the line size, so
    32      }                                                   # just go to the next iteration
    33      $to_index = index($str, ",", $from_index);
    34  }
    35  if (not $done) {                                        # print only if a chunk was left out at the end
    36      print $str, "\n";
    37  }
$
$ echo "abc,def,ghi,jk,lm,n,opqr,stu,vwx,yz" > mydata.txt
$
$ cat -n mydata.txt
     1  abc,def,ghi,jk,lm,n,opqr,stu,vwx,yz
$
$ perl align.pl mydata.txt 6
abc,
def,
ghi,
jk,lm,
n,
opqr,
stu,
vwx,yz
$
$ perl align.pl mydata.txt 7
abc,
def,
ghi,jk,
lm,n,
opqr,
stu,
vwx,yz
$
$ perl align.pl mydata.txt 8
abc,def,
ghi,jk,
lm,n,
opqr,
stu,vwx,
yz
$
$ perl align.pl mydata.txt 12
abc,def,ghi,
jk,lm,n,
opqr,stu,
vwx,yz
$
$ perl align.pl mydata.txt 20
abc,def,ghi,jk,lm,n,
opqr,stu,vwx,yz
$
$

The program reads the single line from the specified file and cuts it into chunks of the specified line size.

You could then use this program to create a SQL script and run it like so:

Code:
perl align.pl aux.txt 2400 > mydata.txt
echo "select myfield1 from mytable where myfield2 IN (" > exec.sql
cat mydata.txt >> exec.sql
echo ");" >> exec.sql
echo >> exec.sql
sqlplus -s user/password@db @exec.sql

However, you will still hit the limitation if any token in the "IN list" is of length 2500 characters or higher. Most likely, SQL*Plus would not be a good choice in that case.

And as Robin mentioned earlier, you'd have to load such data into a temporary table, perhaps using sqlldr or external tables, and then run your query accordingly.
This User Gave Thanks to durden_tyler For This Post:
# 7  
Old 01-10-2014
durden_tyler,
I don't have words to thank you enough for such a detailed answer, for the SQL*PLUS documentation and for the perl script.
Being a sysadmin, this wasn't actually my responsibility, but the DBA's to come up with the right query that I could use in a shell script (which in turns, formats the results of a set of queries in html and sends the output via email). That is why my user does not have sufficient privileges to use sqlldr, create a view (another thing I thought of) or a temporary table.
Anyway, I was glad that this guy finally came up (or found in his docs) the right query to use with SQL*PLUS.
I am glad I submitted the post because I got such excellent answers.
Thanks again and look forward to learning more from Yoda, Robin, and you!
Login or Register to Ask a Question

Previous Thread | Next Thread

2 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

ksh scripts migrating(Oracle 11G) from Solaris Sparc 10 to RHEL 7

Hi All, Now we are migrating oracle 11G from Solaris Sparc 10 to RHEL 7. We have 1000+ ksh scripts.. Could you please let em know what would be the best way to use exiting scripts in RHEL with minimal changes,. my concern was "Is it all Solaris command work in RHEL". (3 Replies)
Discussion started by: mssprince
3 Replies

2. Shell Programming and Scripting

Switching user to oracle to connect Oracle 11g DB with 'sysdba'

I need to connect my Oracle 11g DB from shell script with 'sysdba' permissions. To do this I have to switch user from 'root' to 'oracle'. I've tried the following with no success. su - oracle -c "<< EOF1 sqlplus -s "/ as sysdba" << EOF2 whenever sqlerror exit sql.sqlcode;... (2 Replies)
Discussion started by: NetBear
2 Replies
Login or Register to Ask a Question