Formation of sql files


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Formation of sql files
# 1  
Old 12-08-2011
Lightbulb Formation of sql files

Dear Experts,

I have a scenario where I have to form sql queries in a file using echo command.
I have file1 which has more than 10 lac numbers. For each Number in file 1, I need to create 5 sql queries resulting in 50 lac queries totally.

For doing this I use the below simple method,
Code:
for i in `cat file1 | tr -d '\r'`
do
echo "update table set id1=0 where no='$i';" >>query.unl
echo "update table set id2=0 where no='$i';" >>query.unl
echo "update table set id3=0 where no='$i';" >>query.unl
echo "update table set id4=0 where no='$i';" >>query.unl
echo "update table set id5=0 where no='$i';" >>query.unl
done

It takes me more than 12 hours to form the query.unl. Any other efficient and simple way to do this without modifying the file 1 (source file) ???
# 2  
Old 12-08-2011
I'm not surprised... That's a dangerous use of backticks and useless use of cat. It works as a quick fix only but when you use it for anything big, as you've discovered, it has major problems. Like storing the entire file in memory while it's working on it, and possibly truncating the end if it's just too large.

It also produces 5 update statements when you only needed 1, and reopens the query file 5000 times to process 1000 lines when you only needed to open it once for all of them.

Code:
SPLIT="$(printf "\r\n");

while IFS="${SPLIT}" read i
do
        echo "update table set id1=0,id2=0,id3=0,id4=0,id5=0 where no='$i';"
done > output < input

Or a version in awk which may be faster yet:

Code:
awk -v RS="\r\n" '$1 { printf("update table set id1=0, id2=0, id3=0,id4=0,id5=0 where no=\"%s\";", $1); }' filename > output

If you're not on Linux, use gawk or nawk.

---------- Post updated at 01:08 PM ---------- Previous update was at 01:08 PM ----------

Depending on what your database is, it's probably also possible to pipe that output directly into your database, so it can process it as it happens, instead of having to store it in a gigantic temp file in the meantime then start over from the beginning when done.
This User Gave Thanks to Corona688 For This Post:
# 3  
Old 12-08-2011
Very informative corona......... Thanks a ton for saving me from a dangerous scenario........

---------- Post updated at 04:10 AM ---------- Previous update was at 02:49 AM ----------

The awk statement is not working.......... Is it enough if I replace for statement with while statement??
# 4  
Old 12-08-2011
the while statement ought to work but may not be the fastest way to do it.

In what way is awk "not working"?
# 5  
Old 12-08-2011
Its giving an fatal "attempt by zero" error
# 6  
Old 12-08-2011
Attempt by zero?

Do you mean "attempted division by zero"?

Could you show what you typed, word for word, letter for letter, keystroke for keystroke?
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Programming

Not a repeated question (Perl Script Create Football Formation)

https://www.unix.com/programming/252468-perl-script-create-football-formation.html https://www.unix.com/members/43551.html, it is not repeated question. please read it before u block my question. Unblock it for me. Thanks:mad: (0 Replies)
Discussion started by: Tzeronone
0 Replies

2. Shell Programming and Scripting

Football formation

I'm trying to create a perl script that will automatically fit the player's name to the formation. For example, in my: DR.txt Arbeloa 200 Carvajal 190 Ramos 180 DC.txt Ramos 200 Pepe 190 Varane 180 DL.txt Marcelo 200 Coentrao 190 Arbeloa 180 formation.txt: DR DC DC DL DR (7 Replies)
Discussion started by: Tzeronone
7 Replies

3. Programming

Perl script to create football formation

I need help to create varieties of football formation. The available positions are: GK SW DR DC DL WBR DM WBL MR MC ML AMR AMC AML ST But the conditions are: a. the maximum number in 1 formation: GK is 1 SW is 1 (1 Reply)
Discussion started by: Tzeronone
1 Replies

4. Shell Programming and Scripting

How to get sql command with output in files?

I tried executing set of queries from shell script but not able to capture the input query in the log file,but the input query is not displayed along with output. Only the outputs are being captured in the log file. Is there a way to capture the input query along with the corresponding input?? (8 Replies)
Discussion started by: pallvi_mahajan
8 Replies

5. UNIX for Advanced & Expert Users

Data formation

I have a data like as follows, I need to format it as shown in as below. Request you to help me here ? I/P aa|3|1 aa|4|2 bb|3|1 bb|4|1 cc|3|26 cc|4|1 O/P aa|3|1|4|2 bb|3|1|4|1 cc|3|26|4|1 Thanks, Srikanth (5 Replies)
Discussion started by: srikanth38
5 Replies

6. Shell Programming and Scripting

Perl string formation from array

HI I ma using perl programming my perl is like this $InputFile = $ENV{UDE_TMP} . "/" ."cre_fmr_gen.temp_data_file_gen.dat"; @duplicates = `cat $InputFile | cut -d "|" -f 1,1 | sort | uniq -c | awk '{ if(\$1>1) {print \$2;}}'`; my $cusiplist ; foreach $cusip (@duplicates) {... (1 Reply)
Discussion started by: ptappeta
1 Replies

7. Shell Programming and Scripting

Help to merge multiple .sql files

Hello all, I have a shell script that uses multiple .sql files. These .sql files mainly contain Oracle SQL queries to pull fields from the database. I want to place all the contents of these .sql files in one .sql file and have some parameter sent based on which the respective block or query... (6 Replies)
Discussion started by: snvniranjanrao
6 Replies

8. Shell Programming and Scripting

How to retrieve all the linked script files/ctl files/sql files?

Hi I am going to migrate our datawarehouse system from HP Tru 64 Unix to the Red Hat Linux. Inside the box, it is running around 40 cron jobs; inside each cron job, it is calling other shell script files, and the shell script files may again call other shell script files or ctl files(for... (1 Reply)
Discussion started by: franksubramania
1 Replies

9. Shell Programming and Scripting

Need to extract .sql files

Hi, I am trying to extract all .sql files present in a particular directory and its sub directories. How can i do this using shell script or awk? any help would be earnestly appreciated. thanks. (1 Reply)
Discussion started by: sprinleo
1 Replies

10. Shell Programming and Scripting

date command op formation

Dear All I am facing below mention problem plz suggest me solution. Op of date command: > date Tue Jan 1 12:17:52 IST 2008 Now i want Jan 1 12: or Jan 1 12 op in some another variable. I had tried awk but if give me problem when there is date comes in 2 digit i.e. greater that 9.... (1 Reply)
Discussion started by: jaydeep_sadaria
1 Replies
Login or Register to Ask a Question