|
|
Sponsored Content
Top Forums
Programming
Regarding Multi-Threading
Post 78496 by mbb on Wednesday 20th of July 2005 05:54:40 AM
07-20-2005
Sorry to hear that you cannot stop the querying process. This will limit your options.
Insert cursors are a way for your Esql/C process to buffer the inserts before they are committed to the database. I cannont comment on how that will affect the performance of your system overall, you would have to measure that. But they are the fastest way for you to insert records into a table via your program. After your load has been completed I recommend you still run update statistics.
As far a performance goes, you may want to pre-process the file and peform your validations before you attempt an insert run. Any exception records can be excluded from the insert run.
Since you have to run a large load in the exisiting environment and it is a one off, my best suggestion would be to validate and then load the file in much smaller chunks. Say 10,000 records at a time.
Therefore split the file into multiple files of 10,000 record chunks.
Write a validation program to validate a single file.
Write a load program to load a single file.
If you dont want to manage this all manually then,
write another(!) program to manage the loading of the small 10K files. You could create a table to store the name of the file, total number of rows, number of rows validated, rows loaded. This program would limit the number of concurrent loading processes (e.g. 10 at a time) and continue until all files have completed. You could add extra features such a reloading a failed file or allowing processing during a specific time frame.
Overall the loading process may take some time to complete, but I think you are now in a situation where you should be trying to spread and manage the workload.
Anyway, given all that you have told me that is how I would approach this task and I do appreciate that what I have described is not a trivial piece of work. (Loading high volumes of data to a live database never is
)
MBB
Insert cursors are a way for your Esql/C process to buffer the inserts before they are committed to the database. I cannont comment on how that will affect the performance of your system overall, you would have to measure that. But they are the fastest way for you to insert records into a table via your program. After your load has been completed I recommend you still run update statistics.
As far a performance goes, you may want to pre-process the file and peform your validations before you attempt an insert run. Any exception records can be excluded from the insert run.
Since you have to run a large load in the exisiting environment and it is a one off, my best suggestion would be to validate and then load the file in much smaller chunks. Say 10,000 records at a time.
Therefore split the file into multiple files of 10,000 record chunks.
Write a validation program to validate a single file.
Write a load program to load a single file.
If you dont want to manage this all manually then,
write another(!) program to manage the loading of the small 10K files. You could create a table to store the name of the file, total number of rows, number of rows validated, rows loaded. This program would limit the number of concurrent loading processes (e.g. 10 at a time) and continue until all files have completed. You could add extra features such a reloading a failed file or allowing processing during a specific time frame.
Overall the loading process may take some time to complete, but I think you are now in a situation where you should be trying to spread and manage the workload.
Anyway, given all that you have told me that is how I would approach this task and I do appreciate that what I have described is not a trivial piece of work. (Loading high volumes of data to a live database never is
)MBB
|
|
10 More Discussions You Might Find Interesting
1. Programming
hi all,
can anyone tell me some good site for the mutithreading tutorials, its application, and some code examples.
-sushil (2 Replies)
Discussion started by: shushilmore
2 Replies
2. Programming
I've been doing some reading lately about threading (Posix threads) and I'm really curious about a couple things that I've read. I'm not sure if many people here have threading experience, but I thought it would be nice to be able to discuss some questions about it.
(For the record, I did... (1 Reply)
Discussion started by: DreamWarrior
1 Replies
3. Programming
Hi,
I have written a code which will run a set of process using
fork.
I want to know from You how can i start another job when one of my job in my loop is completed
My code is
#include<stdio.h>
#include<ctype.h>
main() {
int pid,cid;
ChildProcess();
... (1 Reply)
Discussion started by: sureshraju_ma
1 Replies
4. Programming
I am not sure if multi threading is the correct term, but here is what I am trying to do. I have a while loop that displays the number 1, pauses, displays the number 2, pauses , displays the number 3 ad infinitum. It just keeps counting.
While the screen displays the sequence of numbers counting... (4 Replies)
Discussion started by: enuenu
4 Replies
5. Programming
Sir,
Can I call same function in the start routines of different Threads.
I have created two different threads....and wanna call same function from both threads....is it possible???
Also can I fork inside a thread??? (1 Reply)
Discussion started by: arunchaudhary19
1 Replies
6. Programming
Hi, If we create 10 threads to invoke runQuery method at same time, Will queryProcessor will be overriden sometime or 10 different copies will be created?
We are not using any sunchronzation mechnism in runQuery(). so there is not gurantee on QueryProcessor class variables right OR each 10... (1 Reply)
Discussion started by: jramesh1
1 Replies
7. Programming
In this piece i implemented the gossip method. The first thread is invoked from inside the (msg is first sent from node -1 to 0 from main()) and the other threads are invoked from inside of the thread function itself. I used two mutexes and a condition variable to control the synchronization. ... (4 Replies)
Discussion started by: saman_glorious
4 Replies
8. UNIX for Dummies Questions & Answers
Hi, I am trying to get my head round Multi Threading and I have a few queries to try and clear up my confusion
Q1. Is multi threading a hardware / chip level concept, an OS level or an application level concept ? I am trying to work out where SMT architecture fits in.
Q2. What's the multi... (3 Replies)
Discussion started by: jimthompson
3 Replies
9. UNIX for Beginners Questions & Answers
Not just background process running ... but im looking if unix has any multi-threading concept like in Java, C# ... if not present, can you pls share the nearest feature in unix that is close to multi-threaded concept (3 Replies)
Discussion started by: i4ismail
3 Replies
10. UNIX for Beginners Questions & Answers
Hi,
Can we apply multi threading in Unix. I am using bash shell.
We have a generic script to load the data to table based on file input. For each file there is an individual table to load.
For each file found in directory I want to load the data in parallel to target table using
... (3 Replies)
Discussion started by: vedanta
3 Replies
LEARN ABOUT DEBIAN
oakdecode
oakdecode(1) General Commands Manual oakdecode(1) NAME
oakdecode - Decode an OAKT printer stream into human readable form. SYNOPSIS
oakdecode [options] <OAKT-file DESCRIPTION
oakdecode decodes an OAKT printer stream into human readable form. An OAKT printer stream is the printer langauge used by the HP Color LaserJet 1500 and other printers. COMMAND LINE OPTIONS
These are the options that can appear on the command line. -d basename Basename of .pbm file for saving decompressed planes. -r basename Basename of .jbg file for saving raw planes -i Suppress display of image records. -o Print file offsets. -D level Set Debug level [0]. EXAMPLES
Decode an OAKT file created by foo2oak. $ oakdecode < testpage.oak 0d(80) 1 OTHER 0c(64) Wed Nov 05 16:30:50 2003 a07d3 100005 32001e 0a(80) testpage.pdf 14(16) (no args) 28(16) Source=Tray1 29(80) PaperType=0 UNK8=2,0,0,0, blanks(63) 2a(32) Copies=1 UNK=0 2b(32) papercode=25 xwid=4648 ywid=9000 UNK=0 33(64) u0 u1 w h resx resy nBits x0 x0 2128 4300 600 600 x1 15(16) (no args) bih0 w h l0 bih5 dlen plen unk yOff P subP 3c(64) 00010000 2176 256 256 58030020 1050 1056 000 64 3 0 DL = 0, D = 0, P = 1, - = 0, XY = 2176 x 256 L0 = 256, MX = 32, MY = 0 Order = 3 ILEAVE SMID Options = 88 LRLTWO TPDON TPBON 1 stripes, 0 layers, 1 planes 3c(64) 00010000 2176 256 256 58030020 3668 3680 000 320 3 0 3c(64) 00010000 2176 256 256 58030020 1463 1472 000 640 3 0 3c(64) 00010000 2176 256 256 58030020 1975 1984 000 896 3 0 3c(64) 00010000 2176 224 224 58030020 2744 2752 000 1152 3 0 3c(64) 00010000 2176 256 256 58030020 988 992 000 1440 3 0 3c(64) 00010000 2176 256 256 58030020 2892 2896 000 1696 3 0 3c(64) 00010000 2176 256 256 58030020 3634 3648 000 1952 3 0 3c(64) 00010000 2176 256 256 58030020 3236 3248 000 2208 3 0 3c(64) 00010000 2176 256 256 58030020 2279 2288 000 2464 3 0 3c(64) 00010000 2176 256 256 58030020 3746 3760 000 2720 3 0 3c(64) 00010000 2176 200 200 58030020 2404 2416 000 2976 3 0 3c(64) 00010000 2176 256 256 58030020 3114 3120 000 3240 3 0 3c(64) 00010000 2176 96 96 58030020 1142 1152 000 3496 3 0 3c(64) 00010000 2176 256 256 58030020 2094 2112 000 3752 3 0 3c(64) 00010000 2176 256 256 58030020 1319 1328 000 4008 3 0 3c(64) 00010000 2176 36 36 58030020 208 224 000 4264 3 0 17(16) (no args) 18(16) UNK=0 0b(16) (no args) FILES
/usr/bin/oakdecode SEE ALSO
foo2oak-wrapper(1), foo2oak(1), jbg2pbm(1) AUTHOR
Rick Richardson <rick.richardson@comcast.net> http://foo2oak.rkkda.com/