Sponsored Content
Full Discussion: Regarding Multi-Threading
Top Forums Programming Regarding Multi-Threading Post 78496 by mbb on Wednesday 20th of July 2005 05:54:40 AM
Old 07-20-2005
Sorry to hear that you cannot stop the querying process. This will limit your options.

Insert cursors are a way for your Esql/C process to buffer the inserts before they are committed to the database. I cannont comment on how that will affect the performance of your system overall, you would have to measure that. But they are the fastest way for you to insert records into a table via your program. After your load has been completed I recommend you still run update statistics.

As far a performance goes, you may want to pre-process the file and peform your validations before you attempt an insert run. Any exception records can be excluded from the insert run.

Since you have to run a large load in the exisiting environment and it is a one off, my best suggestion would be to validate and then load the file in much smaller chunks. Say 10,000 records at a time.

Therefore split the file into multiple files of 10,000 record chunks.
Write a validation program to validate a single file.
Write a load program to load a single file.

If you dont want to manage this all manually then,

write another(!) program to manage the loading of the small 10K files. You could create a table to store the name of the file, total number of rows, number of rows validated, rows loaded. This program would limit the number of concurrent loading processes (e.g. 10 at a time) and continue until all files have completed. You could add extra features such a reloading a failed file or allowing processing during a specific time frame.

Overall the loading process may take some time to complete, but I think you are now in a situation where you should be trying to spread and manage the workload.

Anyway, given all that you have told me that is how I would approach this task and I do appreciate that what I have described is not a trivial piece of work. (Loading high volumes of data to a live database never is Smilie )

MBB
 

10 More Discussions You Might Find Interesting

1. Programming

Multi threading using posix thread library

hi all, can anyone tell me some good site for the mutithreading tutorials, its application, and some code examples. -sushil (2 Replies)
Discussion started by: shushilmore
2 Replies

2. Programming

Multi-threading questions

I've been doing some reading lately about threading (Posix threads) and I'm really curious about a couple things that I've read. I'm not sure if many people here have threading experience, but I thought it would be nice to be able to discuss some questions about it. (For the record, I did... (1 Reply)
Discussion started by: DreamWarrior
1 Replies

3. Programming

Multi threading using fork

Hi, I have written a code which will run a set of process using fork. I want to know from You how can i start another job when one of my job in my loop is completed My code is #include<stdio.h> #include<ctype.h> main() { int pid,cid; ChildProcess(); ... (1 Reply)
Discussion started by: sureshraju_ma
1 Replies

4. Programming

Multi threading?

I am not sure if multi threading is the correct term, but here is what I am trying to do. I have a while loop that displays the number 1, pauses, displays the number 2, pauses , displays the number 3 ad infinitum. It just keeps counting. While the screen displays the sequence of numbers counting... (4 Replies)
Discussion started by: enuenu
4 Replies

5. Programming

Multi-threading-- calling same function through different threads

Sir, Can I call same function in the start routines of different Threads. I have created two different threads....and wanna call same function from both threads....is it possible??? Also can I fork inside a thread??? (1 Reply)
Discussion started by: arunchaudhary19
1 Replies

6. Programming

Multi-threading

Hi, If we create 10 threads to invoke runQuery method at same time, Will queryProcessor will be overriden sometime or 10 different copies will be created? We are not using any sunchronzation mechnism in runQuery(). so there is not gurantee on QueryProcessor class variables right OR each 10... (1 Reply)
Discussion started by: jramesh1
1 Replies

7. Programming

Multi-threading

In this piece i implemented the gossip method. The first thread is invoked from inside the (msg is first sent from node -1 to 0 from main()) and the other threads are invoked from inside of the thread function itself. I used two mutexes and a condition variable to control the synchronization. ... (4 Replies)
Discussion started by: saman_glorious
4 Replies

8. UNIX for Dummies Questions & Answers

Confusion over Multi Threading

Hi, I am trying to get my head round Multi Threading and I have a few queries to try and clear up my confusion Q1. Is multi threading a hardware / chip level concept, an OS level or an application level concept ? I am trying to work out where SMT architecture fits in. Q2. What's the multi... (3 Replies)
Discussion started by: jimthompson
3 Replies

9. UNIX for Beginners Questions & Answers

Does UNIX support multi-Threading ?

Not just background process running ... but im looking if unix has any multi-threading concept like in Java, C# ... if not present, can you pls share the nearest feature in unix that is close to multi-threaded concept (3 Replies)
Discussion started by: i4ismail
3 Replies

10. UNIX for Beginners Questions & Answers

Multi threading in UNIX

Hi, Can we apply multi threading in Unix. I am using bash shell. We have a generic script to load the data to table based on file input. For each file there is an individual table to load. For each file found in directory I want to load the data in parallel to target table using ... (3 Replies)
Discussion started by: vedanta
3 Replies
funtbl(1)							SAORD Documentation							 funtbl(1)

NAME
funtbl - extract a table from Funtools ASCII output SYNOPSIS
funtable [-c cols] [-h] [-n table] [-p prog] [-s sep] <iname> DESCRIPTION
[NB: This program has been deprecated in favor of the ASCII text processing support in funtools. You can now perform fundisp on funtools ASCII output files (specifying the table using bracket notation) to extract tables and columns.] The funtbl script extracts a specified table (without the header and comments) from a funtools ASCII output file and writes the result to the standard output. The first non-switch argument is the ASCII input file name (i.e. the saved output from funcnts, fundisp, funhist, etc.). If no filename is specified, stdin is read. The -n switch specifies which table (starting from 1) to extract. The default is to extract the first table. The -c switch is a space-delimited list of column numbers to output, e.g. -c "1 3 5" will extract the first three odd-numbered columns. The default is to extract all columns. The -s switch specifies the separator string to put between columns. The default is a single space. The -h switch specifies that column names should be added in a header line before the data is output. With- out the switch, no header is prepended. The -p program switch allows you to specify an awk-like program to run instead of the default (which is host-specific and is determined at build time). The -T switch will output the data in rdb format (i.e., with a 2-row header of column names and dashes, and with data columns separated by tabs). The -help switch will print out a message describing program usage. For example, consider the output from the following funcnts command: [sh] funcnts -sr snr.ev "ann 512 512 0 9 n=3" # source # data file: /proj/rd/data/snr.ev # arcsec/pixel: 8 # background # constant value: 0.000000 # column units # area: arcsec**2 # surf_bri: cnts/arcsec**2 # surf_err: cnts/arcsec**2 # summed background-subtracted results upto net_counts error background berror area surf_bri surf_err ---- ------------ --------- ------------ --------- --------- --------- --------- 1 147.000 12.124 0.000 0.000 1600.00 0.092 0.008 2 625.000 25.000 0.000 0.000 6976.00 0.090 0.004 3 1442.000 37.974 0.000 0.000 15936.00 0.090 0.002 # background-subtracted results reg net_counts error background berror area surf_bri surf_err ---- ------------ --------- ------------ --------- --------- --------- --------- 1 147.000 12.124 0.000 0.000 1600.00 0.092 0.008 2 478.000 21.863 0.000 0.000 5376.00 0.089 0.004 3 817.000 28.583 0.000 0.000 8960.00 0.091 0.003 # the following source and background components were used: source_region(s) ---------------- ann 512 512 0 9 n=3 reg counts pixels sumcnts sumpix ---- ------------ --------- ------------ --------- 1 147.000 25 147.000 25 2 478.000 84 625.000 109 3 817.000 140 1442.000 249 There are four tables in this output. To extract the last one, you can execute: [sh] funcnts -s snr.ev "ann 512 512 0 9 n=3" | funtbl -n 4 1 147.000 25 147.000 25 2 478.000 84 625.000 109 3 817.000 140 1442.000 249 Note that the output has been re-formatted so that only a single space separates each column, with no extraneous header or comment informa- tion. To extract only columns 1,2, and 4 from the last example (but with a header prepended and tabs between columns), you can execute: [sh] funcnts -s snr.ev "ann 512 512 0 9 n=3" | funtbl -c "1 2 4" -h -n 4 -s " " #reg counts sumcnts 1 147.000 147.000 2 478.000 625.000 3 817.000 1442.000 Of course, if the output has previously been saved in a file named foo.out, the same result can be obtained by executing: [sh] funtbl -c "1 2 4" -h -n 4 -s " " foo.out #reg counts sumcnts 1 147.000 147.000 2 478.000 625.000 3 817.000 1442.000 SEE ALSO
See funtools(7) for a list of Funtools help pages version 1.4.2 January 2, 2008 funtbl(1)
All times are GMT -4. The time now is 03:28 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy