Sponsored Content
Full Discussion: Regarding Multi-Threading
Top Forums Programming Regarding Multi-Threading Post 78335 by mbb on Tuesday 19th of July 2005 09:24:28 AM
Old 07-19-2005
Sorry for the delay. Informix is right up my street, so I can be more specific.

Disabling the transaction logging would mean that the inserts are not logged in the logical logs. If something went wrong you could not roll back your transactions. Transaction logging is an overhead you could do without, but by not using it there would be a risk. If your system is being used during the load it will not be an option. To switch transaction logging, use ontape. Use ontape to make a backup of the database before you do anything. It might actually be quicker to restore from a level 0 rather than trying to fix a botched, half done, load with your volumes.

Your indicies are definitely slowing things down, but without complete control of the table, there is not much you could do about transaction logging or dropping the indicies. Consider, the more records you insert, the larger the indicies become and the overhead becomes greater. In terms of the optimiser, it will progressively become more and more expensive to insert a row into your table. Your initial calculation of 16 hours may become even worse as the load continues.

Is it possible to stop the querying process? Is there a time during the 24hr period when the table is not used and can be loaded to?

Since you have posted in this forum, and not the scripting forum, I will assume you are using ESQL/C.

You could write a process that would:

1. Drop the indicies.
2. Load some or all of the data to the table.
3. Re-create the indicies
4. Update statistics for the table

When you write the process make sure that the insert statement is prepared
e.g.
sprintf(sqlvar, "insert into mytable values (?, ?, ?, %ld, ?)", num_var);

$prepare p_ins for $lv_sql;

Then, later in program, whenever a row is required to be inserted:

$execute p_ins using $var1, $var2, $var3, $var4;

This will help your inserts go a little quicker. The fastest way to perform inserts programatically is with using insert cursors.

$declare c_ins cursor for p_ins;
$open c_ins;
$put c_ins using $var1, $var2, $var3, $var4;
$close c_ins;

Read up some more on this, but your real problem is getting rid of the indicies on the table. They really, really do slow things up. Also consider that if you have other demands on your Informix system during the load, it will have a detrimental effect on your load's performance. Maybe a quiet time would be beneficial. You can always run a program you have written from cron.

Generally the principle of multi-threading or using multiple processes is good, but not too many at once. From my own experience, there will come a point that running more load processes concurrently will slow things down rather than speed them up.

As a digression, which might yet apply to your loading process, have you considered the impact of the queries on a table which is 5+ million rows? Are your indicies all working? Do you have any sequential scans?

You may investigate the optimiser by using the sql command sqlexplain. Also the sysmaster database has tables which hold the costs on sql statements run against the database. You should certainly investigate high cost sql statements and how they may be affecting your overall system performance.

Well hopefully I have given you plenty to be getting on with!

MBB
 

10 More Discussions You Might Find Interesting

1. Programming

Multi threading using posix thread library

hi all, can anyone tell me some good site for the mutithreading tutorials, its application, and some code examples. -sushil (2 Replies)
Discussion started by: shushilmore
2 Replies

2. Programming

Multi-threading questions

I've been doing some reading lately about threading (Posix threads) and I'm really curious about a couple things that I've read. I'm not sure if many people here have threading experience, but I thought it would be nice to be able to discuss some questions about it. (For the record, I did... (1 Reply)
Discussion started by: DreamWarrior
1 Replies

3. Programming

Multi threading using fork

Hi, I have written a code which will run a set of process using fork. I want to know from You how can i start another job when one of my job in my loop is completed My code is #include<stdio.h> #include<ctype.h> main() { int pid,cid; ChildProcess(); ... (1 Reply)
Discussion started by: sureshraju_ma
1 Replies

4. Programming

Multi threading?

I am not sure if multi threading is the correct term, but here is what I am trying to do. I have a while loop that displays the number 1, pauses, displays the number 2, pauses , displays the number 3 ad infinitum. It just keeps counting. While the screen displays the sequence of numbers counting... (4 Replies)
Discussion started by: enuenu
4 Replies

5. Programming

Multi-threading-- calling same function through different threads

Sir, Can I call same function in the start routines of different Threads. I have created two different threads....and wanna call same function from both threads....is it possible??? Also can I fork inside a thread??? (1 Reply)
Discussion started by: arunchaudhary19
1 Replies

6. Programming

Multi-threading

Hi, If we create 10 threads to invoke runQuery method at same time, Will queryProcessor will be overriden sometime or 10 different copies will be created? We are not using any sunchronzation mechnism in runQuery(). so there is not gurantee on QueryProcessor class variables right OR each 10... (1 Reply)
Discussion started by: jramesh1
1 Replies

7. Programming

Multi-threading

In this piece i implemented the gossip method. The first thread is invoked from inside the (msg is first sent from node -1 to 0 from main()) and the other threads are invoked from inside of the thread function itself. I used two mutexes and a condition variable to control the synchronization. ... (4 Replies)
Discussion started by: saman_glorious
4 Replies

8. UNIX for Dummies Questions & Answers

Confusion over Multi Threading

Hi, I am trying to get my head round Multi Threading and I have a few queries to try and clear up my confusion Q1. Is multi threading a hardware / chip level concept, an OS level or an application level concept ? I am trying to work out where SMT architecture fits in. Q2. What's the multi... (3 Replies)
Discussion started by: jimthompson
3 Replies

9. UNIX for Beginners Questions & Answers

Does UNIX support multi-Threading ?

Not just background process running ... but im looking if unix has any multi-threading concept like in Java, C# ... if not present, can you pls share the nearest feature in unix that is close to multi-threaded concept (3 Replies)
Discussion started by: i4ismail
3 Replies

10. UNIX for Beginners Questions & Answers

Multi threading in UNIX

Hi, Can we apply multi threading in Unix. I am using bash shell. We have a generic script to load the data to table based on file input. For each file there is an individual table to load. For each file found in directory I want to load the data in parallel to target table using ... (3 Replies)
Discussion started by: vedanta
3 Replies
MYSQLSLAP(1)						       MySQL Database System						      MYSQLSLAP(1)

NAME
mysqlslap - load emulation client SYNOPSIS
mysqlslap [options] DESCRIPTION
mysqlslap is a diagnostic program designed to emulate client load for a MySQL server and to report the timing of each stage. It works as if multiple clients are accessing the server. Invoke mysqlslap like this: shell> mysqlslap [options] Some options such as --create or --query enable you to specify a string containing an SQL statement or a file containing statements. If you specify a file, by default it must contain one statement per line. (That is, the implicit statement delimiter is the newline character.) Use the --delimiter option to specify a different delimiter, which enables you to specify statements that span multiple lines or place multiple statements on a single line. You cannot include comments in a file; mysqlslap does not understand them. mysqlslap runs in three stages: 1. Create schema, table, and optionally any stored programs or data you want to using for the test. This stage uses a single client connection. 2. Run the load test. This stage can use many client connections. 3. Clean up (disconnect, drop table if specified). This stage uses a single client connection. Examples: Supply your own create and query SQL statements, with 50 clients querying and 200 selects for each: mysqlslap --delimiter=";" --create="CREATE TABLE a (b int);INSERT INTO a VALUES (23)" --query="SELECT * FROM a" --concurrency=50 --iterations=200 Let mysqlslap build the query SQL statement with a table of two INT columns and three VARCHAR columns. Use five clients querying 20 times each. Do not create the table or insert the data (that is, use the previous test's schema and data): mysqlslap --concurrency=5 --iterations=20 --number-int-cols=2 --number-char-cols=3 --auto-generate-sql Tell the program to load the create, insert, and query SQL statements from the specified files, where the create.sql file has multiple table creation statements delimited by ';' and multiple insert statements delimited by ';'. The --query file will have multiple queries delimited by ';'. Run all the load statements, then run all the queries in the query file with five clients (five times each): mysqlslap --concurrency=5 --iterations=5 --query=query.sql --create=create.sql --delimiter=";" mysqlslap supports the options in the following list. It also reads option files and supports the options for processing them described at Section 4.2.3.2.1, "Command-Line Options that Affect Option-File Handling". o --help, -? Display a help message and exit. o --auto-generate-sql, -a Generate SQL statements automatically when they are not supplied in files or via command options. o --auto-generate-sql-add-autoincrement Add an AUTO_INCREMENT column to automatically generated tables. o --auto-generate-sql-execute-number=N Specify how many queries to generate automatically. o --auto-generate-sql-guid-primary Add a GUID-based primary key to automatically generated tables. o --auto-generate-sql-load-type=type Specify the test load type. The allowable values are read (scan tables), write (insert into tables), key (read primary keys), update (update primary keys), or mixed (half inserts, half scanning selects). The default is mixed. o --auto-generate-sql-secondary-indexes=N Specify how many secondary indexes to add to automatically generated tables. By default, none are added. o --auto-generate-sql-select-columns=str The string to use for the select columns used in automatic tests. You can use this to determine the effect on performance of selecting or excluding particular columns. This option was added in MySQL 6.0.3. o --auto-generate-sql-unique-query-number=N How many different queries to generate for automatic tests. For example, if you run a key test that performs 1000 selects, you can use this option with a value of 1000 to run 1000 unique queries, or with a value of 50 to perform 50 different selects. The default is 10. o --auto-generate-sql-unique-write-number=N How many different queries to generate for --auto-generate-sql-write-number. The default is 10. o --auto-generate-sql-write-number=N How many row inserts to perform on each thread. The default is 100. o --burnin Run the full test case in an infinite loop. This option was added in MySQL 6.0.3. o --commit=N How many statements to execute before committing. The default is 0 (no commits are done). o --compress, -C Compress all information sent between the client and the server if both support compression. o --concurrency=N, -c N The number of clients to simulate when issuing the SELECT statement. o --create=value The file or string containing the statement to use for creating the table. o --create-schema=value The schema in which to run the tests. o --csv[=file_name] Generate output in comma-separated values format. The output goes to the named file, or to the standard output if no file is given. o --debug[=debug_options], -# [debug_options] Write a debugging log. A typical debug_options string is 'd:t:o,file_name'. The default is 'd:t:o,/tmp/mysqlslap.trace'. o --debug-check Print some debugging information when the program exits. o --debug-info, -T Print debugging information and memory and CPU usage statistics when the program exits. o --delimiter=str, -F str The delimiter to use in SQL statements supplied in files or via command options. o --delayed-start=N The maximum delay in microseconds. Startup of each thread is delayed by a random number of microseconds up to this maximum. The default is 0. This option was added in MySQL 6.0.3. o --detach=N Detach (close and reopen) each connection after each N statements. The default is 0 (connections are not detached). o --engine=engine_name, -e engine_name The storage engine to use for creating tables. o --host=host_name, -h host_name Connect to the MySQL server on the given host. o --ignore-sql-errors Ignore SQL errors during the test run. By default, errors cause mysqlslap to exit. This option was added in MySQL 6.0.4. o --iterations=N, -i N The number of times to run the tests. o --label=str The label to use in printed and CSV output. This option was added in MySQL 6.0.3. o --number-blob-cols=str, The number of BLOB columns to use if --auto-generate-sql is specified. --number-blob-cols=3:1024/2048 would give you 3 BLOB columns with a random size between 1024 and 2048. This option was added in MySQL 6.0.3. o --number-char-cols=N, -x N The number of VARCHAR columns to use if --auto-generate-sql is specified. o --number-int-cols=N, -y N The number of INT columns to use if --auto-generate-sql is specified. o --number-of-queries=N Limit each client to approximately this number of queries. o --only-print Do not connect to databases. mysqlslap only prints what it would have done. o --password[=password], -p[password] The password to use when connecting to the server. If you use the short option form (-p), you cannot have a space between the option and the password. If you omit the password value following the --password or -p option on the command line, you are prompted for one. Specifying a password on the command line should be considered insecure. See Section 5.5.6.2, "End-User Guidelines for Password Security". o --pipe, -W On Windows, connect to the server via a named pipe. This option applies only for connections to a local server, and only if the server supports named-pipe connections. o --port=port_num, -P port_num The TCP/IP port number to use for the connection. o --post-query=value The file or string containing the statement to execute after the tests have completed. This execution is not counted for timing purposes. o --shared-memory-base-name=name On Windows, the shared-memory name to use, for connections made via shared memory to a local server. This option applies only if the server supports shared-memory connections. o --post-system=str The string to execute via system() after the tests have completed. This execution is not counted for timing purposes. o --pre-query=value The file or string containing the statement to execute before running the tests. This execution is not counted for timing purposes. This option was added in MySQL 5.1.18. o --pre-system=str The string to execute via system() before running the tests. This execution is not counted for timing purposes. o --preserve-schema Preserve the schema from the mysqlslap run. The --auto-generate-sql and --create options disable this option. This option was removed in MySQL 6.0.5. o --protocol={TCP|SOCKET|PIPE|MEMORY} The connection protocol to use for connecting to the server. It is useful when the other connection parameters normally would cause a protocol to be used other than the one you want. For details on the allowable values, see Section 4.2.2, "Connecting to the MySQL Server". o --query=value, -q value The file or string containing the SELECT statement to use for retrieving data. o --set-random-seed=value, The seed value for the randomizer. o --silent, -s Silent mode. No output. o --slave Follow master locks for other mysqlslap clients. Use this option if you are trying to synchronize around one master server with --lock-directory plus NFS. o --socket=path, -S path For connections to localhost, the Unix socket file to use, or, on Windows, the name of the named pipe to use. o --ssl* Options that begin with --ssl specify whether to connect to the server via SSL and indicate where to find SSL keys and certificates. See Section 5.5.7.3, "SSL Command Options". o --timer-length=N The duration in seconds to run each test. Tests that run longer are terminated. This option was added in MySQL 6.0.3. o --user=user_name, -u user_name The MySQL user name to use when connecting to the server. o --verbose, -v Verbose mode. Print more information about what the program does. This option can be used multiple times to increase the amount of information. o --version, -V Display version information and exit. COPYRIGHT
Copyright 2007-2008 MySQL AB, 2009 Sun Microsystems, Inc. This documentation is free software; you can redistribute it and/or modify it only under the terms of the GNU General Public License as published by the Free Software Foundation; version 2 of the License. This documentation is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with the program; if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA or see http://www.gnu.org/licenses/. SEE ALSO
For more information, please refer to the MySQL Reference Manual, which may already be installed locally and which is also available online at http://dev.mysql.com/doc/. AUTHOR
Sun Microsystems, Inc. (http://www.mysql.com/). MySQL 6.0 05/23/2009 MYSQLSLAP(1)
All times are GMT -4. The time now is 02:47 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy