Multithreading program


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Multithreading program
# 1  
Old 10-01-2009
Multithreading program

Hi

I need to insert 1million records into MySQL database, but it is taking lot of time as there is no bulk insert support.

I want to spawn 10 processes which will insert 100k records each parallely.

Can somebody help me with a example program to execute this task through shell scripting.
# 2  
Old 10-01-2009
You mean bulk insert like in LOAD DATA? Have you tried using transactions, which will only commit to storage (aka "slow you down") when you're finished?
# 3  
Old 10-01-2009
xargs

HINT:
use xargs -n <XXX> -P <XXX> for multithreading
# 4  
Old 10-01-2009
Quote:
Originally Posted by inotech
HINT:
use xargs -n <XXX> -P <XXX> for multithreading
Thanks for the reply but I didnt understand how xargs can be helpful in my case.
Can you please explain it in my case, how it can be used with an example?

Thanks.
# 5  
Old 10-01-2009
A:

echo -e {1..100}"\n" | xargs -n 1 -P 10 script.sh

in script.sh:
<insert file to db> $1.ext ...

split 1 000 000 records to files with names 1 to 100 (*.ext)

**one of the many ways
# 6  
Old 10-01-2009
1. Multithreading means "you must be stupid to do this in shell"
2. Loading 1 huge set of data at once is in general faster than loading dzylion of microsets in parallel
3. In Oracle you have SQLLoader and it has mode called "direct" - check if you cannot use similar thing in MySQL (I have no experience in MySQL)
4. If there is no option of direct loading of the data you might want to use bulk load methods
5. Loading in multiple threads means that you most probably are going to create dzylion of connections... if creation and closing of a single connection takes 1 second and inserting 1 row of data takes 0.0000000001 of a second... imagine how much slower would it work if you create multiple threads
6. Transaction is your friend - until you commit your application (executable compiled application) should be able to insert rows with only a minor performance decrease - if this is really really required by you then you might assume that there is a probable chance that you might need to create an application that loads row-by-row and commits the changes afterward.... really no bulk methods? I was thinking that MySQL is a developed database but it is not if there are no bulk operations.
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. Programming

Help with multithreading

I take this question of the The Linux Programming Interface: A Linux and Unix System Programming page 652 exercise 30.1 I want someone to explain the under line statement because it sounds complex to me couldn't understand anything 30-1 Modify the program (thread_incr.c) so that each loop in... (3 Replies)
Discussion started by: fwrlfo
3 Replies

2. What is on Your Mind?

Alarm interrupt and multithreading

Hi Friends any know how became a friend in this Android Programming Language (0 Replies)
Discussion started by: ljarun
0 Replies

3. Programming

Multithreading in reading file

Dear all, I am having a huge XML file, as below structure <EMPLOYEE> <RECORD id =aaa> <Salary>99999</Salary> <section>ssss</section> </RECORD> <RECORD id =bbb> <Salary>77777</Salary> <section>ssss</section> </RECORD> </EMPLOYEE> This is a 50 GB file I want to read this file in... (9 Replies)
Discussion started by: arunkumar_mca
9 Replies

4. Programming

how to do udp broadcast with multithreading

hello to all i want to use multithreading to my UDP broadcast server client program. will anyone help me by proving C code. i am working in fedora. also my requirement is POSIX compliance.please help me..... (6 Replies)
Discussion started by: moti12
6 Replies

5. IP Networking

how to do udp broadcast with multithreading

hello to all i want to use multithreading to my UDP broadcast server client program. will anyone help me by proving C code. i am working in fedora. also my requirement is POSIX compliance.please help me..... (0 Replies)
Discussion started by: moti12
0 Replies

6. Programming

MultiThreading using Pthreads

Situation: i have multiple pthread_create calls like this: pthread_create(...., ThreadFunc1,.....); pthread_create(...., ThreadFunc2,.....); . . which i am using to create multiple threads.All the "ThreadFunc<i>" functions are actually calling same function "Receive" of a class using same... (3 Replies)
Discussion started by: Sastra
3 Replies

7. UNIX for Advanced & Expert Users

multithreading in UNIX

Hi, Can you please give me a suitable reference to learn multithreading programming in C in UNIX? Thanks (3 Replies)
Discussion started by: naan
3 Replies

8. Programming

multithreading on OSX

Hi all, I have a query about multithreading. What I would like to do is, at the start of my main update() function, start a couple of threads in parallel, once they are all complete carry on with my main update function. void update() { thread1->update(); // fluid solver ... (3 Replies)
Discussion started by: memoid
3 Replies

9. Programming

Multithreading in Pro*C

:confused: Hi! I have created a Multhreaded Application in Pro*C (using pthreads) with about 5 Threads running simultaneously. The Application is basically to Update a Centralized Table in Oracle, which updates different rows in the Table (Each Thread updates different rows!). The... (16 Replies)
Discussion started by: shaik786
16 Replies
Login or Register to Ask a Question