<< Threading inside multiprocessing using queues >>


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting << Threading inside multiprocessing using queues >>
# 1  
Old 07-02-2018
<< Threading inside multiprocessing using queues >>

Hi All,

I am trying to achieve threading inside each process of multiprocessing. I have 2 queues one for multiprocess (process) & another inside each process. when i execute it got hung after below output. My goal here is to go through p_source queue & for each process picks up all t_source queue (parallel) so i have used thread inside process.

Pls correct what i have missed here. Thanks !

Code:
#!/usr/bin/env python2.7
import multiprocessing
from threading import Thread
from Queue import Queue

def do_stuff(q):
    print ("its do_stuff fn - {}".format(q.get()))

def start_my_work(i,q):
    print ("Start work for {}".format(q))

    t_source = ['My','Name','Is','John','I','Work','In','IT']

    qu = Queue()

    num_threads = 2

    for i in range(num_threads):
        worker = Thread(target=do_stuff, args=(qu,))
        worker.setDaemon(True)
        worker.start()

    for item in t_source:
        qu.put(item)

    qu.join()


q = multiprocessing.Queue()
jobs = []

p_source = ['hi','there','how','are','you','doing']

for item in p_source:
  q.put(item)

for i in range(3): # 3 multiprocess
    p = multiprocessing.Process(target=start_my_work, args=(i,q))
    jobs.append(p)
    p.start()

for p in jobs: #join on all processes ...
    p.join()

print "List processing complete."


Code:
Start work for <multiprocessing.queues.Queue object at 0x7f9fdfc1bc90>
its do_stuff fn - My
 its do_stuff fn - Name
Start work for <multiprocessing.queues.Queue object at 0x7f9fdfc1bc90>
its do_stuff fn - My
Start work for <multiprocessing.queues.Queue object at 0x7f9fdfc1bc90>
 its do_stuff fn - Name
its do_stuff fn - My
 its do_stuff fn - Name

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Multiprocessing in Python

Hi there, I have a code that can take in any function with two arguements and do processing. However, I would like to implement a feature whether it can limit a number of process running concurrently so as not take up too much resources. I have tried researching for pool.map however I am unable... (1 Reply)
Discussion started by: alvinoo
1 Replies

2. Programming

Multiprocessing multipointers

I have a complex problem..... I have to search files on directory "text files" then search on all of them for a word or sentence....the user inter my problem is,,,, if I want to create a child for each file...and point a file by pointer to search...and I don't know how much files i have in... (2 Replies)
Discussion started by: fwrlfo
2 Replies

3. UNIX for Dummies Questions & Answers

Multiprocessing Help

Hello all, I recently wrote a simple script for the analysis jobs I do at work. I have to run multiple files through 5 different stages of an analysis program, the script simply runs all of the files through each stage automatically. My question is this: The computer I'm using has 12 cores, each... (8 Replies)
Discussion started by: Tyler_92
8 Replies

4. Shell Programming and Scripting

Threading in unix

Hii all, Is there any way to use thread scripting in unix.?? i.e. running multiple command at same time ? Thankss (3 Replies)
Discussion started by: yashwantkumar
3 Replies

5. Programming

threading problem

#include <stdio.h> #include <stdlib.h> #include <pthread.h> void *print_message_function( void *ptr ); main() { pthread_t thread1, thread2; char *message1 = "Thread 1"; char *message2 = "Thread 2"; int iret1, iret2; /* Create independent threads each of... (3 Replies)
Discussion started by: vishy_85
3 Replies

6. Programming

Multi threading?

I am not sure if multi threading is the correct term, but here is what I am trying to do. I have a while loop that displays the number 1, pauses, displays the number 2, pauses , displays the number 3 ad infinitum. It just keeps counting. While the screen displays the sequence of numbers counting... (4 Replies)
Discussion started by: enuenu
4 Replies

7. UNIX for Dummies Questions & Answers

Threading...

Sorry, i may be posting the wrong question in this forum... but just curious, suppose we have a set of pre-emptively scheduled processes. A process, B, is multi-threaded with user-level thread scheduling. If one of the threads has a long calculation before making a thread library call, does this... (2 Replies)
Discussion started by: ianlow
2 Replies

8. UNIX for Dummies Questions & Answers

Multiprocessing under Linux

I'm writing C programs to be executed on a multi-processor UNIX (GNU/Linux, kernel 2.6.11) Do I need to add a special kind of code to somewhere or run a special utility to execute the program file to be executed by all processors? Or is it handled automatically by kernel? (1 Reply)
Discussion started by: rayne
1 Replies

9. Programming

Regarding Multi-Threading

Hi All, Here's my question I have a 385 MB file containing 5,000,000 records. I need to read from the file and load into a table. Initially i thought of doing it in a single thread (execution of a single program) but when calculated accounted 16 hours of time on a standard benchmark. Hence... (5 Replies)
Discussion started by: matrixmadhan
5 Replies

10. UNIX for Dummies Questions & Answers

About threading

hai is unix a multithreading environment or multitasking environment? which one exist in UNIX? (1 Reply)
Discussion started by: rajashekaran
1 Replies
Login or Register to Ask a Question