Visit Our UNIX and Linux User Community

Persisting message queues to disk

Thread Tools Search this Thread
Top Forums Programming Persisting message queues to disk
# 1  
Old 12-16-2009
Persisting message queues to disk


I have searched the forums and could not find a relavant thread discussing my use case, hence the new post.

Basically am trying to pass on work to dummy worker instances from controller which will pass on work to workers (client)
To make use of host capacity, am planning to serialize actual command structure ( the structure that contains the details of work, how to execute, which log file etc ) and dump it to NFS directory that is shared by hosts in which both worker and controller runs.

In a way, it could be explained as -
1) controller creates the structure serializes and dump it to one of the NFS directory as a file
2) worker waits for the file, reads the structure and deserialize it and start executing the work as specified in the structure

I actually don't like this approach as its not crash resistant, not scalable and have been thinking to use message queues (POSIX) to implement this asynchronous communication between worker and controller. But I don't know how to create a message queue that can be accessed across process in multiple hosts.
If that is possible, then its very good, but how?
If not, is there a way to dump message queue as a serialized structure to NFS directory and the worker can deserialize the structure and read that again.
I know, this also sounds messy as the very first point is simultanity in read and write to MQ is lost with this approach.

Also, if you could suggest alternative methods that would be awesome.

Other methods, I could think of is
- socket conn establishment and communicating via that - much of handling and house keeping
- use Amazon SQS, high available centralized queue - I need to pay and its a big setup for my small use case run in 4 hosts shared via 1 NFS mount.

Many thanks for suggestions in advance. Smilie
# 2  
Old 12-16-2009
Rather than reinvent the wheel, I suggest you use a message queueing protocol such as AMQP. Implementations include Redhat's MRG, RabbitMQ, Apache Qpid and OpenAMQ.
# 3  
Old 12-16-2009
hi fpmurphy,

thanks for the pointer.

Any idea, how easy and quick is to make use of the framework? My use case is really simple and straight forward, distribute work, get it done and collate the results.

Previous Thread | Next Thread
Test Your Knowledge in Computers #86
Difficulty: Easy
Windows NT was developed as a Unix-like operating system.
True or False?

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

Performance calculation for Message Queues

i have a program(C++ Code) that sends/receives information through queue's (Uses MQ) Is there any UNIX/LINUX tool that calculates the load and performance time for the same. If not how do i design the program that calculates the performance time. i know that time.h can be used but it gives... (2 Replies)
Discussion started by: vkca
2 Replies

2. Shell Programming and Scripting

Cleaning Message Queues

i have an application installed on AIX 5.3 and i have made a script that shutdown a proccesses that exceeded 10000kb of memory usage but i have a problem with cleaning the message queues of these proccesses after shutting them down. Is there any way to clean the message queues for this particular... (8 Replies)
Discussion started by: Portabello
8 Replies

3. UNIX for Dummies Questions & Answers

message queues

can any body provide a tutorial that explains the concept of message queues in UNIX in great detail (1 Reply)
Discussion started by: asalman.qazi
1 Replies

4. UNIX for Advanced & Expert Users

message queues

#include <sys/ipc.h> #include <sys/msg.h> int main() { int qid; int t; struct msgbuf mesg; qid=msgget(IPC_PRIVATE,IPC_CREAT); mesg.mtype=1L; mesg.mtext=1; t=msgsnd(qid,&mesg,1,0); printf("%d",t); } the program prints -1 as the result of msgsnd ,which means that msgsnd doesn't... (1 Reply)
Discussion started by: tolkki
1 Replies

5. Programming

message queues and multi-process

Hi, Am supposed to use message queues to send and receive messages between the processes. when i was working on that i realised that the message qid and the message queue related data should be maintained in a shared memory so that it can be accessed by all the processes. Could anybody refer... (10 Replies)
Discussion started by: rvan
10 Replies

6. UNIX for Advanced & Expert Users

UNIX Message Queues vs. Sockets

If I use sockets for IPC, and can easily distribute my applications. UNIX Message Queues are local to the processor. As I understand it, Message Queues still incur system call overhead, just like socket calls. What advantage does a UNIX Message Queue provide versus a TCP or UDP Socket,... (2 Replies)
Discussion started by: zen29sky
2 Replies

7. Linux

maximun number of message queues

how to check the maximun number of message queues in current linux enviornment? is there any command ? (4 Replies)
Discussion started by: princelinux
4 Replies

8. Solaris

rogue message queues solaris 9

We have message queues created from our ERP system to our tax system via an application api written by the ERP software vendor. Occasionally when a user does not gracefully exit the ERP application, the message queue hangs. After a few months, this becomes a problem as the queues are all used... (2 Replies)
Discussion started by: MizzGail
2 Replies

9. UNIX for Dummies Questions & Answers

message queues

let 3 processes a, b and c are sharing msgs using msg queues.process 'a' sending msg to 'c' and in turn 'c' send sthat msg to 'b'.if something happens to c how can 'a' and 'b' know that 'c' is not available?????? (2 Replies)
Discussion started by: sukaam
2 Replies

10. Programming

Message queues

Hi all, I've been trying for hours to figure out how to turn my 2-program (one to send and one to receive) "chat system" using message queues, into a single program where each concurrent component (entity) will both send and receive messages. PLEASE give me a hand with this, I'm starting to... (9 Replies)
Discussion started by: mgchato
9 Replies

Featured Tech Videos