epoll, sockets and threads


 
Thread Tools Search this Thread
Top Forums UNIX for Advanced & Expert Users epoll, sockets and threads
# 1  
Old 05-20-2009
epoll, sockets and threads

Hello Forum,

There is a transient bug in my code which I just can't catch. Could you help please?

The basic idea is this. I have a multithreaded server and two thread pools, the IO pool and Worker pool. The main server thread listens to incoming connections. When it gets one, it dispatches a thread from the IO pool to read the socket. An IO thread reads the socket. If there is data, it invokes a thread from the Worker pool. This thread then finishes off the processing including closing the socket. The code follows.

Before I show the code, let me tell you what's happening when I am testing it. I have a client program which has about 30 threads, each opening a separate socket connection with the server. Each sends a request which the server responds to. Sometimes I have all the threads finish in a zip. Sometimes both the client and the server hangs. And yet at other times, the client gets a connection refused message.

Now the code. I have omitted some of the nitty-gritties, including possibly variable declarations. I'll post them if it's required.

Code:
static void *server(void *arg)                  
struct sigaction sa;    
int i, num_fd, client_fd;    
struct sockaddr_in client_addr;    
 
socklen_t len = sizeof(client_addr);                
if (listen(s_sock_debugsvr, s_backlog) == -1) {            
         log("Debug Server: cannot listen");            
         pthread_exit(NULL);        
}       
sa.sa_handler = sigchld_handler; // reap all dead processes       
sigemptyset(&sa.sa_mask);        
sa.sa_flags = SA_RESTART;        
if (sigaction(SIGCHLD, &sa, NULL) == -1) {            
         log("Debug Server: Error in sigaction");            
         pthread_exit(NULL);        
}    
 
epfd = epoll_create(EPOLL_INIT_SIZE);    
if (epfd < 0) {        
         log("Exiting debug server worker thread...");        
         pthread_exit(NULL);    
}    
 
ev.events = EPOLLIN | EPOLLET;    
ev.data.fd = s_sock_debugsvr;    // add server socket to epoll    
epoll_ctl(epfd, EPOLL_CTL_ADD, s_sock_debugsvr, &ev);    
 
while (1) {        
         num_fd = epoll_wait(epfd, events, MAXEVENTS, 500);                
         for (i=0; i<num_fd; i++) {            
               if(events[i].data.fd == s_sock_debugsvr) {               
               /* New connection. Add it to list of epoll. Will allow notification when                   
                  data is received on this connection later. */                                
                      client_fd = accept(s_sock_debugsvr,(struct sockaddr *) &client_addr,&len);                
                      set_nonblocking(client_fd);                
                      ev.data.fd = client_fd;                
                      ev.events = EPOLLIN | EPOLLET;                
                      epoll_ctl(epfd, EPOLL_CTL_ADD, client_fd, &ev);                
                      printf("Added: %d\n", client_fd);           
               }            
               else if(events[i].events & EPOLLIN) {                
                      /* A request has come from a connected client. */                
                     client_fd = events[i].data.fd;               
                     /* Now dispatch the function to read from client socket */                
                    dispatch(io_thread_pool, read_socket, (void *)client_fd);           
               }       
        }    
}    
close(epfd);   
 return NULL;
}

Now the read_socket thread which reads from the sockets.
Code:
static voidread_socket(void *arg) {    
      struct debug_query *client_debug_query;    
      int num_bytes, client_fd;        
      client_fd = (int)arg;    
      num_bytes = recv(client_fd, request, MAX_BUF_SZ-1, 0);        
      if (num_bytes < 0) {        /* An error has occured */                     
           close_proper(client_fd);        /* Delete from epoll */
           io_thread_cleaner_fn((void *)client_debug_query);        
           return;    
      }    
      if (num_bytes == 0) {        /* Client has closed connection */        
           close_proper(client_fd);        
           io_thread_cleaner_fn((void *) client_debug_query);        
           return;   
       }       
        /* If I have come here, it means there is data fom client that I need to act on. */        
      dispatch_with_cleanup(worker_thread_pool, on_debugquery, (void*)client_debug_query, io_thread_cleaner_fn, (void*)client_debug_query);    }

Finally the worker thread function.
Code:
  static voidon_debugquery(void *arg){                  
            /* Do some processing */               
           ev.data.fd = client_fd;               
           epoll_ctl(epfd, EPOLL_CTL_DEL, client_fd, &ev);               
          close(client_fd);}

I use the following setup:

Intel Xeon Quad Core E5405(2.0Ghz).
Linux 2.6.18-92.el5PAE #1 SMP i686 i686 i386 GNU/Linux

I have posted this on the CentOS forum but without any response. I hope there will be someone here who will be able to offer a clue.

If you know of a more appropriate forum, please direct me there.

Thanks in advance,
Rij
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Programming

Event driven programming / epoll / typedef union / session data array

Sorry for the “word salad” subject, but I wanted to cast a wide net for help. I've created an IP (Internet Protocol) server which serves HTTP, SMTP, and FTP requests. As you probably know, they all require creating a socket, listening on it, accepting connections, and then having a short... (3 Replies)
Discussion started by: John S.
3 Replies

2. Red Hat

Sockets

hai guys, I'm doing a project in which one server communicates with several clients. How can i do it when i have different port numbers???:confused: (0 Replies)
Discussion started by: rajeshb6
0 Replies

3. Programming

Help with sockets in C

if i have a server which wants to connect to exactly 5 clients, does that mean i need 5 socket file descriptors and use listen(socket_fd,1); for each one or just do listen(socket_fd,5) also whats the second parameter number mean? what happens if i put 0 there? also if i am connected... (28 Replies)
Discussion started by: omega666
28 Replies

4. Programming

epoll problem with tcp connect()

I am using epoll to manage my network connections. At the client side, the idea is to send the message upon detecting the completion of connect(). I've set up the socket to be nonblocking, then after calling of connect(), I add the socket into the epoll and wait on the event to be detected from... (0 Replies)
Discussion started by: tuesday420
0 Replies

5. UNIX for Advanced & Expert Users

Threads and Threads Count ?

Hi all, How can I get the list of all Threads and the Total count of threads under a particular process ? Do suggest !! Awaiting for the replies !! Thanks Varun:b: (2 Replies)
Discussion started by: varungupta
2 Replies

6. Programming

epoll detecting client connections

I've got a program running using epoll to poll activity on a listening socket. I want to be able to output a message to the terminal when the socket is connected to from the client app. It appears that epoll doesn't throw any events on the connect but only on the socket activity. Is there any... (1 Reply)
Discussion started by: mstaylor
1 Replies

7. Programming

Wanted: References on Sockets and Threads for C

I'm looking at http://www.opengroup.org/pubs/online/7908799/xsh/pthread.h.html trying to understand mutexs and semaphores. Windows makes a distinction between the two. Is a mutex and semaphore different in unix land? Is there a tutorial on threading in unix somewhere? I'm also looking at... (4 Replies)
Discussion started by: siegfried
4 Replies

8. UNIX for Dummies Questions & Answers

sockets

how do i mointor how many sockets are opened from a particular foriegn address? (2 Replies)
Discussion started by: kirpond
2 Replies

9. Programming

sockets

Hai, How cani declare socket and collect the data in a string varialbe. Since i am new to this i am asking this. Can we connect multiple port. Thank you. (6 Replies)
Discussion started by: arunkumar_mca
6 Replies

10. Programming

epoll problem

I programmed simple tcp server using nonblocking sockets and epoll. But I am facing some problems. 1. I can recv from events.data.fd but I can't send any data over events.data.fd 2. When I make multiple connections at once (say 100) I get segmentation fault... What am I doing wrong? ... (1 Reply)
Discussion started by: Parahat Melayev
1 Replies
Login or Register to Ask a Question