libcurl multi interface problem


 
Thread Tools Search this Thread
Top Forums Programming libcurl multi interface problem
# 1  
Old 07-05-2010
libcurl multi interface problem

Hello, I'm trying to use libcurl multi interface to fetch several data in parallel. I would expect this to be faster than performing repeated fetches using the easy interface, but for some reason I can't obtain any speed up at all: using the multi interface actually turns out to be MUCH slower than launching several sequential "easy" requests.
This is the code that actually performs the requests:

Code:
  curlmh_ = curl_multi_init();
  int min_req = MIN(n_req_, urls_.size());
  CURL** curleh = new CURL*[min_req];
  
  // create min_req easy handles, to be added to a single multi handle
  for (int i=0; i<min_req; i++) {
    curleh[i] = curl_easy_init();
    curl_easy_setopt (curleh[i], CURLOPT_URL, urls_[i].c_str());
    curl_easy_setopt (curleh[i], CURLOPT_WRITEFUNCTION, fetcher_thread::write_data);
    curl_multi_add_handle (curlmh_, curleh[i]);
  }

  int active, curl_res, fd_res, max_fd;
  fd_set fds, fdsw, fdse;
  struct timeval timeout;
  timeout.tv_sec = 5;
  timeout.tv_usec = 0;

  do {
    do {
      curl_res = curl_multi_perform (curlmh_, &active);
    }
    while ( curl_res == CURLM_CALL_MULTI_PERFORM );

    FD_ZERO (&fds);  FD_ZERO (&fdsw);  FD_ZERO (&fdse);

    max_fd = 0;
    fd_res = curl_multi_fdset (curlmh_, &fds, &fdsw, &fdse, &max_fd);
    if ( max_fd == -1 ) {
      continue;
    }

    do {      
      fd_res = select (max_fd+1, &fds, &fdsw, &fdse, (struct timeval*)NULL);
    }
    while ( fd_res == -1  &&  errno == EINTR );      

    msleep(100);		// I'd like to do other useful stuff meanwhile...
  }
  while ( curl_res == CURLM_OK  &&  active > 0 );

  // operations finished, doing the cleanup
  for (int i=0; i<min_req; i++) {
    curl_multi_remove_handle (curlmh_, curleh[i]);
    curl_easy_cleanup (curleh[i]);
  }
  curl_multi_cleanup(curlmh_);

I am using libcurl version 7.20.0-r2 under Gentoo Linux.

Any help/suggestion/idea would be really appreciated.
Thank you
# 2  
Old 07-06-2010
What is msleep?
# 3  
Old 07-06-2010
msleep makes the process wait for a given amount of milliseconds.
I expect this not to slow down the requests, since this interface is by definition asynchronous. Anyway, removing that call does not help to avoid the problem, since repeated calls to easy_perform on single easy_handles are still faster...
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. IP Networking

Port based multi interface routing

Hello, I wanted to setup routing certain traffic (http/s) out via a second (faster) interface, like described in the following docs (may not post urls): linux-ip.net /html/adv-multi-internet.html thegeekstuff.com /2014/08/add-route-ip-command/ I already had this working years ago on... (0 Replies)
Discussion started by: hyphan
0 Replies

2. Red Hat

libcurl.so.4 problem

libcurl.so.4: cannot open shared object file: No such file or directory ERROR: Loading network library (net.so) failed! Press Q to shut down the server! http://a1108.hizliresim.com/11/8/17/8183.jpg What must I do ? :wall: (14 Replies)
Discussion started by: Stark0010
14 Replies

3. IP Networking

DHCP Server on Vxworks multi interface question

Hi, We currently operate a DHCP Server on Vxworks system. It seems that the server is functioning only over the boot interface. While trying to use it on a different interface the DHCP client messages seems to reach the interface but stay without response. From a... (1 Reply)
Discussion started by: zohara
1 Replies

4. Solaris

Network interface problem

HI, genunix: NOTICE: ce0: xcvr addr:0x01 - link up 100 Mbps half duplex genunix: WARNING: ce0: fault detected external to device; service degraded genunix: WARNING: ce0: xcvr addr:0x01 - link down genunix: NOTICE: ce0: fault cleared external to device; service available genunix: ... (4 Replies)
Discussion started by: sunnybee
4 Replies

5. Debian

Problem with graphical interface

Hi, i have problems with an installation of Debian i386 505 in a pc. The hardware it is a mother soyo with a chipset via. When i install the xorg, the installation it`s succesfull, but when i type startx, the pc don`t respond. I am from Argentina and my English it isn`t good. Sorry. (0 Replies)
Discussion started by: Kritar
0 Replies

6. AIX

Multi Link Interface Runtime - where to download ?

Hello, I need "devices.common.IBM.ml 1.4.0.0 C F Multi Link Interface Runtime" to be installed on my machine. I need it for two SAN cards to work correctly. Where do I get it ? thanks Vilius (1 Reply)
Discussion started by: vilius
1 Replies

7. UNIX for Advanced & Expert Users

interface problem

hi all, i have problem with my box, until now i can't investigate the root cause of my issue at my box. here the problem. i have a box as a squid server just forward all request packet from one interface and receive the packet then forward to client at the same interface. but after 5 hours i... (1 Reply)
Discussion started by: tindasz
1 Replies

8. UNIX for Dummies Questions & Answers

Multi Network card interface problem

My system info is show below:- #uname -a SunOS qfserver 5.8 Generic_117350-29 sun4u sparc SUNW,Sun-Blade-2500 and I have two network card as shown below:- #ifconfig -a lo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 1 inet 127.0.0.1 netmask ff000000 bge0:... (1 Reply)
Discussion started by: sarifudin
1 Replies

9. AIX

Problem with a Network Interface

Hi every body, I have a Fiber Channel interface (fcs2) in AIX 5.2. This interface was fine & up but for some reason I could not return this interface UP again after I set it DOWN. When I tried to set this interface UP I encountered the following error: Method error... (7 Replies)
Discussion started by: aldowsary
7 Replies

10. UNIX for Advanced & Expert Users

network interface problem

Hi expert, Need some help on network interface issue.. I have added 2 x NIC card onto the Ultra 2 system recently and configured as hme1 and hme2. I have unconfigured the onboard hme0 network interface and it was running fine till few days later, i keep recieving error messages showing hme0... (6 Replies)
Discussion started by: sc2005
6 Replies
Login or Register to Ask a Question