Sponsored Content
Top Forums Programming libcurl multi interface problem Post 302434920 by clalfa on Monday 5th of July 2010 06:34:11 PM
Old 07-05-2010
libcurl multi interface problem

Hello, I'm trying to use libcurl multi interface to fetch several data in parallel. I would expect this to be faster than performing repeated fetches using the easy interface, but for some reason I can't obtain any speed up at all: using the multi interface actually turns out to be MUCH slower than launching several sequential "easy" requests.
This is the code that actually performs the requests:

Code:
  curlmh_ = curl_multi_init();
  int min_req = MIN(n_req_, urls_.size());
  CURL** curleh = new CURL*[min_req];
  
  // create min_req easy handles, to be added to a single multi handle
  for (int i=0; i<min_req; i++) {
    curleh[i] = curl_easy_init();
    curl_easy_setopt (curleh[i], CURLOPT_URL, urls_[i].c_str());
    curl_easy_setopt (curleh[i], CURLOPT_WRITEFUNCTION, fetcher_thread::write_data);
    curl_multi_add_handle (curlmh_, curleh[i]);
  }

  int active, curl_res, fd_res, max_fd;
  fd_set fds, fdsw, fdse;
  struct timeval timeout;
  timeout.tv_sec = 5;
  timeout.tv_usec = 0;

  do {
    do {
      curl_res = curl_multi_perform (curlmh_, &active);
    }
    while ( curl_res == CURLM_CALL_MULTI_PERFORM );

    FD_ZERO (&fds);  FD_ZERO (&fdsw);  FD_ZERO (&fdse);

    max_fd = 0;
    fd_res = curl_multi_fdset (curlmh_, &fds, &fdsw, &fdse, &max_fd);
    if ( max_fd == -1 ) {
      continue;
    }

    do {      
      fd_res = select (max_fd+1, &fds, &fdsw, &fdse, (struct timeval*)NULL);
    }
    while ( fd_res == -1  &&  errno == EINTR );      

    msleep(100);		// I'd like to do other useful stuff meanwhile...
  }
  while ( curl_res == CURLM_OK  &&  active > 0 );

  // operations finished, doing the cleanup
  for (int i=0; i<min_req; i++) {
    curl_multi_remove_handle (curlmh_, curleh[i]);
    curl_easy_cleanup (curleh[i]);
  }
  curl_multi_cleanup(curlmh_);

I am using libcurl version 7.20.0-r2 under Gentoo Linux.

Any help/suggestion/idea would be really appreciated.
Thank you
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

network interface problem

Hi expert, Need some help on network interface issue.. I have added 2 x NIC card onto the Ultra 2 system recently and configured as hme1 and hme2. I have unconfigured the onboard hme0 network interface and it was running fine till few days later, i keep recieving error messages showing hme0... (6 Replies)
Discussion started by: sc2005
6 Replies

2. AIX

Problem with a Network Interface

Hi every body, I have a Fiber Channel interface (fcs2) in AIX 5.2. This interface was fine & up but for some reason I could not return this interface UP again after I set it DOWN. When I tried to set this interface UP I encountered the following error: Method error... (7 Replies)
Discussion started by: aldowsary
7 Replies

3. UNIX for Dummies Questions & Answers

Multi Network card interface problem

My system info is show below:- #uname -a SunOS qfserver 5.8 Generic_117350-29 sun4u sparc SUNW,Sun-Blade-2500 and I have two network card as shown below:- #ifconfig -a lo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4> mtu 8232 index 1 inet 127.0.0.1 netmask ff000000 bge0:... (1 Reply)
Discussion started by: sarifudin
1 Replies

4. UNIX for Advanced & Expert Users

interface problem

hi all, i have problem with my box, until now i can't investigate the root cause of my issue at my box. here the problem. i have a box as a squid server just forward all request packet from one interface and receive the packet then forward to client at the same interface. but after 5 hours i... (1 Reply)
Discussion started by: tindasz
1 Replies

5. AIX

Multi Link Interface Runtime - where to download ?

Hello, I need "devices.common.IBM.ml 1.4.0.0 C F Multi Link Interface Runtime" to be installed on my machine. I need it for two SAN cards to work correctly. Where do I get it ? thanks Vilius (1 Reply)
Discussion started by: vilius
1 Replies

6. Debian

Problem with graphical interface

Hi, i have problems with an installation of Debian i386 505 in a pc. The hardware it is a mother soyo with a chipset via. When i install the xorg, the installation it`s succesfull, but when i type startx, the pc don`t respond. I am from Argentina and my English it isn`t good. Sorry. (0 Replies)
Discussion started by: Kritar
0 Replies

7. Solaris

Network interface problem

HI, genunix: NOTICE: ce0: xcvr addr:0x01 - link up 100 Mbps half duplex genunix: WARNING: ce0: fault detected external to device; service degraded genunix: WARNING: ce0: xcvr addr:0x01 - link down genunix: NOTICE: ce0: fault cleared external to device; service available genunix: ... (4 Replies)
Discussion started by: sunnybee
4 Replies

8. IP Networking

DHCP Server on Vxworks multi interface question

Hi, We currently operate a DHCP Server on Vxworks system. It seems that the server is functioning only over the boot interface. While trying to use it on a different interface the DHCP client messages seems to reach the interface but stay without response. From a... (1 Reply)
Discussion started by: zohara
1 Replies

9. Red Hat

libcurl.so.4 problem

libcurl.so.4: cannot open shared object file: No such file or directory ERROR: Loading network library (net.so) failed! Press Q to shut down the server! http://a1108.hizliresim.com/11/8/17/8183.jpg What must I do ? :wall: (14 Replies)
Discussion started by: Stark0010
14 Replies

10. IP Networking

Port based multi interface routing

Hello, I wanted to setup routing certain traffic (http/s) out via a second (faster) interface, like described in the following docs (may not post urls): linux-ip.net /html/adv-multi-internet.html thegeekstuff.com /2014/08/add-route-ip-command/ I already had this working years ago on... (0 Replies)
Discussion started by: hyphan
0 Replies
curl_multi_add_handle(3)					  libcurl Manual					  curl_multi_add_handle(3)

NAME
curl_multi_add_handle - add an easy handle to a multi session SYNOPSIS
#include <curl/curl.h> CURLMcode curl_multi_add_handle(CURLM *multi_handle, CURL *easy_handle); DESCRIPTION
Adds a standard easy handle to the multi stack. This function call will make this multi_handle control the specified easy_handle. Further- more, libcurl now initiates the connection associated with the specified easy_handle. When an easy handle has been added to a multi stack, you can not and you must not use curl_easy_perform(3) on that handle! If the easy handle is not set to use a shared (CURLOPT_SHARE) or global DNS cache (CURLOPT_DNS_USE_GLOBAL_CACHE), it will be made to use the DNS cache that is shared between all easy handles within the multi handle when curl_multi_add_handle(3) is called. The easy handle will remain added until you remove it again with curl_multi_remove_handle(3). You should remove the easy handle from the multi stack before you terminate first the easy handle and then the multi handle: 1 - curl_multi_remove_handle(3) 2 - curl_easy_cleanup(3) 3 - curl_multi_cleanup(3) RETURN VALUE
CURLMcode type, general libcurl multi interface error code. SEE ALSO
curl_multi_cleanup(3),curl_multi_init(3) libcurl 7.9.5 4 March 2002 curl_multi_add_handle(3)
All times are GMT -4. The time now is 04:29 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy