06-02-2010
Using cURL to save online search results
Hi, I'm attacking this from ignorance because I am not sure how to even ask the question.
Here is the mission:
I have a list of about 4,000 telephone numbers for past customers. I need to determine how many of these customers are still in business.
Obviously, I could call all the numbers.
But, I could also query whitepages.com's reverse lookup to see if the listing still shows up and eliminate those that don't show up from the list.
I have found out the correct URL to feed whitepages.com
It is
Free People Search | WhitePages
Dropping this into a browser address bar works every time.
I need to automate feeding URLs to whitepages.com and save the results for future analysis.
Using curl -O
Free People Search | WhitePages downloads nothing since there is no static page at
Free People Search | WhitePages.
How can I get curl to "wait" for the results of the query and save that?
Also, if anyone knows of another way to do this, please let me know. In the dark here - the 'services' that do this want $1 a lookup for bulk submission, and offer way more info than I need - which is basically "Is this number still working?"
Thanks!
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Hi All,
I want to edit a file using shell script..For ex...a file called /etc/passwd..here I am searching for "ftp" if it is there just change it to "tftp" without using any temporary file. (3 Replies)
Discussion started by: Vichu
3 Replies
2. Shell Programming and Scripting
hello hackers.
i have a curl process running as cgi directly pushing stdout to the client. but i want to additionally save that stream to a file at the same time.
any directions madly welcome.
thanks in advance (3 Replies)
Discussion started by: scarfake
3 Replies
3. Shell Programming and Scripting
Hi
I want to read a file line by line and search for a particular string in each line(say for example string containing @ )and save that string into a variable.
Can someone suggest me the way to implement it.I am using K- shell
Thanks
Ishita (5 Replies)
Discussion started by: Ishita
5 Replies
4. Shell Programming and Scripting
hi there !
i have exactly the same problem like this guy here
https://www.unix.com/shell-programming-scripting/127668-getting-curl-output-verbose-file.html
i am not able to save the curl verbose output..
the sollution in this thread (redirecting stderr to a file) does not work for me.... (0 Replies)
Discussion started by: crabmeat
0 Replies
5. Shell Programming and Scripting
Can any one please help, the code works...I want the output of $result to be saved in an output.txt file which is lcoated in c:\\temp\\output.txt.
$filepath="C:\\temp\\ip.txt";
open (HOSTLIST,"$filepath");
@hosts=(<HOSTLIST>);
foreach $host(@hosts)
{
$results = `nslookup... (1 Reply)
Discussion started by: sureshcisco
1 Replies
6. Shell Programming and Scripting
Hello,
I have a note pad at /usr/abc location with the following content, since it is a huge file i need to split it into multiple .txt files.
A123|akdhj |21kjsdff |b212b1b21 |0
A123asdasd |assdd |asdasdsdqw|6
A123|QEWQ |NMTGHJK |zxczxczx|3
A123|GEGBGH |RTYBN ... (15 Replies)
Discussion started by: j_panky
15 Replies
7. Shell Programming and Scripting
Hello ,
When using vim, can ctag and cscope support recording search results and displaying the history results ? Once I jump to one tag, I can use :tnext to jump to next tag, but how can I display the preview search result? (0 Replies)
Discussion started by: 915086731
0 Replies
8. Shell Programming and Scripting
I am very very new to this (as in, I didn't even know awk existed till today)
I have a huuuuge csv file. In column 1, there is a ton of emails. I need to find which emails are unique, and save those rows to a separate file. I also need to find which emails are duplicates, and save a record of... (10 Replies)
Discussion started by: shonna
10 Replies
9. What is on Your Mind?
Hey,
I added an animation switch on the search results page; so by default the thread previews are off, but if you want to look at them, just click on the green button and the thread previews will turn on (and back off).
See image and attached animation:
... (1 Reply)
Discussion started by: Neo
1 Replies
10. Linux
All,
I'm trying to learn scrapers, webcrawlers, search engines and CURL. I've chosen to interrogate the
following sites:
Manta,
SuperPages,
Yellow Book,
Yellow Pages.
These show organizations/businesses by search type/category, so effective in
finding potential clients.
... (3 Replies)
Discussion started by: TBotNik
3 Replies
LEARN ABOUT PHP
curl_multi_add_handle
CURL_MULTI_ADD_HANDLE(3) 1 CURL_MULTI_ADD_HANDLE(3)
curl_multi_add_handle - Add a normal cURL handle to a cURL multi handle
SYNOPSIS
int curl_multi_add_handle (resource $mh, resource $ch)
DESCRIPTION
Adds the $ch handle to the multi handle $mh
PARAMETERS
o $mh
-A cURL multi handle returned by curl_multi_init(3).
o $ch
-A cURL handle returned by curl_init(3).
RETURN VALUES
Returns 0 on success, or one of the CURLM_XXX errors code.
EXAMPLES
Example #1
curl_multi_add_handle(3) example
This example will create two cURL handles, add them to a multi handle, and process them asynchronously.
<?php
// create both cURL resources
$ch1 = curl_init();
$ch2 = curl_init();
// set URL and other appropriate options
curl_setopt($ch1, CURLOPT_URL, "http://www.example.com/");
curl_setopt($ch1, CURLOPT_HEADER, 0);
curl_setopt($ch2, CURLOPT_URL, "http://www.php.net/");
curl_setopt($ch2, CURLOPT_HEADER, 0);
//create the multiple cURL handle
$mh = curl_multi_init();
//add the two handles
curl_multi_add_handle($mh,$ch1);
curl_multi_add_handle($mh,$ch2);
$running=null;
//execute the handles
do {
curl_multi_exec($mh,$running);
} while($running > 0);
//close all the handles
curl_multi_remove_handle($mh, $ch1);
curl_multi_remove_handle($mh, $ch2);
curl_multi_close($mh);
?>
SEE ALSO
curl_multi_remove_handle(3), curl_multi_init(3), curl_init(3).
PHP Documentation Group CURL_MULTI_ADD_HANDLE(3)