Using cURL to save online search results


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Using cURL to save online search results
# 1  
Old 06-02-2010
Using cURL to save online search results

Hi, I'm attacking this from ignorance because I am not sure how to even ask the question.

Here is the mission:

I have a list of about 4,000 telephone numbers for past customers. I need to determine how many of these customers are still in business.

Obviously, I could call all the numbers.

But, I could also query whitepages.com's reverse lookup to see if the listing still shows up and eliminate those that don't show up from the list.

I have found out the correct URL to feed whitepages.com
It is Free People Search | WhitePages

Dropping this into a browser address bar works every time.

I need to automate feeding URLs to whitepages.com and save the results for future analysis.

Using curl -O Free People Search | WhitePages downloads nothing since there is no static page at Free People Search | WhitePages.

How can I get curl to "wait" for the results of the query and save that?

Also, if anyone knows of another way to do this, please let me know. In the dark here - the 'services' that do this want $1 a lookup for bulk submission, and offer way more info than I need - which is basically "Is this number still working?"

Thanks!
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Linux

Learning scrapers, webcrawlers, search engines and CURL

All, I'm trying to learn scrapers, webcrawlers, search engines and CURL. I've chosen to interrogate the following sites: Manta, SuperPages, Yellow Book, Yellow Pages. These show organizations/businesses by search type/category, so effective in finding potential clients. ... (3 Replies)
Discussion started by: TBotNik
3 Replies

2. What is on Your Mind?

Search Results (Search, New, and Today's Topics) Animation Switch

Hey, I added an animation switch on the search results page; so by default the thread previews are off, but if you want to look at them, just click on the green button and the thread previews will turn on (and back off). See image and attached animation: ... (1 Reply)
Discussion started by: Neo
1 Replies

3. Shell Programming and Scripting

Using awk to find unique, how to save results?

I am very very new to this (as in, I didn't even know awk existed till today) I have a huuuuge csv file. In column 1, there is a ton of emails. I need to find which emails are unique, and save those rows to a separate file. I also need to find which emails are duplicates, and save a record of... (10 Replies)
Discussion started by: shonna
10 Replies

4. Shell Programming and Scripting

Can ctag and cscope support recording search results and displaying the history results ?

Hello , When using vim, can ctag and cscope support recording search results and displaying the history results ? Once I jump to one tag, I can use :tnext to jump to next tag, but how can I display the preview search result? (0 Replies)
Discussion started by: 915086731
0 Replies

5. Shell Programming and Scripting

Pattern search and save it as .txt file with some name..

Hello, I have a note pad at /usr/abc location with the following content, since it is a huge file i need to split it into multiple .txt files. A123|akdhj |21kjsdff |b212b1b21 |0 A123asdasd |assdd |asdasdsdqw|6 A123|QEWQ |NMTGHJK |zxczxczx|3 A123|GEGBGH |RTYBN ... (15 Replies)
Discussion started by: j_panky
15 Replies

6. Shell Programming and Scripting

Perl - save results to output file.

Can any one please help, the code works...I want the output of $result to be saved in an output.txt file which is lcoated in c:\\temp\\output.txt. $filepath="C:\\temp\\ip.txt"; open (HOSTLIST,"$filepath"); @hosts=(<HOSTLIST>); foreach $host(@hosts) { $results = `nslookup... (1 Reply)
Discussion started by: sureshcisco
1 Replies

7. Shell Programming and Scripting

Save cURL verbose output to file or do it like browser "save as.."

hi there ! i have exactly the same problem like this guy here https://www.unix.com/shell-programming-scripting/127668-getting-curl-output-verbose-file.html i am not able to save the curl verbose output.. the sollution in this thread (redirecting stderr to a file) does not work for me.... (0 Replies)
Discussion started by: crabmeat
0 Replies

8. Shell Programming and Scripting

search a string in a line and save it in a variable

Hi I want to read a file line by line and search for a particular string in each line(say for example string containing @ )and save that string into a variable. Can someone suggest me the way to implement it.I am using K- shell Thanks Ishita (5 Replies)
Discussion started by: Ishita
5 Replies

9. Shell Programming and Scripting

let curl output to stdout AND save to a file

hello hackers. i have a curl process running as cgi directly pushing stdout to the client. but i want to additionally save that stream to a file at the same time. any directions madly welcome. thanks in advance (3 Replies)
Discussion started by: scarfake
3 Replies

10. Shell Programming and Scripting

How search,edit and save the file

Hi All, I want to edit a file using shell script..For ex...a file called /etc/passwd..here I am searching for "ftp" if it is there just change it to "tftp" without using any temporary file. (3 Replies)
Discussion started by: Vichu
3 Replies
Login or Register to Ask a Question