help pulling ${VARS} out of a web page user curl Post: 302683919

Sponsored Content

Top Forums Shell Programming and Scripting help pulling ${VARS} out of a web page user curl Post 302683919 by agama on Wednesday 8th of August 2012 11:32:18 PM

08-09-2012

Registered User

I assume you wanted the value into shell variables, not awk variables since you used a dollar sign prefix in your example.

First example works only in Kshell (cleaner if you're willing to use kshell). Second example, with a hack needed to get the variables will work in bash. I don't regularly use bash, so maybe there is a better/easier way, but I'm not finding it in my brain tonight.

Code:

#!/usr/bin/env ksh

url="http://www.everymac.com/systems/apple/macbook_pro/specs/macbook-pro-core-2-duo-2.8-aluminum-17-mid-2009-unibody-specs.html"

curl --silent "$url" | awk  '{gsub( "<[^>]*>", "\n" ); gsub( "\r", "" );  print;}'   |awk '
    /Standard RAM/      {snarf = 1; what = "sram"; next; }
    /Maximum RAM:/      {snarf = 1; what = "mram"; next; }
    /Apple Model No:/   {snarf = 1; what = "modeln"; next; }
    /Model ID:/         {snarf = 1; what = "modeli"; next; }

    NF > 0 && snarf { v[what] = $0; snarf = 0; next; }

    END { printf( "%s,%s,%s,%s\n", v["sram"], v["mram"], v["modeln"], v["modeli"] ); } ' | IFS=, read var1 var2 var3 var4

echo "v1=$var1  v2=$var2  v3=$var3  v4=$var4"

Code:

#!/usr/bin/env bash
url="http://www.everymac.com/systems/apple/macbook_pro/specs/macbook-pro-core-2-duo-2.8-aluminum-17-mid-2009-unibody-specs.html"

# not as clean, but works...
curl --silent "$url" | awk  '{gsub( "<[^>]*>", "\n" ); gsub( "\r", "" );  print;}'  | awk '
    /Standard RAM/      {snarf = 1; what = "sram"; next; }
    /Maximum RAM:/      {snarf = 1; what = "mram"; next; }
    /Apple Model No:/   {snarf = 1; what = "modeln"; next; }
    /Model ID:/         {snarf = 1; what = "modeli"; next; }

    NF > 0 && snarf {  v[what] = $0;  snarf = 0; next; }
   
 END {  printf( "var1=\"%s\"\nvar2=\"%s\"\nvar3=\"%s\"\nvar4=\"%s\"\n", v["sram"], v["mram"], v["modeln"], v["modeli"] ); } ' >/tmp/hack.$$

. /tmp/hack.$$
rm /tmp/hack.$$

echo "v1=$var1  v2=$var2  v3=$var3  v4=$var4"

Hope one of these helps.

agama

View Public Profile for agama

Find all posts by agama

9 More Discussions You Might Find Interesting

1. Programming

fetching a web page in C

Hello, I'm a total newbie to HTTP commands, so I'm not sure how to do this. What I'd like is to write a C program to fetch the contents of a html page of a given address. Could someone help with this? Thanks in advance!

2. UNIX for Dummies Questions & Answers

curl command with web pages

I can't quite seem to understand what the curl command does with a web address. I tried this: curl O'Reilly Media: Tech Books, Conferences, Courses, News but I just got the first few lines of a web page, and it's nowhere on my machine. Can someone elaborate?

3. Shell Programming and Scripting

web service call: curl output to xsltproc input

I need to invoke a web service and extract what I need from the response using a combination of curl and xsltproc. However, any file-based parameters that must be supplied to both these programs must be from stdin and not actual files. At least with curl, it seems to think that I am supplying a...

4. Shell Programming and Scripting

Grabbin a html code from a page (Var = Curl)

Hi there. Im not very good on shell yet. This line, will print me YES or NO in console. Its the HTML code returned from the website, simply YES or NO curl -L "http://www.thewebsite.net/auth/log.jsp?user=$user&sessionId=$sid&serverId=$hash" How could i save this into a variable, so i...

5. UNIX for Dummies Questions & Answers

How to switch the user before executing a shell script from web page??

hi, i want to execute a shell script as a different user. the flow is like this. there is a html web page from which i have to call a shell script. web server is apache. to call the shell script from html page, a perl script is required. so the html page calls the perl script and the perl...

6. Shell Programming and Scripting

Use curl to send a static xml file using url encoding to a web page using pos

Hi I am try to use curl to send a static xml file using url encoding to a web page using post. This has to go through a particular port on our firewall as well. This is my first exposure to curl and am not having much success, so any help you can supply, or point me in the right direction would be...

7. UNIX for Beginners Questions & Answers

How to use cURL to download web page with authentification (form)?

Hello, I'm new in the forum and really beginer, and also sorry form my bad english. I use linux and want to create little program to download automaticaly some pdf (invoices) and put in a folder of my computer. I learn how to do and programme with ubuntu but the program will be implemented...

8. Linux

Curl login to web page

Hello dears, I am trying to log in the website using curl but no luck so far. The web page Content-Type is : text/html;charset=ISO-8859-1 I try the following command using curl: curl \ --header "Content-type: text/html" \ --request POST \ --data '{"user": "someusername",...

9. Shell Programming and Scripting

How to get contents of php page using curl in C language?

Hi, I want to write code in C using curl library to get output of php page , the output is in xml. Thanks

LEARN ABOUT MOJAVE

curlopt_httpget

CURLOPT_HTTPGET(3)					     curl_easy_setopt options						CURLOPT_HTTPGET(3)

NAME

       CURLOPT_HTTPGET - ask for a HTTP GET request

SYNOPSIS

       #include <curl/curl.h>

       CURLcode curl_easy_setopt(CURL *handle, CURLOPT_HTTPGET, long useget);

DESCRIPTION

       Pass  a	long. If useget is 1, this forces the HTTP request to get back to using GET. Usable if a POST, HEAD, PUT, etc has been used previ-
       ously using the same curl handle.

       When setting CURLOPT_HTTPGET(3) to 1, it will automatically set CURLOPT_NOBODY(3) to 0 and CURLOPT_UPLOAD(3) to 0.

DEFAULT

       0

PROTOCOLS

       HTTP(S)

EXAMPLE

       curl = curl_easy_init();
       if(curl) {
	 curl_easy_setopt(curl, CURLOPT_URL, "http://example.com");

	 /* use a GET to fetch this */
	 curl_easy_setopt(curl, CURLOPT_HTTPGET, 1L);

	 /* Perform the request */
	 curl_easy_perform(curl);
       }

AVAILABILITY

       Along with HTTP

RETURN VALUE

       Returns CURLE_OK if HTTP is supported, and CURLE_UNKNOWN_OPTION if not.

SEE ALSO

       CURLOPT_NOBODY(3), CURLOPT_UPLOAD(3),

libcurl 7.54.0							 February 03, 2016						CURLOPT_HTTPGET(3)