help pulling ${VARS} out of a web page user curl Post: 302683919

Sponsored Content

Top Forums Shell Programming and Scripting help pulling ${VARS} out of a web page user curl Post 302683919 by agama on Wednesday 8th of August 2012 11:32:18 PM

08-09-2012

Registered User

I assume you wanted the value into shell variables, not awk variables since you used a dollar sign prefix in your example.

First example works only in Kshell (cleaner if you're willing to use kshell). Second example, with a hack needed to get the variables will work in bash. I don't regularly use bash, so maybe there is a better/easier way, but I'm not finding it in my brain tonight.

Code:

#!/usr/bin/env ksh

url="http://www.everymac.com/systems/apple/macbook_pro/specs/macbook-pro-core-2-duo-2.8-aluminum-17-mid-2009-unibody-specs.html"

curl --silent "$url" | awk  '{gsub( "<[^>]*>", "\n" ); gsub( "\r", "" );  print;}'   |awk '
    /Standard RAM/      {snarf = 1; what = "sram"; next; }
    /Maximum RAM:/      {snarf = 1; what = "mram"; next; }
    /Apple Model No:/   {snarf = 1; what = "modeln"; next; }
    /Model ID:/         {snarf = 1; what = "modeli"; next; }

    NF > 0 && snarf { v[what] = $0; snarf = 0; next; }

    END { printf( "%s,%s,%s,%s\n", v["sram"], v["mram"], v["modeln"], v["modeli"] ); } ' | IFS=, read var1 var2 var3 var4

echo "v1=$var1  v2=$var2  v3=$var3  v4=$var4"

Code:

#!/usr/bin/env bash
url="http://www.everymac.com/systems/apple/macbook_pro/specs/macbook-pro-core-2-duo-2.8-aluminum-17-mid-2009-unibody-specs.html"

# not as clean, but works...
curl --silent "$url" | awk  '{gsub( "<[^>]*>", "\n" ); gsub( "\r", "" );  print;}'  | awk '
    /Standard RAM/      {snarf = 1; what = "sram"; next; }
    /Maximum RAM:/      {snarf = 1; what = "mram"; next; }
    /Apple Model No:/   {snarf = 1; what = "modeln"; next; }
    /Model ID:/         {snarf = 1; what = "modeli"; next; }

    NF > 0 && snarf {  v[what] = $0;  snarf = 0; next; }
   
 END {  printf( "var1=\"%s\"\nvar2=\"%s\"\nvar3=\"%s\"\nvar4=\"%s\"\n", v["sram"], v["mram"], v["modeln"], v["modeli"] ); } ' >/tmp/hack.$$

. /tmp/hack.$$
rm /tmp/hack.$$

echo "v1=$var1  v2=$var2  v3=$var3  v4=$var4"

Hope one of these helps.

agama

View Public Profile for agama

Find all posts by agama

9 More Discussions You Might Find Interesting

1. Programming

fetching a web page in C

Hello, I'm a total newbie to HTTP commands, so I'm not sure how to do this. What I'd like is to write a C program to fetch the contents of a html page of a given address. Could someone help with this? Thanks in advance!

2. UNIX for Dummies Questions & Answers

curl command with web pages

I can't quite seem to understand what the curl command does with a web address. I tried this: curl O'Reilly Media: Tech Books, Conferences, Courses, News but I just got the first few lines of a web page, and it's nowhere on my machine. Can someone elaborate?

3. Shell Programming and Scripting

web service call: curl output to xsltproc input

I need to invoke a web service and extract what I need from the response using a combination of curl and xsltproc. However, any file-based parameters that must be supplied to both these programs must be from stdin and not actual files. At least with curl, it seems to think that I am supplying a...

4. Shell Programming and Scripting

Grabbin a html code from a page (Var = Curl)

Hi there. Im not very good on shell yet. This line, will print me YES or NO in console. Its the HTML code returned from the website, simply YES or NO curl -L "http://www.thewebsite.net/auth/log.jsp?user=$user&sessionId=$sid&serverId=$hash" How could i save this into a variable, so i...

5. UNIX for Dummies Questions & Answers

How to switch the user before executing a shell script from web page??

hi, i want to execute a shell script as a different user. the flow is like this. there is a html web page from which i have to call a shell script. web server is apache. to call the shell script from html page, a perl script is required. so the html page calls the perl script and the perl...

6. Shell Programming and Scripting

Use curl to send a static xml file using url encoding to a web page using pos

Hi I am try to use curl to send a static xml file using url encoding to a web page using post. This has to go through a particular port on our firewall as well. This is my first exposure to curl and am not having much success, so any help you can supply, or point me in the right direction would be...

7. UNIX for Beginners Questions & Answers

How to use cURL to download web page with authentification (form)?

Hello, I'm new in the forum and really beginer, and also sorry form my bad english. I use linux and want to create little program to download automaticaly some pdf (invoices) and put in a folder of my computer. I learn how to do and programme with ubuntu but the program will be implemented...

8. Linux

Curl login to web page

Hello dears, I am trying to log in the website using curl but no luck so far. The web page Content-Type is : text/html;charset=ISO-8859-1 I try the following command using curl: curl \ --header "Content-type: text/html" \ --request POST \ --data '{"user": "someusername",...

9. Shell Programming and Scripting

How to get contents of php page using curl in C language?

Hi, I want to write code in C using curl library to get output of php page , the output is in xml. Thanks

LEARN ABOUT DEBIAN

apsfilter-bug

APSFILTER-BUG(1)					    BSD General Commands Manual 					  APSFILTER-BUG(1)

NAME

     apsfilter-bug -- create a half-automatic bug report for apsfilter

SYNOPSIS

     apsfilter-bug

OPTIONS

     none

DESCRIPTION

     With the apsfilter-bug script you can automatically create a bug report for apsfilter(1).	A template report file is created which you can
     edit with your favourite editor ( emacs(1) by default, or whatever is entered in the environment variable EDITOR ).

     You will see a couple of lines starting with '#'; these lines will be removed before the report is sent. So if you want to use the '#' char-
     acter at the beginning of a line, just indent it with a space.

     A very important piece of the bug report is the debugging log created by aps2file(1), probably with a command like
	   aps2file -D -o /dev/null [-P...] [-Z...] [input] 2> log.txt
     Unless you don't even have a chance to create a debugging log, it is crucial for serious bug hunting.

     After you have filled in as much (useful) information as possible, you must save the file under its original name, then exit your editor.

     If you invoked apsfilter-bug by accident, or if you need some additional system information to include into the bug report, just exit your
     editor without saving the file.

NOTES

     In addition to the stuff you've edited, some more information is appended to the bug report automatically:

     apsfilter version	  7.2.6-stable
     ghostscript version  (from 'gs --version')
     system id		  (from 'uname -a')
     configure options	  --prefix=/usr --sysconfdir=/etc --mandir=/usr/share/man --with-awk=/usr/bin/awk --with-sendmail=/usr/sbin/sendmail
     shell executable	  /bin/bash
     awk executable	  /usr/bin/awk
     sendmail executable  /usr/sbin/sendmail

FILES

     /usr/bin/aps2file	script to create debugging output

SEE ALSO

     apsfilter(1), aps2file(1)

BUGS

     See apsfilter software center - http://www.apsfilter.org/ - for new versions, bugfixes and known bugs.

     Please use the new tool apsfilter-bug(1) to release bug- or problem reports. It automatically presents you a form in an editor window which
     asks you some standard questions.	If you save and quit the editor session, then this report is sent automatically via e-mail to the proper
     apsfilter mailinglist.

     If apsfilter fails to print something or prints it in a way you wouldn't expect and you want to report an apsfilter error then please save
     the debugging output of one print session using the new aps2file(1) utility by typing aps2file -D -Z options file > /dev/null 2> file.debug
     and including the debugging output in the file file.debug into the edit session of the apsfilter-bug utility, so that it is included into the
     mail to the apsfilter mailinglist.

     Please note that you need to run /bin/sh (Bourne Shell), bash or a compatible shell, so that the above mentioned output redirection works.
     Under C-shell (/bin/csh) or tcsh it would't work. If you don't know, then simply make sure you use the Bournce shell by typing /bin/sh or
     bash, then you should have no problems with redirection of stdout and stderr (> /dev/null 2> file.debug).

DOCUMENTATION

     See official apsfilter homepage
     http://www.apsfilter.org/handbook.html   Apsfilter Handbook including the Frequently Asked Questions (FAQ)

USER FORUM

     Please send questions to the official apsfilter help channel apsfilter-help@apsfilter.org.  The above section BUGS and the file
     HOWTO-BUGREPORTS tells you how to report bugs.  If you want to know how to troubleshoot your apsfilter installation, please read the manpage
     aps2file(1) and apsfilter-bug(1) as well as the Apsfilter Handbook carefully.

HISTORY

     The apsfilter-bug manpage has been written by Michael LoBin <phallobst@web.de> and first appeared in apsfilter V 7.1.0.

BSD
								   Dec 26, 2001 							       BSD