Read webpage from shell script


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Read webpage from shell script
# 8  
Old 03-15-2011
Quote:
Originally Posted by sk1418
do you have wget installed? try this
Code:
 wget -O - https://www.unix.com| [awk or grep]

Hard luck my friend not installed.
# 9  
Old 03-15-2011
@sk1418:
The problem is pages with active content won't get downloaded as you'd want to. E.g.
Code:
curl www.unix.com | grep Copyright

returns 3 lines, whereas
Code:
wget www.unix.com -O - | grep Copyright

returns none due to some active contents...
# 10  
Old 03-15-2011
Well since you don't have curl AND wget.. do you have python installed?

save this into a .py file, e.g t.py

Code:
#!/usr/bin/python
import urllib
print urllib.urlopen("https://www.unix.com").read()

then you could do something like this:
Code:
kent$ python t.py| grep -c "Copyright"       
3



@mirni
Code:
wget www.unix.com -O - | grep Copyright

works here. I can see 3 red "Copyright" Smilie I 've alias grep with color.
# 11  
Old 03-16-2011
Hi Guys,

I do not have wget or lynx installed and pyton doesn't seems to work. Is there ne way to write a shell script.?

Your response is appreciated.
# 12  
Old 03-16-2011
Quote:
Originally Posted by bankimmehta
I do not have wget or lynx installed
This is why it's always helpful to know what your system is.
Quote:
and pyton doesn't seems to work.
Doesn't seem to work? What, exactly, does it do? It's "python", by the way, check your spelling in the script too.
Quote:
Is there any way to write a shell script.?
No netspeak, please.

Some versions of ksh allow you to create a network socket but this would be a very hackish and breakage-prone way to do it. You should try and get the right tool for the job.

What is your system, anyway? Did you try curl?
# 13  
Old 03-16-2011
Some shellssupport sockets.
With some versions of bash you could write something like this:

Code:
exec 3<>/dev/tcp/www.unix.com/80
printf 'GET / HTTP/1.1\nHost: www.unix.com\nConnection: close\n\n' >&3
grep <&3 Copyright
exec 3>&-

---------- Post updated at 11:44 PM ---------- Previous update was at 11:42 PM ----------

Quote:
Originally Posted by Corona688
[...]
Some versions of ksh allow you to create a network socket but this would be a very hackish and breakage-prone way to do it. You should try and get the right tool for the job.
I agree, I would use Perl.

The OP seems to be using AIX.

Last edited by radoulov; 03-16-2011 at 07:50 PM.. Reason: Useless Use of Cat Award :)
# 14  
Old 03-16-2011
i don't have experience of AIX. but python is not a standard package of AIX?
btw, @bankimmehta, if you say "python doesn't seem to work", what is the output then?
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Executing Shell Script from PHP Webpage

Hello , I am trying to execute a small shell script from a PHP Webpage. Below is the shell script : 1.sh $ cat /home/oracle/rahul/1.sh #!/bin/ksh echo "`date` : Report generation in progress" > /home/oracle/rahul/report.txt echo "Run successfully script"; Below is the PHP... (1 Reply)
Discussion started by: rahul2662
1 Replies

2. Shell Programming and Scripting

Launch shell script with parameters from a webpage button

I want to create a simple html page that should contain 2 fields in which the user can write the input. Then I want to have a button that should launch a shell script with the parameters inserted by user in the fields from the webpage. This is the code behind my webpage: <html> <form... (2 Replies)
Discussion started by: black_fender
2 Replies

3. UNIX for Dummies Questions & Answers

Shell script to read lines in a text file and filter user data Shell Programming and Scripting

sxsaaas (3 Replies)
Discussion started by: VikrantD
3 Replies

4. UNIX for Dummies Questions & Answers

Read version from a webpage and Upgrading the file.

Hi, I am trying to read version from a web url like <url/daily-builds/sdk-3.2.v20140312170355-osx.zip> and download and store in my filesystem if its the latest. Kindly help me how to read version 3.2.v20140312170355 and compare same with a folder named similar in my directory and... (1 Reply)
Discussion started by: pragyarastogi
1 Replies

5. Programming

Async webpage read

hi i need to asynchronous connect to webpage which has only text file. need to read its content, line by line using linux socket. any samples? (1 Reply)
Discussion started by: leo2008
1 Replies

6. Shell Programming and Scripting

Use Webpage to Start Script

I apologize if this is in the incorrect section, I'm not quite sure which section it should go in. Anyways... I've got a script that I'd like to be able to start with a webpage, something that just has a button that says "Start this Bot", which will start the bot and put it in the background.... (4 Replies)
Discussion started by: JoeGazz84
4 Replies

7. Homework & Coursework Questions

Passing shell variables to a webpage

Use and complete the template provided. The entire template must be completed. If you don't, your post may be deleted! 1. The problem statement, all variables and given/known data: This is my assignment as a whole - Use SVG to present certain dynamic (the raw data should change at least once... (5 Replies)
Discussion started by: ChedWick
5 Replies

8. Programming

socket function to read a webpage (socket.h)

Why does this socket function only read the first 1440 chars of the stream. Why not the whole stream ? I checked it with gdm and valgrind and everything seems correct... #include <stdio.h> #include <stdlib.h> #include <sys/types.h> #include <sys/stat.h> #include <string.h> #include... (3 Replies)
Discussion started by: cyler
3 Replies

9. Shell Programming and Scripting

Run a script from a webpage

Hello, I was trying to run a shell script when I click on webpage link. My webpage is hosted on Sun One web server. This is something I was looking, Chris Johnson's Test Bed I already have my shell script and it works fine when I run it from the command line. Can somebody tell me what are... (3 Replies)
Discussion started by: grajp002
3 Replies

10. Shell Programming and Scripting

Run shell commands via HTML webpage/button?

Hey guys, I got a issue here... we have a development box and a UAT box that our webmasters use, they do the webpage development on the development box and and save changed files to a particular directory on the dev machine. At a certain time of the day a cronjob kicks off and the... (3 Replies)
Discussion started by: zeekblack
3 Replies
Login or Register to Ask a Question