Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Python Request Persistent
Top Forums Programming Python Request Persistent Post 302805829 by metallica1973 on Saturday 11th of May 2013 01:48:27 PM
Old 05-11-2013
Python Request Persistent
I noticed that when attempting to download videos from the url, I receive a 403 forbidden when I get through to a certain point in my downloads. I can download a third of the videos but will error:
Code:
Retrieving file 'blah-video.f4v'...
Traceback (most recent call last):                                                                                                                                        ] ETA:  --:--:--   0.00  B/s
  File "./download.py", line 90, in <module>
    retrieve_flv(hashval_url)
  File "./lynda.saint.py", line 61, in retrieve_flv
    remote = urllib2.urlopen(hashval_url)
  File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 407, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 520, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 445, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 379, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 528, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 403: Forbidden

Here is my code that I think is pertenent:
Code:
persistent = requests.Session()
heads = {'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:18.0) Gecko/20100101 Firefox/18.0'}
auths = requests.get('http://www.blah.com/user/login/modal', auth=('blah@blah.com', 'blah-password'), headers = heads)

def find_all_flvs(url):
    soup = BeautifulSoup(urllib2.urlopen(url))
    flvs = []
    for link in soup.findAll(onclick=re.compile("doShowCHys=1*")):
        link = str(link)
        #startpos = link.find("lpk4=") + 5
        #endpos   = link.find("&amp")
        vidnum   = re.findall("(?<==)\d{5,7}", link, re.U)
        vidurl   = "http://www.blah.com/home/GetPlayerXML.aspx?lpk4=%s" % vidnum[0]

        for hashval_url in BeautifulSoup(urllib2.urlopen(vidurl)).findAll("flv"):
            flvs.append(hashval_url.text)

    return flvs

def retrieve_flv (url):
        blah blah blah code.....

I am thinking that when using requests sessions(), its not keeping my cookies persistent across all the request, therefor cause the site to spit out a 403 forbidden. What am I doing wrong?

---------- Post updated at 01:48 PM ---------- Previous update was at 12:57 PM ----------

It must be that my requests.Sessions it not persisting my requests across the spectrum. In checking "auths" after my script fails to download the video, I get this at the end:
Code:
We require that your browser can accept cookies in order to login. Please enable your browser cookies and then close and restart your browser to login.\');\r\n\t\t\t});\r\n\t\t</script>\r\n\t</body>\r\n</html

 

8 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

named pipe with persistent buffer

Hey folks, i need communicate between 2 processes in a reliable manner. The information delivery has to be guarenteed. I thought about proc 2 sending a signal to proc 1 when information has being written to disc and wirte() has been verified (sync/flush). The IPC method for the data is named... (4 Replies)
Discussion started by: heck
4 Replies

2. Shell Programming and Scripting

Persistent variable

I found a way to make a numeric variable persistent for a script : #!/bin/bash function Persist() { # 1:Expression like VARIABLE=Value (numeric) local V=${1%=*} eval "$1" sed -i "s/^$V=*/$1/" $(which $(basename $0)) || return 1 }And how to use itAA=12 read -p "Enter a... (2 Replies)
Discussion started by: frans
2 Replies

3. AIX

HACMP Persistent IP blacklisted

Hi, I want to use the service IP incase for any network activity rather than the persistent IP as the Persistent IP is blacklisted in our network. Is there any way to make the service ip as LPARs default IP to be used as the lpars source IP incase it pings anything or acceses any external... (6 Replies)
Discussion started by: aixromeo
6 Replies

4. AIX

Unable to add persistent IP

i am trying to make HACMP but when i add a persistent ip, error shows unable to determine address for 'UPIDGIS1_pers' pls help me out AIX - 5.3 HACMP -5.4 thanks (2 Replies)
Discussion started by: reply.ravi
2 Replies

5. Solaris

aggr not persistent after a reboot

All, I hope someone can help me on my problem with an aggregate, as I am a Solaris noob. I tried doing this according to the official documentation from Oracle (unfortunately, as a new user to the forum, I may not post URLs...) and also googled a lot around, but have not found any solution yet.... (15 Replies)
Discussion started by: Captainquark
15 Replies

6. War Stories

Why am I persistent to be WRONG!

:wall::wall::wall::wall: I am sharing some techno-geeky-work related issue, which can best be dissected here only. I am expecting more of a discussion,as serious as when two guys sharing experience over smoke and beer. The thing is, till now i have been only a student, so never cared about how... (10 Replies)
Discussion started by: animesharma
10 Replies

7. Programming

Java HTTP PUT Request/JSON Not Working But Using Python It Does ?

I have some code that I have been playing around with learning Java: import java.io.BufferedReader; import java.io.IOException; import java.io.InputStreamReader; import java.io.OutputStream; import java.net.HttpURLConnection; import java.net.MalformedURLException; import java.net.URL; ... (1 Reply)
Discussion started by: metallica1973
1 Replies

8. AIX

BootIP vs Persistent IP in HACMP

I have done other clusters (HP MC/Service Guard and oracle Clusters, and RHEL Cluster services), and have good idea about hacmp (a little older knowledge). However the term "Boot IP" for some reason is messing with my head. Have not done HACMP since the 4.1.2.X days. Is the Bootip the... (1 Reply)
Discussion started by: mrmurdock
1 Replies

LEARN ABOUT DEBIAN

webservice::youtube::feeds

WebService::YouTube::Feeds(3pm) 			User Contributed Perl Documentation			   WebService::YouTube::Feeds(3pm)

NAME

       WebService::YouTube::Feeds - Perl interface to YouTube RSS Feeds

VERSION

       This document describes WebService::YouTube::Feeds version 1.0.3

SYNOPSIS

	   use WebService::YouTube::Feeds;

	   my $feeds = WebService::YouTube::Feeds->new( { ua => '...' } );

	   my @videos = $feeds->tag($tag);
	   my @videos = $feeds->user($user);
	   my @videos = $feeds->recently_added;
	   my @videos = $feeds->recently_featured;
	   my @videos = $feeds->top_favorites;
	   my @videos = $feeds->top_rated;
	   my @videos = $feeds->most_discussed_month;
	   my @videos = $feeds->most_discussed_today;
	   my @videos = $feeds->most_discussed_week;
	   my @videos = $feeds->top_viewed;
	   my @videos = $feeds->top_viewed_month;
	   my @videos = $feeds->top_viewed_today;
	   my @videos = $feeds->top_viewed_week;

DESCRIPTION

       This is a Perl interface to YouTube RSS Feeds.

       See About RSS <http://youtube.com/rssls> for details.

SUBROUTINES
/METHODS
   new( \%fields )
       Creates and returns a new WebService::YouTube::Feeds object.  %fields can contain parameters enumerated in "ACCESSORS" section.

   parse_rss($rss)
       Parses RSS and returns the result.  $rss should be an object that XML::Simple can understand.

   tag( $tag )
       Returns an array of WebService::YouTube::Video object.  $tag is a keyword string separated by a space.

       See <http://youtube.com/rssls> for details.

   user( $user )
       Returns an array of WebService::YouTube::Video object.  $user is an username.

       See <http://youtube.com/rssls> for details.

   recently_added( )
       Returns an array of WebService::YouTube::Video object.

       See <http://youtube.com/rssls> for details.

   recently_featured( )
       Returns an array of WebService::YouTube::Video object.

       See <http://youtube.com/rssls> for details.

   top_favorites( )
       Returns an array of WebService::YouTube::Video object.

       See <http://youtube.com/rssls> for details.

   top_rated( )
       Returns an array of WebService::YouTube::Video object.

       See <http://youtube.com/rssls> for details.

   most_discussed_month( )
       Returns an array of WebService::YouTube::Video object.

       See <http://youtube.com/rssls> for details.

   most_discussed_today( )
       Returns an array of WebService::YouTube::Video object.

       See <http://youtube.com/rssls> for details.

   most_discussed_week( )
       Returns an array of WebService::YouTube::Video object.

       See <http://youtube.com/rssls> for details.

   top_viewed( )
       Returns an array of WebService::YouTube::Video object.

       See <http://youtube.com/rssls> for details.

   top_viewed_month( )
       Returns an array of WebService::YouTube::Video object.

       See <http://youtube.com/rssls> for details.

   top_viewed_today( )
       Returns an array of WebService::YouTube::Video object.

       See <http://youtube.com/rssls> for details.

   top_viewed_week( )
       Returns an array of WebService::YouTube::Video object.

       See <http://youtube.com/rssls> for details.

   ACCESSORS
       ua

       LWP::UserAgent object

DIAGNOSTICS

CONFIGURATION AND ENVIRONMENT

       WebService::YouTube::Feeds requires no configuration files or environment variables.

DEPENDENCIES

       HTTP::Date, LWP::UserAgent, XML::Simple, WebService::YouTube::Util, WebService::YouTube::Video

INCOMPATIBILITIES

       None reported.

BUGS AND LIMITATIONS

       No bugs have been reported.

       Please report any bugs or feature requests to "bug-webservice-youtube@rt.cpan.org", or through the web interface at
       <http://rt.cpan.org/NoAuth/ReportBug.html?Queue=WebService-YouTube>.  I will be notified, and then you'll automatically be notified of
       progress on your bug as I make changes.

AUTHOR

       Hironori Yoshida <yoshida@cpan.org>

LICENSE AND COPYRIGHT

       This module is free software; you can redistribute it and/or modify it under the same terms as Perl itself. See perlartistic.

perl v5.10.1							    2010-08-01					   WebService::YouTube::Feeds(3pm)
All times are GMT -4. The time now is 07:45 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy