Sponsored Content
Full Discussion: Python Request Persistent
Top Forums Programming Python Request Persistent Post 302805829 by metallica1973 on Saturday 11th of May 2013 01:48:27 PM
Old 05-11-2013
Python Request Persistent

I noticed that when attempting to download videos from the url, I receive a 403 forbidden when I get through to a certain point in my downloads. I can download a third of the videos but will error:
Code:
Retrieving file 'blah-video.f4v'...
Traceback (most recent call last):                                                                                                                                        ] ETA:  --:--:--   0.00  B/s
  File "./download.py", line 90, in <module>
    retrieve_flv(hashval_url)
  File "./lynda.saint.py", line 61, in retrieve_flv
    remote = urllib2.urlopen(hashval_url)
  File "/usr/lib/python2.7/urllib2.py", line 127, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 407, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 520, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 445, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 379, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 528, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 403: Forbidden

Here is my code that I think is pertenent:
Code:
persistent = requests.Session()
heads = {'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux i686; rv:18.0) Gecko/20100101 Firefox/18.0'}
auths = requests.get('http://www.blah.com/user/login/modal', auth=('blah@blah.com', 'blah-password'), headers = heads)

def find_all_flvs(url):
    soup = BeautifulSoup(urllib2.urlopen(url))
    flvs = []
    for link in soup.findAll(onclick=re.compile("doShowCHys=1*")):
        link = str(link)
        #startpos = link.find("lpk4=") + 5
        #endpos   = link.find("&amp")
        vidnum   = re.findall("(?<==)\d{5,7}", link, re.U)
        vidurl   = "http://www.blah.com/home/GetPlayerXML.aspx?lpk4=%s" % vidnum[0]

        for hashval_url in BeautifulSoup(urllib2.urlopen(vidurl)).findAll("flv"):
            flvs.append(hashval_url.text)

    return flvs

def retrieve_flv (url):
        blah blah blah code.....

I am thinking that when using requests sessions(), its not keeping my cookies persistent across all the request, therefor cause the site to spit out a 403 forbidden. What am I doing wrong?

---------- Post updated at 01:48 PM ---------- Previous update was at 12:57 PM ----------

It must be that my requests.Sessions it not persisting my requests across the spectrum. In checking "auths" after my script fails to download the video, I get this at the end:
Code:
We require that your browser can accept cookies in order to login. Please enable your browser cookies and then close and restart your browser to login.\');\r\n\t\t\t});\r\n\t\t</script>\r\n\t</body>\r\n</html

 

8 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

named pipe with persistent buffer

Hey folks, i need communicate between 2 processes in a reliable manner. The information delivery has to be guarenteed. I thought about proc 2 sending a signal to proc 1 when information has being written to disc and wirte() has been verified (sync/flush). The IPC method for the data is named... (4 Replies)
Discussion started by: heck
4 Replies

2. Shell Programming and Scripting

Persistent variable

I found a way to make a numeric variable persistent for a script : #!/bin/bash function Persist() { # 1:Expression like VARIABLE=Value (numeric) local V=${1%=*} eval "$1" sed -i "s/^$V=*/$1/" $(which $(basename $0)) || return 1 }And how to use itAA=12 read -p "Enter a... (2 Replies)
Discussion started by: frans
2 Replies

3. AIX

HACMP Persistent IP blacklisted

Hi, I want to use the service IP incase for any network activity rather than the persistent IP as the Persistent IP is blacklisted in our network. Is there any way to make the service ip as LPARs default IP to be used as the lpars source IP incase it pings anything or acceses any external... (6 Replies)
Discussion started by: aixromeo
6 Replies

4. AIX

Unable to add persistent IP

i am trying to make HACMP but when i add a persistent ip, error shows unable to determine address for 'UPIDGIS1_pers' pls help me out AIX - 5.3 HACMP -5.4 thanks (2 Replies)
Discussion started by: reply.ravi
2 Replies

5. Solaris

aggr not persistent after a reboot

All, I hope someone can help me on my problem with an aggregate, as I am a Solaris noob. I tried doing this according to the official documentation from Oracle (unfortunately, as a new user to the forum, I may not post URLs...) and also googled a lot around, but have not found any solution yet.... (15 Replies)
Discussion started by: Captainquark
15 Replies

6. War Stories

Why am I persistent to be WRONG!

:wall::wall::wall::wall: I am sharing some techno-geeky-work related issue, which can best be dissected here only. I am expecting more of a discussion,as serious as when two guys sharing experience over smoke and beer. The thing is, till now i have been only a student, so never cared about how... (10 Replies)
Discussion started by: animesharma
10 Replies

7. Programming

Java HTTP PUT Request/JSON Not Working But Using Python It Does ?

I have some code that I have been playing around with learning Java: import java.io.BufferedReader; import java.io.IOException; import java.io.InputStreamReader; import java.io.OutputStream; import java.net.HttpURLConnection; import java.net.MalformedURLException; import java.net.URL; ... (1 Reply)
Discussion started by: metallica1973
1 Replies

8. AIX

BootIP vs Persistent IP in HACMP

I have done other clusters (HP MC/Service Guard and oracle Clusters, and RHEL Cluster services), and have good idea about hacmp (a little older knowledge). However the term "Boot IP" for some reason is messing with my head. Have not done HACMP since the 4.1.2.X days. Is the Bootip the... (1 Reply)
Discussion started by: mrmurdock
1 Replies
LWP-DOWNLOAD(1p)					User Contributed Perl Documentation					  LWP-DOWNLOAD(1p)

NAME
lwp-download - Fetch large files from the web SYNOPSIS
lwp-download [-a] [-s] <url> [<local path>] DESCRIPTION
The lwp-download program will save the file at url to a local file. If local path is not specified, then the current directory is assumed. If local path is a directory, then the last segment of the path of the url is appended to form a local filename. If the url path ends with slash the name "index" is used. With the -s option pick up the last segment of the filename from server provided sources like the Content- Disposition header or any redirect URLs. A file extension to match the server reported Content-Type might also be appended. If a file with the produced filename already exists, then lwp-download will prompt before it overwrites and will fail if its standard input is not a terminal. This form of invocation will also fail is no acceptable filename can be derived from the sources mentioned above. If local path is not a directory, then it is simply used as the path to save into. If the file already exists it's overwritten. The lwp-download program is implemented using the libwww-perl library. It is better suited to down load big files than the lwp-request program because it does not store the file in memory. Another benefit is that it will keep you updated about its progress and that you don't have much options to worry about. Use the "-a" option to save the file in text (ascii) mode. Might make a difference on dosish systems. EXAMPLE
Fetch the newest and greatest perl version: $ lwp-download http://www.perl.com/CPAN/src/latest.tar.gz Saving to 'latest.tar.gz'... 11.4 MB received in 8 seconds (1.43 MB/sec) AUTHOR
Gisle Aas <gisle@aas.no> perl v5.14.2 2012-01-14 LWP-DOWNLOAD(1p)
All times are GMT -4. The time now is 08:41 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy