Sponsored Content
Top Forums Programming extract the username and the url and print it using Python 3 and BeautifulSoup 4 Post 303045287 by bob123 on Sunday 15th of March 2020 04:01:52 PM
Old 03-15-2020
extract the username and the url and print it using Python 3 and BeautifulSoup 4

using python 3 and BeautifulSoup 4
my code when run returns a big list like the txt below
but 100 times more i have only posted one as an example


Code:
urls = ['https://example.com']
for url in urls:
    response = requests.get(url)
    soup = BeautifulSoup(response.content, "html.parser")
    for text in soup.find_all(text= True):
           print(text)


code above returns
this

Code:
{"users":{"userId":222222222,"username":"some_name","hlsPreviewUrl":"https://example.com/322/example-origin-live/some_name-322-33babf1b-6030-479a-ac41-b90bd8313678_aac/playlist.m3u8 }

i want to extract the username and the url and print it

i have tried to add this piece of code but i get nothing

Code:
 if 'username' in ['text']:

what code am i missing
to print just the username and the url
with python 3 and BeautifulSoup 4

thanks

Last edited by bob123; 03-15-2020 at 05:07 PM..
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

SED extract url - please help a lamer

Hello everybody. I have lines that looks something like this: <done16=""118"" done18=""$ title=""thisisatitle"" href=""/JoeBanana" alt=""Joe""><done16=""118"" done18=""$ title=""thisisatitle"" href=""/GeraldGiraffe" alt=""Gerald""> What kind of SED command would I need to use to extract... (4 Replies)
Discussion started by: digi
4 Replies

2. Shell Programming and Scripting

How to extract url from html page?

for example, I have an html file, contain <a href="http://awebsite" id="awebsite" class="first">website</a>and sometime a line contains more then one link, for example <a href="http://awebsite" id="awebsite" class="first">website</a><a href="http://bwebsite" id="bwebsite"... (36 Replies)
Discussion started by: 14th
36 Replies

3. Shell Programming and Scripting

help, extract username from file

there is a text file like this text here text here text here text here USER=someusername here text here text here text here how can i extract the someusername from this file which has a path like /home/myfile.txt? i want to store it in a variable $u (3 Replies)
Discussion started by: vanessafan99
3 Replies

4. UNIX for Dummies Questions & Answers

Awk: print all URL addresses between iframe tags without repeating an already printed URL

Here is what I have so far: find . -name "*php*" -or -name "*htm*" | xargs grep -i iframe | awk -F'"' '/<iframe*/{gsub(/.\*iframe>/,"\"");print $2}' Here is an example content of a PHP or HTM(HTML) file: <iframe src="http://ADDRESS_1/?click=5BBB08\" width=1 height=1... (18 Replies)
Discussion started by: striker4o
18 Replies

5. UNIX for Dummies Questions & Answers

Launch a URL,validate username and password using wget or curl

Hi All, I want to launch "ex: http://gmail.com" from the cmd window and validate the credentials with username and password, is it possible? I have found something like this "wget --http-user=USER' --http-password=PASSWORD http://gmail.com" am new to this and unable to find a solution, i... (0 Replies)
Discussion started by: harsha85
0 Replies

6. UNIX for Dummies Questions & Answers

Mailx: extract username from a file

Hey guys, I am trying to send email to the usernames listed in a ascii file. I know about the .mailrc file. I want to know if I there is a way to extract the usernames from a file and email them. I tried something like this: mailx -s "subject" '~cat... (2 Replies)
Discussion started by: Nirav4
2 Replies

7. Shell Programming and Scripting

Newbie Python Url Scraper

I setup Zoneminder and have been playing around with setting up a couple of Wanscam PTZ ip cameras in which I have been running into road blocks with streaming and etc. I cant find much information on the camera and its webserver that sits on it and wanted to get a an absolute directory structure... (5 Replies)
Discussion started by: metallica1973
5 Replies

8. Shell Programming and Scripting

Connect to Remote DB2 ,using URL ,username,Password

Hi All, I am new to Shell scripting, I want to connect to DB2 database through URL,username and password Please help me out. I read so many posts in that there is no where it is mentioned about how to connect to a remote database. Thanks in advance :) (0 Replies)
Discussion started by: Rohit G
0 Replies

9. Shell Programming and Scripting

Python BeautifulSoup Re Finding Digits Within Tags

I am writing a little python script that needs to grab version numbers between "<td>4.2.2</td>" within the tbody of the page: blah blah blah blah blah Is it possible to use a one-liner to scrap only the digits between the tags: "<td>4.2.2</td>" so it spits out: 4.2.2 4.2.1 etc..... (2 Replies)
Discussion started by: metallica1973
2 Replies

10. Programming

[Python] BeautifulSoup tags > </a>

using BeautifulSoup how can i get the txt between all the > </a> example >The Student .mp4</a> thanks (10 Replies)
Discussion started by: bob123
10 Replies
LWP::Authen::Wsse(3pm)					User Contributed Perl Documentation				    LWP::Authen::Wsse(3pm)

NAME
LWP::Authen::Wsse - Library for enabling X-WSSE authentication in LWP VERSION
This document describes version 0.05 of LWP::Authen::Wsse, released December 27, 2005. SYNOPSIS
use LWP::UserAgent; use HTTP::Request::Common; my $url = 'http://www.example.org/protected_page.html'; # Set up the WSSE client my $ua = LWP::UserAgent->new; $ua->credentials('example.org', '', 'username', 'password'); $request = GET $url; print "--Performing request now...----------- "; $response = $ua->request($request); print "--Done with request------------------- "; if ($response->is_success) { print "It worked!->", $response->code, " "; } else { print "It didn't work!->", $response->code, " "; } DESCRIPTION
"LWP::Authen::Wsse" allows LWP to authenticate against servers that are using the "X-WSSE" authentication scheme, as required by the Atom Authentication API. The module is used indirectly through LWP, rather than including it directly in your code. The LWP system will invoke the WSSE authentication when it encounters the authentication scheme while attempting to retrieve a URL from a server. You also need to set the credentials on the UserAgent object like this: $ua->credentials('www.company.com:80', '', "username", "password"); Alternatively, you may also subclass LWP::UserAgent and override the "get_basic_credentials()" method. See LWP::UserAgent for more details. SEE ALSO
LWP, LWP::UserAgent, lwpcook. AUTHORS
Audrey Tang <audrey@audrey.org> COPYRIGHT
Copyright 2004, 2005 by Audrey Tang <audrey@audrey.org>. This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself. See <http://www.perl.com/perl/misc/Artistic.html> perl v5.14.2 2012-11-27 LWP::Authen::Wsse(3pm)
All times are GMT -4. The time now is 05:04 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy