Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Two while loops with wget issue

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Two while loops with wget issue
# 1  
Old 08-24-2018
Two while loops with wget issue
Hello,

When I run below script, everything is okay:

Code:
#!/bin/bash
wget --user-agent="Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:40.0) Gecko/20100101 Firefox/40.0 referer" \
 "http://xx.yy.zz/something" -O downloaded_file

                while read c1
                do
                       $do_something$
                        done < list

When I ask shell to read the url from a file as shown below, it gives "http://: Invalid host name." error:
Code:
#!/bin/bash
while read COL1
do
wget --user-agent="Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:40.0) Gecko/20100101 Firefox/40.0 referer" \
 ""$COL1"" -O downloaded_file

                while read c1
                do
                       $do_something$
                        done < list
        done < url

My url file:
Code:
http://xx.yy.zz/something

Would you please redirect me? What am I doing wrong?

Thanks in advance
Boris
# 2  
Old 08-24-2018
Doubling up on double quotes means your string isn't quoted at all.
Code:
"$COL1"

# 3  
Old 08-24-2018
Also, I have no idea what's in your file. Anything wrong with that could cause problems.
# 4  
Old 08-25-2018
Dear Corona,
Single quote gives the same error.
when I wget from command line with url , it gives below file:

Code:
#EXTINF:0,RU: MIR
http://xx.yy.zz/aa/bb/13.ts
#EXTINF:0,RU: PIK
http://xx.yy.zz/aa/bb/21.ts
#EXTINF:0,RU: REN TV
http://xx.yy.zz/aa/bb/18.ts
#EXTINF:0,RU: RUSSIA 24
http://xx.yy.zz/aa/bb/44.ts
#EXTINF:0,RU: AAA
http://xx.yy.zz/aa/bb/55.ts

many thanks
Boris

------ Post updated 08-25-18 at 05:47 AM ------

Hello,
I renamed the file url to url.txt then, sorted out with curl


Code:
#!/bin/bash
for url in $(cat url.txt); do
  curl -A "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:40.0) Gecko/20100101 Firefox/40.0 referer" "$url" >> downloaded_file

                while read c1
                do
                       $do_something$
                        done < list
done

Kind regards
Boris
# 5  
Old 08-27-2018
" " are not single quotes. ' ' are single quotes.

Code:
for url in $(cat url.txt); do

This is a useless use of cat and dangerous and would throw up on many parts of your file.

Frankly I suspect little of your code is how you actually presented it by the fact any of it works at all.

Post your whole code and maybe we can talk. Until then we're only wild guessing.
# 6  
Old 08-27-2018
Hello Corona,
Thanks,
After reading your posts, I removed for url in field.
As I do not understand well single quote, double quotes and their differences, sometimes I stuck with such problems.
I have changed to "$COL1" now and wget works fine. As you asked me to share more info, I am just giving the latest and small part of the script. Of course, I will replace curl by wget

script:
Code:
#!/bin/bash
while read -r COL1 COL2; do
  curl -A "Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:40.0) Gecko/20100101 Firefox/40.0 referer" "$COL1" >> "$COL2".m3u
dos2unix "$COL2".m3u
        done < url.txt

cd /root/test2
while read -r COL1 ; do

sed -i '/EXTINF/s/.*/\U&/' $COL1.m3u
sed -i 's/EXTINF:-1/EXTINF:0/g' $COL1.m3u
sed -i 's/EXTINF:0,MTV/EXTINF:0,UK: MTV/g' $COL1.m3u
sed -i 's/MTV-HITsssss/MTV HITS/g' $COL1.m3u
..
..
done < /root/test2/friend

url.txt:
Code:
http://212.47.236.97:60000/get.php?username=unixcom&password=unixcom&type=m3u&output=mpegts          his_skype_id

friend
Code:
kostas
krzysztof
stephen



Many thanks
Boris
# 7  
Old 08-27-2018
A single quote is a single ' character. Strings inside single quotes are purely literal -- i.e. echo '$VARIABLE' prints $VARIABLE instead of the value of that variable.

A double-quote is a single " character. Variables, expressions, backslashes, etc get evaluated inside double-quotes, which is why you want them for "$COL".

Doubling up on any of them, i.e. '' or "", is a meaningless empty string. ""$VAR"" amounts to the same thing as not quoting $VAR at all.
This User Gave Thanks to Corona688 For This Post:
baris35
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Wget - working in browser but cannot download from wget

Hi, I need to download a zip file from my the below US govt link. https://www.sam.gov/SAMPortal/extractfiledownload?role=WW&version=SAM&filename=SAM_PUBLIC_MONTHLY_20160207.ZIP I only have wget utility installed on the server. When I use the below command, I am getting error 403... (2 Replies)
Discussion started by: Prasannag87
2 Replies

2. Homework & Coursework Questions

If and Loops

Use and complete the template provided. The entire template must be completed. If you don't, your post may be deleted! 1. The problem statement, all variables and given/known data: In this script you will take a directory as input from the user and change the end of line sequence from a Unix... (1 Reply)
Discussion started by: Pcarson
1 Replies

3. Shell Programming and Scripting

Wget vs Curl - Proxy issue

Hi, My script needs to crawl the data from a third party site. Currently it is written in wget. The third party site is of shared interface with different IP addresses. My wget works with all the IP address but not with one. Whereas the curl is able to hit that IP address and comes out... (2 Replies)
Discussion started by: sathyaonnuix
2 Replies

4. Shell Programming and Scripting

korn shell for loops with expect issue

Hi I have the following Korn script having multiple for loops. #!/bin/ksh EXPECT=/usr/local/bin/expect exp_internal for d in 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 19 20 21 i22 23 24 25 26; do for i in 01 02 03 04 05 06 07 ; do for h in 00 01 02 03 04 05 06 07 08 09 10 11 12... (2 Replies)
Discussion started by: cic
2 Replies

5. Shell Programming and Scripting

Too many for loops

My apologies for the long post.... I have a structure problem with my ksh script on linux. right now it does what i want (which i know is the point) but its really really ugly and i know there must be a simpler way to achieve my task. The data is a list of usernames. This list will help... (2 Replies)
Discussion started by: maverick72
2 Replies

6. UNIX for Dummies Questions & Answers

Help with for loops

Hi, I am starting to enhance my scripting knowledge and need some assistance with simple 1 line for loops. This may help to do a mass permissions change on a big apache doc root while I have an insane customer on my phone. What is the best resource to learn this skill and what are some that... (5 Replies)
Discussion started by: Oshie74
5 Replies

7. UNIX for Dummies Questions & Answers

loops with tr

Hello, I'm not sure if this is more appropriate for the 'unix for dummies' or the 'unix for experts' forum because I'm new to this forum and this is the second topic I've discussed, but if you could let me know which one was more appropriate for something like this, please do! So in tr (an... (2 Replies)
Discussion started by: juliette salexa
2 Replies

8. Shell Programming and Scripting

Wget issue

Hello there, I'm trying to download a dynamically generated page with Wget, but it downloads the wrong contents. The page link is below(it's a sitemap). I need this to work to set a Cron job that downloads it automatically every hour. ... (1 Reply)
Discussion started by: rlopes
1 Replies

9. Shell Programming and Scripting

while loops

Hi I've a file like so: Now, I want to read my file and take ex. the Media ID and the Type for each groups of Media (Media1,Media2,...,Media(n): cat /tmp/file|\ while read FILE do while $(FILE|cut -d: -f1)=Media$i do #here will be some test, ex: #if Media ID < 23 ... (4 Replies)
Discussion started by: nymus7
4 Replies

10. UNIX for Dummies Questions & Answers

loops?

hello....very new user to unix...and i have a question..i am not sure if there is such a thing For example...the user is asked if he likes Bananas....if he says yes.... echo You like Bananas $name at the end of the script it echos all that the user has entered so they can read it.... but... (1 Reply)
Discussion started by: jonas27
1 Replies
Login or Register to Ask a Question