Sponsored Content
Top Forums Shell Programming and Scripting Curl parallel download file list Post 302998235 by Chubler_XL on Saturday 27th of May 2017 01:44:46 PM
Old 05-27-2017
export -f is a bash feature and I use it here to insure the internal function fetch_url is exported to sub shells. This is needed as xargs is an external command to the shell and runs the assembled commands in a new shells.

I assumed, as you were using GNU xargs (-P feature is a GNU extension), that you were also using the bash shell. I've updated my original post to specify the required shell, and this is all you may need to do to get you version working.

However, if you do not wish to use bash, you could put your function in an external script so that it can be called from xargs for example:

$HOME/bin/fetch_url:
Code:
#!/bin/sh
URL=$@
filename=${URL##*=}
curl -r 0-50000 -L "$URL" -o ${filename}.html -a ${filename}.log

And from another script (or the command line) you can call this with:
Code:
xargs -n 1 -P 10 $HOME/bin/fetch_url < url.list


Last edited by Chubler_XL; 05-27-2017 at 02:54 PM..
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

cURL Active FTP Download

Hello, I know this is probably a very silly question for most but how to do I force curl to do active FTP downloads? Thank you Dallas (2 Replies)
Discussion started by: Dallasbr
2 Replies

2. Shell Programming and Scripting

Download using curl and redirect question_please help

Basically I am needing to Download (using curl) in the background some data from(link here), with stderr redirected to /dev/null, to a file named taxcode I was doing this, curl & http:// name here/download/pls/Title_26.txt 2> /dev/null > taxcode but the results were not what I was after. ... (1 Reply)
Discussion started by: santod
1 Replies

3. UNIX for Advanced & Expert Users

Help with using curl to download files from https

Hi I'm trying to download an xml file from a https server using curl on a Linux machine with Ubuntu 10.4.2 I am able to connect to the remote server with my username and password but the output is only "Virtual user <username> logged in". I am expecting to download the xml file. My output... (4 Replies)
Discussion started by: henryN
4 Replies

4. Shell Programming and Scripting

How to download file without curl and wget

Hi I need a Shell script that will download a zip file every second from a http server but i can't use neither curl nor wget. Can anyone will help me go about this task ??? Thanks!! (1 Reply)
Discussion started by: rubber08
1 Replies

5. Shell Programming and Scripting

Curl download zip extract large xml file

Hi i have a php script that works 100% however i don't want this to run on php because of server limits etc. Ideally if i could convert this simple php script to a shell script i can set it up to run on a cron. My mac server has curl on it. So i am assuming i should be using this to download the... (3 Replies)
Discussion started by: timgolding
3 Replies

6. Shell Programming and Scripting

Curl ftp ssl download files

Hello all, I have been struggling with this issue on and off for a couple of weeks now and I just got it all working, so I wanted to share my findings in case some other poor soul needs to know how. First some background on what I'm doing. I am uploading files to different directories based on... (0 Replies)
Discussion started by: msjkadams
0 Replies

7. Shell Programming and Scripting

Curl command to download multiple files with a file prefix

I am using the below curl command to download a single file from client server and it is working as expected curl --ftp-ssl -k -u ${USER}:${PASSWD} ftp://${HOST}:${PORT}/path/to/${FILE} --output ${DEST}/${FILE} let say the client has 3 files hellofile.101, hellofile.102, hellofile.103 and I... (3 Replies)
Discussion started by: r@v!7*7@
3 Replies

8. UNIX for Beginners Questions & Answers

Curl to download using Linux not working

Good Morning. I'm trying to download a file from a server. I was able to upload to the server successfully but when i download, i see the file name in my server but with some unknow data. The file name i'm trying to download is abcd.zip in binary mode. curl -1 -v --ftp-pasv -o abcd.zip -u... (4 Replies)
Discussion started by: Pavan Kumar19
4 Replies

9. Shell Programming and Scripting

Curl to download file from subdivx.com after following location without knowing the file name/extens

This question could be specific to the site subdivx.com In the past, I've been able to download a file following location using cURL but there is something about subdivx.com that's different and can't figure out how to get it to work. I tried the following directly in the terminal with no... (5 Replies)
Discussion started by: MoonD
5 Replies

10. Shell Programming and Scripting

Curl , download file with user:pass in bash script

Hello, My question is about curl command. (ubuntu14.04) In terminal, I am able to download my mainfile with: curl -u user1:pass1 http://11.22.33.44/******* When I convert it into bash script like this: #!/bin/bash cd /root/scripts computer_ip=11.22.33.44 curl -u $1:$2... (8 Replies)
Discussion started by: baris35
8 Replies
NZBGET(1)							   User Commands							 NZBGET(1)

NAME
nzbget - binary news file grabber DESCRIPTION
Usage: nzbget [switches] Switches: -h, --help Print this help-message -v, --version Print version and exit -c, --configfile <file> Filename of configuration-file -n, --noconfigfile Prevent loading of configuration-file (required options must be passed with --option) -p, --printconfig Print configuration and exit -o, --option <name=value> Set or override option in configuration-file -s, --server Start nzbget as a server in console-mode -D, --daemon Start nzbget as a server in daemon-mode -V, --serverversion Print server's version and exit -Q, --quit Shutdown server -A, --append <nzb-file> Send file to server's download queue -C, --connect Attach client to server -L, --list [F|G|O|S|H] Request list of downloads from server F list individual files and server status (default) G list groups (nzb-files) and server status O list post-processor-queue H list history S print only server status -P, --pause [D|D2|O|S] Pause server: D pause download queue (default) D2 pause download queue via second pause-register O pause post-processor queue S pause scan of incoming nzb-directory -U, --unpause [D|D2|O|S] Unpause server: D unpause download queue (default) D2 unpause download queue via second pause-register O unpause post-processor queue S unpause scan of incoming nzb-directory -R, --rate <speed> Set download rate on server, in KB/s -T, --top Add file to the top (beginning) of queue (for using with switch --append) -K, --category <name> Assign category to nzb-file (for using with switch --append) -G, --log <lines> Request last <lines> lines from server's screen-log -W, --write <D|I|W|E|G> "Text" Send text to server's log -S, --scan Scan incoming nzb-directory on server -E, --edit [G|O|H] <action> <IDs> Edit items on server G Affect all files in the group (same nzb-file) O Edit post-processor-queue H Edit history <action> is one of: <+offset|-offset> Move file(s)/group(s)/post-job in queue relative to current position, offset is an integer value T Move file(s)/group(s)/post-job to top of queue B Move file(s)/group(s)/post-job to bottom of queue P Pause file(s)/group(s)/ Postprocess history-item(s) again U Resume (unpause) file(s)/group(s) A Pause all pars (for groups) R Pause extra pars (for groups)/ Return history-item(s) back to download queue D Delete file(s)/group(s)/post-job(s)/history-item(s) K <name> Set category (for groups) N <name> Rename (for groups) M Merge (for groups) O <name>=<value> Set post-process parameter (for groups) I <priority> Set priority (signed integer) for file(s)/group(s) <IDs> Comma-separated list of file-ids or ranges of file-ids, e. g.: 1-5,3,10-22 nzbget version: 0.7.0 May 2011 NZBGET(1)
All times are GMT -4. The time now is 04:50 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy