Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Csv download in a bash script

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Csv download in a bash script
# 1  
Old 06-17-2016
Csv download in a bash script
I am attempting to download a url in csv format. When I download this url in a browser excel opens up and automatically populates with comma separated values.

When I try to use curl or wget I get nothing or garbage.

This on the command line just hangs

wget -b http://url.net/render?target=collect...min&format=csv



http://url.net/render?target=collect...min&format=csv
Code:
#!/bin/bash

while true; do 
wget -b http://url.net/render?target=collectd.network.*.*.*.*.snmp.*Memory*&from=-10min&until=-5min&format=csv

{
                read url
                read time
                read memory
        } < $I
        echo "$url,$time,memory" >> outputfile.csv

done

This code above is my attempt to place this data in a file.
# 2  
Old 06-17-2016
To BASH, & means "put this program in the background". So you probably put a ton of teeny nonsense commands in the background which you will be spammed with when you hit ENTER again.

Put the URL in double quotes to make it not do that.

Code:
wget "http://long.url/with?=stuff&in&it"

# 3  
Old 06-18-2016
Where and how do you "get nothing or garbage"? Without the -O option, wget will put the downloaded contents into an appropriately named file according to the URL. Did you look for that?

What be the contents of the I variable? It is not set in your snippet, and thus you should receive an error msg like
Code:
bash: $I: ambiguous redirect

, and the outputfile.csv will be empty.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Curl , download file with user:pass in bash script

Hello, My question is about curl command. (ubuntu14.04) In terminal, I am able to download my mainfile with: curl -u user1:pass1 http://11.22.33.44/******* When I convert it into bash script like this: #!/bin/bash cd /root/scripts computer_ip=11.22.33.44 curl -u $1:$2... (8 Replies)
Discussion started by: baris35
8 Replies

2. Shell Programming and Scripting

Bash to tell download where specific files are stored

The bash below will download all the files in download to /home/Desktop/folder. That works great, but within /home/Desktop/folder there are several folders bam, other, and vcf, is there a way to specify by extention in the download file where to download it to? For example, all .pdf and .zip... (2 Replies)
Discussion started by: cmccabe
2 Replies

3. Ubuntu

Bash script for FTP download -Mysql

Hi guys, I recently managed to write up my working script, but now I have a problem. If the file isn't there in the remote server, my actual script jumps it and all ok, but I need something like this: Search file -> if there, then download -> if not, download next file in the list. Any... (7 Replies)
Discussion started by: virtus96
7 Replies

4. Shell Programming and Scripting

Bash to download specific files and save in two folders

I am trying to download all files from a user authentication, password protected https site, with a particular extension (.bam). The files are ~20GB each and I am not sure if the below is the best way to do it. I am also not sure how to direct the downloaded files to a folder as well as external... (7 Replies)
Discussion started by: cmccabe
7 Replies

5. Shell Programming and Scripting

Bash script help - removing certain rows from .csv file

Hello Everyone, I am trying to find a way to take a .csv file with 7 columns and a ton of rows (over 600,000) and remove the entire row if the cell in forth column is blank. Just to give you a little background on why I am doing this (just in case there is an easier way), I am pulling... (3 Replies)
Discussion started by: MrTuxor
3 Replies

6. Shell Programming and Scripting

BASH script to parse XML and generate CSV

Hi All, Hope all you are doing good! Need your help. I have an XML file which needs to be converted CSV file. I am not an expert of awk/sed so your help is highly appreciated!! XML file looks like this: <l:event dateTime="2013-03-13 07:15:54.713" layerName="OSB" processName="ABC"... (2 Replies)
Discussion started by: bhaskar_m
2 Replies

7. Shell Programming and Scripting

Bash script to reorder csv

hi guys, im fairly new to unix and bash scripts and therefore your help would really be appreciated. i need to write a bash script that will take a csv file, and reorder the data and output to another csv file. The source csv file will look something like this: HEAD,671061,Add,SS... (3 Replies)
Discussion started by: daz_20
3 Replies

8. UNIX for Advanced & Expert Users

Unix Bash: substitute columns in .csv using other .csv columns

Hi All, I have two .csv's input.csv having values as (7 columns) ABC,A19907103,ABC DEV YUNG,2.17,1000,2157,07/07/2006 XYZ,H00213850,MM TRUP HILL,38.38,580,23308,31/08/2010 output.csv having (25 columns) A,B,C,D,E,F,G,H,I,J,K,L,M,N,O,P,Q,R,S,T,U,V,W,X,Y... (4 Replies)
Discussion started by: abhivyas
4 Replies

9. Shell Programming and Scripting

Need to modify csv-file with bash script

Hi Guys, I need to write a script, that exports the "moz_places" table of the "places.sqlite"-file (firefox browser history) into a csv-file. That part works. After the export, my csv looks like this: ... 4429;http://www.sqlite.org/sqlite.html;"Command Line Shell For... (11 Replies)
Discussion started by: Sebi0815
11 Replies

10. Shell Programming and Scripting

Automatic download of csv file using a shell script

I want to automatically download a CSV file daily which can be found here: http://www.londonstockexchange.com/en-gb/pricesnews/prices/coveredwarrants/search.htm and the link is named "Click to download covered warrants (100Kb)" onthe right hand side. What commands can I use to invoke clicking... (1 Reply)
Discussion started by: figaro
1 Replies
Login or Register to Ask a Question