How to download Images and Json file from server(godaddy) to Local machine (Ubuntu 14.04).?

Login or Register to Ask a Question and Join Our Community

How to download Images and Json file from server(godaddy) to Local machine (Ubuntu 14.04).?

Tags

Login to Discuss or Reply to this Discussion in Our Community

Top Forums Shell Programming and Scripting How to download Images and Json file from server(godaddy) to Local machine (Ubuntu 14.04).?

01-06-2016

Registered User

8, 0

Join Date: Dec 2015

Last Activity: 13 January 2016, 12:36 AM EST

Posts: 8

Thanks Given: 4

Thanked 0 Times in 0 Posts

How to download Images and Json file from server(godaddy) to Local machine (Ubuntu 14.04).?

Hi Guys,

Just entering the Linux word, So I need help to write a script on my local machine(Ubuntu 14.04) that continuously check the particular folder(contains images) and a json file on the server and download whenever new images are added to that folder and whenever there is a change in the Json file.

I prefer to use wget.

Thanks,

g4v1n

View Public Profile for g4v1n

Find all posts by g4v1n

01-08-2016

Moderator

1,484, 567

Join Date: Mar 2011

Last Activity: 28 November 2020, 9:34 AM EST

Posts: 1,484

Thanks Given: 68

Thanked 567 Times in 444 Posts

Something like :

Code:

#!/usr/bin/ksh
LOCKFILE=/path/to/lock/wget.lock
LOGFILE=/path/to/log/wget.log
DDIR=/path/to/destination
if [ -f $LOCKFILE ]; then
printf "%s\n" "Lock file exists on date $(date "+%Y%m%d_%H%M")" >> $LOGFILE
exit 127
fi
cd $DDIR && touch $LOCKFILE || exit 127
wget -m -c -nH --output-file=$LOGFILE --timeout=3 --tries=3 ftp://user:pass@yourftpsite/* && rm $LOCKFILE || exit 127
cd -

This will mirror all the files from ftp site (../*) to your $DDIR.
It will not mirror if the remote file is not newer then then local file.

Also the $LOGFILE will be overwritten upon each execution of wget, which can be changed by adding date command to log name.

Hope that helps
Regards
Peasant.

This User Gave Thanks to Peasant For This Post:

Peasant

View Public Profile for Peasant

Find all posts by Peasant

01-08-2016

Registered User

8, 0

Join Date: Dec 2015

Last Activity: 13 January 2016, 12:36 AM EST

Posts: 8

Thanks Given: 4

Thanked 0 Times in 0 Posts

Thanks a lot! Peasant.
I would try this one today and let you how it goes.
Much appreciated

---------- Post updated at 06:45 PM ---------- Previous update was at 09:38 AM ----------

It works very well.
But the text file that I am trying to download from ftp server to my local machine is not exact same as on server( I mean some words from the files are missing). Is there any way I can put checksum to check that files are not corrupted?

---------- Post updated at 09:59 PM ---------- Previous update was at 06:45 PM ----------

could you please tell me what is $LOCKFILE and why do we need this?

g4v1n

View Public Profile for g4v1n

Find all posts by g4v1n

01-09-2016

Moderator

1,484, 567

Join Date: Mar 2011

Last Activity: 28 November 2020, 9:34 AM EST

Posts: 1,484

Thanks Given: 68

Thanked 567 Times in 444 Posts

Lockfile is used so the script will not run multiple times, for instance from crontab.
IF you have this script in cron to run every 5 minutes, and the download lasts 7 minutes lockfile will prevent it from running during download (unwanted in any case).

As for checksuming, unfortunately wget cannot do this, you will have to do this manually or switch to using ssh with rsync (which has checksuming builtin).

If you could somehow generate a md5sum of all files named for instance mysum.txt on the server you are fetching the data from and transfer that to your server you could run a check from the script after download like :

Code:

cd $DDIR && md5sum -c mysum.txt

Have you tried to fetch those file manually to confirm that the same thing is happening with any FTP client ?

Hope that helps
Regards
Peasant.

This User Gave Thanks to Peasant For This Post:

Peasant

View Public Profile for Peasant

Find all posts by Peasant

01-09-2016

Registered User

8, 0

Join Date: Dec 2015

Last Activity: 13 January 2016, 12:36 AM EST

Posts: 8

Thanks Given: 4

Thanked 0 Times in 0 Posts

I dont wanna use cron job, instead Im trying to use while statement in script itself so in that case where should I put following statements, inside the while loop or outside?

Code:

if [ -f $LOCKFILE ]; then printf "%s\n" "Lock file exists on date $(date "+%Y%m%d_%H%M")" >> $LOGFILE exit 127 fi

---------- Post updated at 01:24 AM ---------- Previous update was at 01:14 AM ----------

Also for security reasons, I do not want to write my ftp username and password in the script. Is there any way that I can fetch username and password from some other file or script in encrypted form?

g4v1n

View Public Profile for g4v1n

Find all posts by g4v1n

01-09-2016

Moderator

1,484, 567

Join Date: Mar 2011

Last Activity: 28 November 2020, 9:34 AM EST

Posts: 1,484

Thanks Given: 68

Thanked 567 Times in 444 Posts

Can you post your entire code if possible ?
Without the && || syntax the code would be longer.

Code:

cd $DDIR
if [ $? -eq 0 ]; then # i have successfully changed directory to $DDIR
touch $LOCKFILE
wget <options>
      if [ $? -eq 0 ]; then # if wget has successfully finished
      rm $LOCKFILE
      else
      exit 127 # wget has failed, i will not exit gracefully, but with error (LOCKFILE remains)
      fi
else # i wasn't able to switch to $DDIR so i will exit with error 127.
exit 127
fi

When i look exit 127 might not be good choice for exit code, so change this to exit 1. Sorry for that.

As for security, using FTP is insecure by default. This is why protocols such as SFTP/SSH or FTPS (ftp over ssl with certificates) were established.

You will have hard time securing FTP (if possible at all), also FTP would be suspectable to man in the middle attacks due to passwords/users traveling over network unencrypted.
A simple tcpdump on your local network would reveal user and passwords...

Can you use ssh and rsync with keys ?

Regards
Peasant.

Peasant

View Public Profile for Peasant

Find all posts by Peasant

01-09-2016

Registered User

8, 0

Join Date: Dec 2015

Last Activity: 13 January 2016, 12:36 AM EST

Posts: 8

Thanks Given: 4

Thanked 0 Times in 0 Posts

Code:

#!/bin/sh
LOCKFILE=/home/gav/Desktop/wgetTest/lock/wget.lock
LOGFILE=/home/gav/Desktop/wgetTest/log/wget.log
DDIR=/home/gav/Desktop/wgetTest/
JSONDIR=/home/gav/Desktop/wgetTest/text/new.json
IMAGEDIR=/home/gav/Desktop/wgetTest/display/

if [ -f $LOCKFILE ]; then
printf "%s\n" "Lock file exists on date $(date "+%Y%m%d_%H%M")" >> $LOGFILE
exit 127
fi

    cd $DDIR && touch $LOCKFILE || exit 127
    wget -m -c -nH --cut-dirs=2 --output-file=$LOGFILE --timeout=3 --tries=3 --passive-ftp ftp://usr:pwd@mywesite.com/www/test/images/* && rm $LOCKFILE || exit 127
    mv images/* $IMAGEDIR

    cd $DDIR && touch $LOCKFILE || exit 127
    wget -m -c -nH --cut-dirs=2 --output-file=$LOGFILE --timeout=3 --tries=3 --passive-ftp ftp://usr:pwd@mywesite.com/www/test/display.json && rm $LOCKFILE || exit 127
    mv display.json $JSONDIR

sleep 60

    cd $DDIR && touch $LOCKFILE 

while true;do
    
    wget -m -c -nH --cut-dirs=2 --output-file=$LOGFILE --timeout=3 --tries=3 --passive-ftp ftp://usr:pwd@mywesite.com/www/test/display.json && rm $LOCKFILE

    if cmp -s display.json $JSONDIR; then
        sleep 15
        echo "same files"
    else
                echo "different files and downloading new files"
        wget -m -c -nH --cut-dirs=2 --output-file=$LOGFILE --timeout=3 --tries=3 --passive-ftp ftp://usr:pwd@mywesite.com/www/test/images/* && rm $LOCKFILE
        mv images/* $IMAGEDIR
        mv display.text $JSONDIR
    fi
sleep 60
done

---------- Post updated at 02:22 AM ---------- Previous update was at 02:19 AM ----------

Sorry, I am new to linux and just trying to create a logic and I write many statements multiple time in my code :/

g4v1n

View Public Profile for g4v1n

Find all posts by g4v1n

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Red Hat

iptables applied in local machine, can't ssh remote machine after chain changed to DROP

I want to SSH to 192.168.1.15 Server from my machine, my ip was 192.168.1.99 Source Destination was UP, with IP 192.168.1.15. This is LAN Network there are 30 Machine's Connected to the network and working fine, I'm Playing around the local machine's because I need to apply the same rules in...

2. UNIX for Dummies Questions & Answers

Copy files from Linux server local windows machine using a shell script

Hello, I need to create a shell script which will copy files - which are created on particular date and starting with particular name - to local windows XP machine. Is this possible.? Currently it is being done manually using winscp

3. UNIX for Advanced & Expert Users

FTP While transfering files to local machine to remote machine

Hi Am using unix Ksh Am getting the problem while transferring zero size files through the script . When i transfer zero size files from local machine to remote machine manually i can able to do it . My question its beause of zero size files am not able to transfer through script ? or its...

4. Solaris

Copying from server to a local machine

Hi, I am trying to copy a file from oracle home on a unix platform to my local machine. I think i can use ftp but i am not sure of the exact syntax and way i should write this. Can someone help please?

5. IP Networking

How to create Local IPv6 and IPv4 networks by using ubuntu server as router

Hi, I have two Cent OS Linux systems and one Ubuntu OS system contains two Ethernet Cards. I need to create two networks by using Ubuntu sys as router. One network need to configure in IPv4 addresses and another network should configure in IPv6 addresses. Please help me how to configure two...

6. Shell Programming and Scripting

How to transfer files from unix machine to local machine using shell script?

Hi All.. Am new to Unix!! Am creating a shell script in which a scenario is like i have transfer the output file from unix machine (Server) to local directory (Windows xp). And also i have to transfer the input file from the local directory to Unix machine (Server) Any help from you...

7. Shell Programming and Scripting

check web server running on local and on remote machine

Hi , How to check whether web server is running from remote machine How to check whether web server is running on web server itself Can any one help me soon

8. Red Hat

To find the LATEST file from a dir on REMOTE machine and SCP to local machine?

Hi All, URGENT - Please help me form a scipt for this: I need the LATEST file from a dir on REMOTE machine to be SCP'd to a dir on local machine. (and I need to execute this from local server) I know that the below cmd is used to find the LATEST file from a dir. But this command is not...

9. UNIX for Dummies Questions & Answers

Copy file from local machine to server?

I am logged into a server via SSH. There is a file on my desktop of my Windows PC I would like to put on that server. How do I do it? SCP looks the likely method but how does the server know where the file is on the local machine? Thanks.

10. UNIX for Advanced & Expert Users

Using SCP command in IBM AIX to download file from remote to local system

Hi, When i run the code in solaris unix machine, the file from remote server is getting downloaded. but when i use the same code in IBM AIX remote machine, it is not running. It is saying "Erro during scp transfer." Below is the code. Please give some resolution. SCPClient client = new...

Login or Register to Ask a Question