Sponsored Content
Top Forums Shell Programming and Scripting sftp mget where file doesn't exist BASH Post 302294580 by unclecameron on Thursday 5th of March 2009 12:30:37 PM
Old 03-05-2009
well I wound up writing one, I know it's probably a terrific hack, but here it is in case it helps someone else, this took me awhile to figure out

Code:
#!/bin/bash

# pipe ls of remote folder to file
# to compare to figure out which files
# are newer
b=`sftp user@ftp.domain.com <<EOF
cd folder_with_hourly_dataset
ls
quit
EOF`
echo $b > remotelist.txt

# sanitize that file, so that all we
# see are the .csv entries
list=$(cat remotelist.txt)
for i in $list
do
    echo $i | grep \.csv$ >> sortedremotelist.txt
done

# create local list of files
ls *.csv >> locallist.txt

# compare local/remote list and then
# sanitize the output file into a file
# with one .csv per line so we can 
# use a for loop to sftp get it
diff locallist.txt sortedremotelist.txt > get
cut -c3- get > get2
grep \.csv get2 > get3

# now go get that list of files
c=$(cat get3)
for i in $c
do
    d=`user@domain.com <<EOF
    cd directory_with_hourly_data
    mget $i
    quit
    EOF`
    echo "-------------"
    echo "getting file: "$i
done

# clean up files for next time
cat /dev/null > remotelist.txt
cat /dev/null > sortedremotelist.txt
cat /dev/null > get
cat /dev/null > get2
cat /dev/null > get3

hope that helps someone else. Of course I could've done the whole thing with a single rsync -auv -e ssh user@example.com:/folder_with_updates/ ./ but I couldn't get a system user account on the box to do it Smilie
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

What hapens if a group member doesn't exist?

As part of a NIS implementation, (I think) I want to create a group (in /etc/group) that has users that do not exist on the target machine. What effect will this have? Will it cause any problems? Thanks, Gary Cooper (1 Reply)
Discussion started by: Gary Cooper
1 Replies

2. Shell Programming and Scripting

Perl: One action if an element doesn't exist in array

Hello, I want to run one (not multiple) action if an element doesn't exist in array. for example: @array = (1..10); foreach $el (@array) { if ($el != 11) { print "number not found\n"; } } the output of this simple script: number not found (3 Replies)
Discussion started by: ahmed_zaher
3 Replies

3. Programming

Kernel module - How to test if file doesn't exist

Hey, I'm currently getting into some kernel module progamming. As a little exercise I want to read the headers out of an ELF file. My code is very simple, here is the important part: struct file *fp; /* ... */ fp = filp_open("some/file/on/my/pc", O_RDONLY, 0); if(fp == NULL) { ... (15 Replies)
Discussion started by: disaster
15 Replies

4. Shell Programming and Scripting

ln -s creates symlink in symlink, if [ -f ... ] says file that exists doesn't exist

Hi Forums, I got a little problem, I made a few modifications to the code of the launch script of a testing server(minecraft) and now updating is broken aswell as the automatic directory creation. These Lines somehow create an endless symlink that refers to itself and I don't know how to fix... (0 Replies)
Discussion started by: Xaymar
0 Replies

5. Solaris

User directory doesn't exist

Hii all, i create the user useradd -d /home/kk kk passwd kk when i tried to login to kk i get a error user directory doesn't exist then i tried useradd kkk passwd kkkwhen i tried to login to kkk i get the same error user directory doesn't exist. (4 Replies)
Discussion started by: vipinkumarr89
4 Replies

6. Red Hat

Removing LVM Volume Group that doesn't exist anymore

Our SAN administrator decided to unpresent then destroy LUN's we were actively using as a volume group (all PV's in said volume group). Now every time I do a pvscan or whatever it complains about I/O errors trying to access those PV's. How do I get it to forget the VG existed completely? vgreduce... (7 Replies)
Discussion started by: thmnetwork
7 Replies

7. Shell Programming and Scripting

Need to generate a file with random data. /dev/[u]random doesn't exist.

Need to use dd to generate a large file from a sample file of random data. This is because I don't have /dev/urandom. I create a named pipe then: dd if=mynamed.fifo do=myfile.fifo bs=1024 count=1024 but when I cat a file to the fifo that's 1024 random bytes: cat randomfile.txt >... (7 Replies)
Discussion started by: Devyn
7 Replies

8. Homework & Coursework Questions

Group Doesn't Exist

Use and complete the template provided. The entire template must be completed. If you don't, your post may be deleted! 1. The problem statement, all variables and given/known data: I'm able to create a group but when I'm trying to delete the group it keeps stating Group Doesn't Exist. I know... (2 Replies)
Discussion started by: GoBoyGo
2 Replies

9. Shell Programming and Scripting

Mget with SFTP is not working

hi Team, I am connecting from one (A) linux server to another(C)/any linux server by sftp on A linux server: sftp userid@C password: mget is Not working fine I am using mget to pull the files. it shows mget as invalid command. But from (B) Linux server to (C) /to Any server Linux... (15 Replies)
Discussion started by: johnsnow
15 Replies

10. Shell Programming and Scripting

Sftp mget file search with casensitive

Hi , we are not able to get file name is different casesensitive in sftp server. for example, if the files are any case that means upper or lower case ,i need to get the filenames using the mget command in sfp. sftp>mget Test.txt Thanks (5 Replies)
Discussion started by: bmk123
5 Replies
sftp-server(1M) 					  System Administration Commands					   sftp-server(1M)

NAME
sftp-server - SFTP server subsystem SYNOPSIS
/usr/lib/ssh/sftp-server DESCRIPTION
sftp-server implements the server side of the SSH File Transfer Protocol as defined in the IETF draft-ietf-secsh-filexfer. sftp-server is a subsystem for sshd(1M) and must not be run directly. There are no options or config settings. To enable the sftp-server subsystem for sshd add the following to /etc/ssh/sshd_config: Subsystem sftp /usr/lib/ssh/sftp-server See sshd_config(4) for a description of the format and contents of that file. There is no relationship between the protocol used by sftp-server and the FTP protocol (RFC 959) provided by in.ftpd. EXIT STATUS
The following exit values are returned: 0 Successful completion. >0 An error occurred. FILES
/usr/lib/sftp-server ATTRIBUTES
See attributes(5) for descriptions of the following attributes: +-----------------------------+-----------------------------+ | ATTRIBUTE TYPE | ATTRIBUTE VALUE | +-----------------------------+-----------------------------+ |Availability |SUNWsshdu | +-----------------------------+-----------------------------+ |Interface Stability |Evolving | +-----------------------------+-----------------------------+ SEE ALSO
sftp(1), ssh(1), ssh-add(1), ssh-keygen(1), sshd(1M), sshd_config(4), attributes(5) To view license terms, attribution, and copyright for OpenSSH, the default path is /var/sadm/pkg/SUNWsshdr/install/copyright. If the Solaris operating environment has been installed anywhere other than the default, modify the given path to access the file at the installed location. AUTHOR
Markus Friedl SunOS 5.10 30 Jul 2003 sftp-server(1M)
All times are GMT -4. The time now is 01:03 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy