Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Making a script to copy files not seen before (using md5sum)

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Making a script to copy files not seen before (using md5sum)
# 1  
Old 08-29-2013
[Solved] Making a script to copy files not seen before (using md5sum)
Hello,

I would like to make a script that searches through a SRC folder and copies only files it's never seen before to a DEST folder.

SRC = /user/.phonesync/photos-backup
DST = /usr/.phonesync/photos-new

So basically, I'd start with a:

Code:
md5sum /user/.phonesync/photos-backup/* > /user/.phonesync/photos-backup.md5

Then, I'd like to run some sort of "cp" or "rsync" from SRC to DEST after comparing the current contents of /photos-backup/ to the .md5 list. If a file is not in the MD5 list, then copy it to /photos-new/ and then add it to the photos-backup.md5. The md5 list will always get appended to and grow and grow, but I'll deal with that at some point by deleting it and generating a new md5 list.
Last edited by nbsparks; 08-29-2013 at 05:09 PM..
# 2  
Old 08-29-2013
You can check if the file name is in your 'md5' file and if not copy it:
Code:
for f in $SRC/*
do
  grep -q $f md5_files.txt
  if [[ $? -ne 0 ]]; then
    cp $SRC/$f $DST
  fi
done

This User Gave Thanks to spacebar For This Post:
nbsparks
# 3  
Old 08-29-2013
So I'm thinking more as I go... I haven't test this yet, but it this a valid modification?

Code:
SRC=/user/nick/.phonesync/photos-backup
DST=/user/nick/.phonesync/photos-new
MD5=/user/nick/.phonesync/photos-backup.md5

for f in $SRC/*
do
  grep -q $f $MD5
  if [[ $? -ne 0 ]]; then
    cp $SRC/$f $DST
  fi
done

Then the one thing I would want to add to this is to do an append of the md5sum of each file copied. So maybe having a "md5sum $f >>$MD5" before the close of the if statement. Does that sound like it would work? If so, how do I properly include that in the if? (Yes, I'm very new to this.)

Thanks!
# 4  
Old 08-29-2013
You seem to have it:
Code:
for f in $SRC/*
do
  grep -q $f $MD5
  if [[ $? -ne 0 ]]; then
    cp $SRC/$f $DST
    md5sum $f >> $MD5
  fi
done

This User Gave Thanks to spacebar For This Post:
nbsparks
# 5  
Old 08-29-2013
OK... I have to go home and test this, but if I wanted to be able to run a command like this...

./copy_new_photos.sh nick

and that would look for the user "nick" for new photos and then copy them, then I would script it like this...

Code:
#!/bin/bash

USER=$1
SRC=/user/$USER/.phonesync/photos-backup
DST=/user/$USER/.phonesync/photos-new
MD5=/user/$USER/.phonesync/photos-backup.md5

for f in $SRC/*
do
  grep -q $f $MD5
  if [[ $? -ne 0 ]]; then
    cp $SRC/$f $DST
    md5sum $f >> $MD5
  fi
done

Did I get it right?
# 6  
Old 08-30-2013
Have you tried rsync?

Code:
rsync -a "${SRC}"/ "${DST}"/

# 7  
Old 08-30-2013
As far as I see, you only copy files that are not on the md5sum list. I do not see you are doing any md5sum check against the files though.

Actually you could just use --ignore-existing of rsync if you want to ignore the files that both exist in the source and destination directories.

Code:
rsync -avv --ignore-existing src/ des/

If you still want to keep track of that list and have the command that does the copying and excluding, you could use --exclude-from={PATH_TO_YOUR_LIST} of rsync
Last edited by MR.bean; 08-30-2013 at 04:27 AM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

[md5sum] script

I am getting No such file or directory if my variable contains white spaces... Is there a way to fix this? This works x="1.md5" md5sum -c "$x" This, does not x="23\ 5\ 6\ 7\ 8\ 9\ 10.md5" md5sum -c "$x" md5sum: '23\ 5\ 6\ 7\ 8\ 9\ 10.md5': No such file or directory How do I fix... (1 Reply)
Discussion started by: soichiro
1 Replies

2. Shell Programming and Scripting

Need script for making files based on some conditions.

Hi All, I have a text file (code_data.txt) with the followig data. AMAR AB123456 XYZ KIRAN CB789 ABC RAJ CS78890 XYZ KAMESH A33535335 ABC KUMAR MD678894 MAT RITESH SR3535355... (26 Replies)
Discussion started by: ROCK_PLSQL
26 Replies

3. Shell Programming and Scripting

Compare files in directories with md5sum

And not to start. I can compare files, that's easy. The problem is that I compare files in a directory, and check if these files exist in another directory. The problem is that the file names are not the same. So I have to compare with "md5sum" or something similar. How I can do? All this in... (7 Replies)
Discussion started by: Jomeaide
7 Replies

4. Shell Programming and Scripting

Md5sum script

Hello, I need to download multiple files from an FTP server but occasionally they arrive in error so I need to perform an integrity check. I've been attempting to write a bash script that does the following: Downloads all files including those in sub directories Perform md5sum using... (4 Replies)
Discussion started by: shadyuk
4 Replies

5. Shell Programming and Scripting

PHP Script Help - Making links to files Clickable

Ok so I wrote a php script that outputs the below to users on a webpage. # Download: /home/content/d/i/v/divine1234/eBookDownloads/ScalpRemedy_jablaa12734.zip the php code that outputs the above is: echo ("<li>Download: $download_link</li>\n"); The thing is, I dont want... (1 Reply)
Discussion started by: SkySmart
1 Replies

6. Shell Programming and Scripting

Script to check MD5SUM on file

Hi, I currently have a shell script that takes an RPM and scp's it to a set of remote servers and installs it. What I would like to be able to do is make the script get the md5sum of the RPM locally (so get the md5sum of the rpm from where im running the script) and then scp the rpm to the... (0 Replies)
Discussion started by: tb1986
0 Replies

7. Shell Programming and Scripting

Making script show command (e.g. copy) being executed and variable substitution?

When script is running you only see when some of the commands are not successfull. Is there a way to see which command are executed and to show the substitution of variables as every line is executed ? (3 Replies)
Discussion started by: gr0124
3 Replies

8. UNIX for Dummies Questions & Answers

Making a copy of an Magneto Optical Disk

We are trying to make duplicates of some Magneto Optical Disks that were created in Irix 6.5. The disks are 2.3 gig and the using a scsi MOD drive. Is there possbily a disk copy like in dos or some simple script to do this - any help appreciated. Thanks (0 Replies)
Discussion started by: drew_holm
0 Replies

9. Shell Programming and Scripting

Running md5sum on a list of files

Hello, I would like to run md5sum on a list of files saved in a text file, and save the result in another file. (ie. md5sum `cat list.txt` > md5list.txt) I have tried several things, but I am always confronted to the same problem: some of the filenames have spaces. I have run sed on the... (5 Replies)
Discussion started by: SDelroen
5 Replies

10. Solaris

making copy of 0 level dump via ufsdump

Hi how do u make "copy" of o level dump taken via ufsdumo in solaris? To elaborate, imagine you have taken a 0 level dump via the following command ufsdump 0ulf /dev/rmt/1n / and then again execute the same command to take a second 0 level dump Now take an incremental dump ufsdump 1ulf... (2 Replies)
Discussion started by: vishalsngh
2 Replies
Login or Register to Ask a Question