Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Making a script to copy files not seen before (using md5sum)

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Making a script to copy files not seen before (using md5sum)
# 15  
Old 08-31-2013
This will do the trick:
Code:
#!/bin/bash
set -x

# The source directory where the photo folder on the phone is mirrored to
SRC=test1

# The destination directory where we want to copy only new photos we have copied before
DST=test2

# The MD5 list file that tracks which files we have copied before
MD5=/hd1/home/nick/test1.md5

# Check files against the MD5 list and then copy if not previously copied
# Then add the md5 for that file to the MD5 list
for f in $SRC/*
do
  FMD5=$(md5sum $f)
  grep -q $FMD5 $MD5
  if [[ $? -ne 0 ]]; then
    cp $SRC/$f $DST
    md5sum $f >> $MD5
  fi
done

Once you fixed the shell expansion thing you had to revert to a relative pathname for both the source and the destination.
This User Gave Thanks to gacanepa For This Post:
nbsparks
# 16  
Old 08-31-2013
OK... well there's only one hitch with that... I actually wanted to start my finalized script like this:

Code:
#!/bin/bash

# The username variable passed by command line to this script
USER=$1

# The source directory where the photo folder on the phone is mirrored to
SRC=/hd1/home/$USER/.phonesync/photos-backup

# The destination directory where we want to copy only new photos we have copied before
DST=/hd1/home/$USER/.phonesync/photos-new

# The MD5 list file that tracks which files we have copied before
MD5=/hd1/home/$USER/.phonesync/photos-backup.md5

This way I could set up cron jobs to execute this one script multiple times but for different users.

---------- Post updated at 09:31 AM ---------- Previous update was at 09:23 AM ----------

So this is getting closer to the final script I would like to have:

Code:
#!/bin/bash
set -x

# The username variable passed by command line to this script
USER=$1

# The source directory where the photo folder on the phone is mirrored
SRC=/hd1/home/$USER/.phonesync/photos-backup

# The destination directory where we want to copy only new photos that we have never copied before
DST=/hd1/home/$USER/.phonesync/photos-new

# The MD5 list file that tracks the files that we have copied before out of photos-backup directory
MD5=/hd1/home/$USER/.phonesync/photos-backup.md5

# Check files against the MD5 list and then copy if not previously copied
# Then add the md5 for that file to the MD5 list
for f in $SRC/*
do
  FMD5=$(md5sum $f)
  grep -q $FMD5 $MD5
  if [[ $? -ne 0 ]]; then
    cp $SRC/$f $DST
    md5sum $f >> $MD5
  fi
done

# In case this script gets run as root, redo the file ownership so users can access their photos
chown -R $USER:$USER $DST

I can't test that at the moment but I will set up a real test with those directories later today and put up the result.
# 17  
Old 08-31-2013
If you do this:
Code:
SRC=/hd1/home/$USER/.phonesync/photos-backup

Then
Code:
DST=/hd1/home/$USER/.phonesync/photos-new

And if your for loop is initialized like this:
Code:
for f in $SRC/*

The copy command cp $SRC/$f $DSTwill end up doing this:
Code:
cp /hd1/home/$USER/.phonesync/photos-backup///hd1/home/$USER/.phonesync/photos-backup/yourfile.jpg /hd1/home/$USER/.phonesync/photos-new
Which, as you can see, just doesn't make sense.

So here's what I would do:
Code:
cd $SRC
for f in *
do
  FMD5=$(md5sum $f)
  grep -q $FMD5 $MD5
  if [[ $? -ne 0 ]]; then
    cp $f $DST
    md5sum $f >> $MD5
  fi
done

As I said earlier, it's all a matter of messing up with pathnames, but this should do the trick.
This User Gave Thanks to gacanepa For This Post:
nbsparks
# 18  
Old 09-01-2013
copying unseen files
Why not simply check the dest folder .....
Code:
SRC=/user/nick/.phonesync/photos-backup
DST=/user/nick/.phonesync/photos-new

for f in $SRC/*
do
  test -x $DST/$f
  if [[ $? -ne 0 ]]; then
    cp $SRC/$f $DST
  fi
done

Last edited by Scott; 09-11-2013 at 02:58 PM.. Reason: Code tags
# 19  
Old 09-01-2013
Quote:
Originally Posted by steve matthews
Why not simply check the dest folder .....


SRC=/user/nick/.phonesync/photos-backup
DST=/user/nick/.phonesync/photos-new


for f in $SRC/*
do
test -x $DST/$f
if [[ $? -ne 0 ]]; then
cp $SRC/$f $DST
fi
done
The "why not" was explained in message #8 in this thread.

But, even if this idea would work, the code presented above would not; $f consists of the expansion of $SRC a slash and filename where filename is the name of a file in $SRC, so:
Code:
  test -x $DST/$f

tests for the existence of a file with the pathname $DST/$SRC/filename which will never be true.
This User Gave Thanks to Don Cragun For This Post:
nbsparks
# 20  
Old 09-04-2013
This is the final working script!

Code:
#!/bin/bash

# The username variable passed by command line to this script
USER=$1

# The source directory where the photo folder on the phone is mirrored to
SRC=/hd1/home/$USER/.phonesync/photos-backup

# The destination directory where we want to copy only new photos we have not copied before
DST=/hd1/home/$USER/.phonesync/photos-new

# The MD5 list file that tracks which files we have copied before
MD5=/hd1/home/$USER/.phonesync/photos-backup.md5

# Check files against the MD5 list and then copy if not previously copied
# Then add the md5 for that file to the MD5 list
cd $SRC
for f in *
do
  FMD5=$(md5sum $f)
  grep -q $FMD5 $MD5
  if [[ $? -ne 0 ]]; then
    cp $f $DST
    md5sum $f >> $MD5
  fi
done

# In case this script gets run as root, redo the file ownership so users can access their photos
chown -R $USER:$USER $DST

THANK YOU ALL! I've learned a little through this process. I wish I could say I understood it all. But the most useful thing I probably learned here was the "set -x" tool to help me see what a failing script is doing. Thank you everyone!
Last edited by nbsparks; 09-04-2013 at 02:52 PM..
# 21  
Old 09-04-2013
I am glad to hear that.
Please mark this thread as solved by going to the "Thread tools" menu and clicking on "Mark this thread as solved", for the future reference of other users.
Also, I would suggest you to take a look at this great book, The Linux command line, which contains lots of useful information on Linux commands and scripting. The last edition was published last month Smilie. Good luck!
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

[md5sum] script

I am getting No such file or directory if my variable contains white spaces... Is there a way to fix this? This works x="1.md5" md5sum -c "$x" This, does not x="23\ 5\ 6\ 7\ 8\ 9\ 10.md5" md5sum -c "$x" md5sum: '23\ 5\ 6\ 7\ 8\ 9\ 10.md5': No such file or directory How do I fix... (1 Reply)
Discussion started by: soichiro
1 Replies

2. Shell Programming and Scripting

Need script for making files based on some conditions.

Hi All, I have a text file (code_data.txt) with the followig data. AMAR AB123456 XYZ KIRAN CB789 ABC RAJ CS78890 XYZ KAMESH A33535335 ABC KUMAR MD678894 MAT RITESH SR3535355... (26 Replies)
Discussion started by: ROCK_PLSQL
26 Replies

3. Shell Programming and Scripting

Compare files in directories with md5sum

And not to start. I can compare files, that's easy. The problem is that I compare files in a directory, and check if these files exist in another directory. The problem is that the file names are not the same. So I have to compare with "md5sum" or something similar. How I can do? All this in... (7 Replies)
Discussion started by: Jomeaide
7 Replies

4. Shell Programming and Scripting

Md5sum script

Hello, I need to download multiple files from an FTP server but occasionally they arrive in error so I need to perform an integrity check. I've been attempting to write a bash script that does the following: Downloads all files including those in sub directories Perform md5sum using... (4 Replies)
Discussion started by: shadyuk
4 Replies

5. Shell Programming and Scripting

PHP Script Help - Making links to files Clickable

Ok so I wrote a php script that outputs the below to users on a webpage. # Download: /home/content/d/i/v/divine1234/eBookDownloads/ScalpRemedy_jablaa12734.zip the php code that outputs the above is: echo ("<li>Download: $download_link</li>\n"); The thing is, I dont want... (1 Reply)
Discussion started by: SkySmart
1 Replies

6. Shell Programming and Scripting

Script to check MD5SUM on file

Hi, I currently have a shell script that takes an RPM and scp's it to a set of remote servers and installs it. What I would like to be able to do is make the script get the md5sum of the RPM locally (so get the md5sum of the rpm from where im running the script) and then scp the rpm to the... (0 Replies)
Discussion started by: tb1986
0 Replies

7. Shell Programming and Scripting

Making script show command (e.g. copy) being executed and variable substitution?

When script is running you only see when some of the commands are not successfull. Is there a way to see which command are executed and to show the substitution of variables as every line is executed ? (3 Replies)
Discussion started by: gr0124
3 Replies

8. UNIX for Dummies Questions & Answers

Making a copy of an Magneto Optical Disk

We are trying to make duplicates of some Magneto Optical Disks that were created in Irix 6.5. The disks are 2.3 gig and the using a scsi MOD drive. Is there possbily a disk copy like in dos or some simple script to do this - any help appreciated. Thanks (0 Replies)
Discussion started by: drew_holm
0 Replies

9. Shell Programming and Scripting

Running md5sum on a list of files

Hello, I would like to run md5sum on a list of files saved in a text file, and save the result in another file. (ie. md5sum `cat list.txt` > md5list.txt) I have tried several things, but I am always confronted to the same problem: some of the filenames have spaces. I have run sed on the... (5 Replies)
Discussion started by: SDelroen
5 Replies

10. Solaris

making copy of 0 level dump via ufsdump

Hi how do u make "copy" of o level dump taken via ufsdumo in solaris? To elaborate, imagine you have taken a 0 level dump via the following command ufsdump 0ulf /dev/rmt/1n / and then again execute the same command to take a second 0 level dump Now take an incremental dump ufsdump 1ulf... (2 Replies)
Discussion started by: vishalsngh
2 Replies
Login or Register to Ask a Question