Making a script to copy files not seen before (using md5sum)


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Making a script to copy files not seen before (using md5sum)
# 8  
Old 08-30-2013
Thank you for suggestions. But rsync is not my solution. I want to be able to delete all the files in the $DST folder but yet not have files copy over again, even if the $DST is empty and the $SRC is full. Bottom line, if the files that are "currently" in $SRC have already been copied once over to $DST, then the don't copy again (unless I intentionally restart be deleting the MD5 list).

I was ponder this code last night too and I realized that it's just parsing a filename in an MD5 list. It's almost like I need to do something like this...

Code:
#!/bin/bash

USER=$1
SRC=/user/$USER/.phonesync/photos-backup
DST=/user/$USER/.phonesync/photos-new
MD5=/user/$USER/.phonesync/photos-backup.md5

for f in $SRC/*
do
  FMD5=md5sum $f
  grep -q $FMD5 $MD5
  if [[ $? -ne 0 ]]; then
    cp $SRC/$f $DST
    md5sum $f >> $MD5
  fi
done

Unfortunately, I'm not where I can test this right now, but does that seem like it would work?

---------- Post updated at 10:34 AM ---------- Previous update was at 09:39 AM ----------

OK... I had a moment to test a script and this is what happened.

Here are the two directories I'm working with as a sample:
Code:
nick@server ~$ ls t* -a -l
test1:
total 12
drwxr-xr-x 2 nick nick 4096 Aug 30 10:12 .
drwxr-xr-x 7 nick nick 4096 Aug 30 10:20 ..
-rw-r--r-- 1 nick nick   51 Aug 30 10:28 testfile1.jpg

test2:
total 8
drwxr-xr-x 2 nick nick 4096 Aug 30 10:09 .
drwxr-xr-x 7 nick nick 4096 Aug 30 10:20 ..

Here is my code:
Code:
#!/bin/bash

# The source directory where the photo folder on the phone is mirrored to
SRC=test1

# The destination directory where we want to copy only new photos we have copied before
DST=test2

# The MD5 list file that tracks which files we have copied before
MD5=test1.md5

# Check files against the MD5 list and then copy if not previously copied
# Then add the md5 for that file to the MD5 list
for f in $SRC/*
do
  FMD5=md5sum $f
  grep -q $FMD5 $MD5
  if [[ $? -ne 0 ]]; then
    cp $SRC/$f $DST
    md5sum $f >> $MD5
  fi
done

Here is the output:
Code:
nick@server ~$ ./copy.sh
./copy2.sh: line 16: test1/testfile1.jpg: Permission denied

Anyone tell me where I went wrong?
# 9  
Old 08-30-2013
Try it using the full path in your variables:
Code:
SRC=/user/nick/test1
DST=/user/nick/test2

# 10  
Old 08-30-2013
OK. Revised the script code.
Code:
#!/bin/bash

# The source directory where the photo folder on the phone is mirrored to
SRC=/hd1/home/nick/test1

# The destination directory where we want to copy only new photos we have copied before
DST=/hd1/home/nick/test2

# The MD5 list file that tracks which files we have copied before
MD5=/hd1/home/nick/test1.md5

# Check files against the MD5 list and then copy if not previously copied
# Then add the md5 for that file to the MD5 list
for f in $SRC/*
do
  FMD5=md5sum $f
  grep -q $FMD5 $MD5
  if [[ $? -ne 0 ]]; then
    cp $SRC/$f $DST
    md5sum $f >> $MD5
  fi
done

Here is the actual location on the system:
Code:
nick@server /$ ls /hd1/home/nick/t* -l
-rw-r--r-- 1 nick nick    0 Aug 30 10:32 /hd1/home/nick/test1.md5

/hd1/home/nick/test1:
total 4
-rw-r--r-- 1 nick nick 51 Aug 30 10:28 testfile1.jpg

/hd1/home/nick/test2:
total 0

And the output:
Code:
nick@server ~$ ./copy2.sh
./copy2.sh: line 16: /hd1/home/nick/test1/testfile1.jpg: Permission denied

# 11  
Old 08-30-2013
Quote:
nick@server ~$ ./copy.sh
./copy2.sh: line 16: test1/testfile1.jpg: Permission denied
First off, I'm a little confused why you're getting an error after running copy.sh but the error points to another file (copy2.sh). Do you have 2 script files?
The error message you are getting points to line #16, but this is what I see in line #16:
Code:
FMD5=md5sum $f

Is that correct?
Also, is the script located in the same directory as the test1 and test2 directories?

---------- Post updated at 07:57 PM ---------- Previous update was at 07:52 PM ----------

Also, you could try turning on debugging mode with
Code:
set -x

right after #!/bin/bash
Like so
Code:
#!/bin/bash
set -x
[The rest of your script goes here]

And run it again.
This will tell you what happens with your script step by step as it is executed.
This User Gave Thanks to gacanepa For This Post:
# 12  
Old 08-30-2013
OK. Here is a simpler view of directory (I hope) so all know what files are where:
Code:
nick@server ~$ cd ~
nick@server ~$ ls -R
.:
copy2.sh  test1  test1.md5  test2

./test1:
testfile1.jpg

./test2:
nick@server ~$ cd /hd1/home/nick
nick@server ~$ ls -R
.:
copy2.sh  test1  test1.md5  test2

./test1:
testfile1.jpg

./test2:

So you can see that my home directory is actually /hd1/home/nick
Inside my home folder there are two directories: test1, and test2
The "test1" folder has file in it.
The "test2" folder is empty.
I have a "test1.md5" file in my home directory that is empty (created with "touch test1.md5" command as one of my troubleshooting attempts).

And I made the modification (Adding set -x) to the copy2.sh
Code:
nick@server ~$ cat copy2.sh
#!/bin/bash
set -x

# The source directory where the photo folder on the phone is mirrored to
SRC=/hd1/home/nick/test1

# The destination directory where we want to copy only new photos we have copied before
DST=/hd1/home/nick/test2

# The MD5 list file that tracks which files we have copied before
MD5=/hd1/home/nick/test1.md5

# Check files against the MD5 list and then copy if not previously copied
# Then add the md5 for that file to the MD5 list
for f in $SRC/*
do
  FMD5=md5sum $f
  grep -q $FMD5 $MD5
  if [[ $? -ne 0 ]]; then
    cp $SRC/$f $DST
    md5sum $f >> $MD5
  fi
done

And here is the new output.
Code:
nick@server ~$ ./copy2.sh
+ SRC=/hd1/home/nick/test1
+ DST=/hd1/home/nick/test2
+ MD5=/hd1/home/nick/test1.md5
+ for f in '$SRC/*'
+ FMD5=md5sum
+ /hd1/home/nick/test1/testfile1.jpg
./copy2.sh: line 17: /hd1/home/nick/test1/testfile1.jpg: Permission denied
+ grep -q /hd1/home/nick/test1.md5
^C
nick@server ~$

It stops there and I have to CTRL+C to stop it and get back to prompt.

Oh and the copy.sh vs copy2.sh issue, I don't know. Perhaps that was an errant keystroke on the delete key from me when typing up my post. Can't reproduce that.
# 13  
Old 08-30-2013
There you go.
Refer to this line:
Code:
+ grep -q /hd1/home/nick/test1.md5

The fact that the script stops there (and the fact that the command is incomplete) tells us that the error is in the previous command.
Replace the following line:
Code:
FMD5=md5sum $f

With
Code:
FMD5=$(md5sum $f)

The error resides in that the shell is not expanding the result of the md5sum command, therefore you lack one argument in the grep command that follows.
Please let me know how it goes.
This User Gave Thanks to gacanepa For This Post:
# 14  
Old 08-31-2013
So I made that change. Here is the new script:

Code:
nick@server ~$ cat copy2.sh
#!/bin/bash
set -x

# The source directory where the photo folder on the phone is mirrored to
SRC=/hd1/home/nick/test1

# The destination directory where we want to copy only new photos we have copied before
DST=/hd1/home/nick/test2

# The MD5 list file that tracks which files we have copied before
MD5=/hd1/home/nick/test1.md5

# Check files against the MD5 list and then copy if not previously copied
# Then add the md5 for that file to the MD5 list
for f in $SRC/*
do
  FMD5=$(md5sum $f)
  grep -q $FMD5 $MD5
  if [[ $? -ne 0 ]]; then
    cp $SRC/$f $DST
    md5sum $f >> $MD5
  fi
done

Here is the new output.

Code:
nick@server ~$ ./copy2.sh
+ SRC=/hd1/home/nick/test1
+ DST=/hd1/home/nick/test2
+ MD5=/hd1/home/nick/test1.md5
+ for f in '$SRC/*'
++ md5sum /hd1/home/nick/test1/testfile1.jpg
+ FMD5='56eb2d747f5e0e08d8e743c968fc53aa  /hd1/home/nick/test1/testfile1.jpg'
+ grep -q 56eb2d747f5e0e08d8e743c968fc53aa /hd1/home/nick/test1/testfile1.jpg /hd1/home/nick/test1.md5
+ [[ 1 -ne 0 ]]
+ cp /hd1/home/nick/test1//hd1/home/nick/test1/testfile1.jpg /hd1/home/nick/test2
cp: cannot stat `/hd1/home/nick/test1//hd1/home/nick/test1/testfile1.jpg': No such file or directory
+ md5sum /hd1/home/nick/test1/testfile1.jpg


nick@server ~$ ls t*
test1.md5

test1:
testfile1.jpg

test2:
nick@server ~$ ~

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

[md5sum] script

I am getting No such file or directory if my variable contains white spaces... Is there a way to fix this? This works x="1.md5" md5sum -c "$x" This, does not x="23\ 5\ 6\ 7\ 8\ 9\ 10.md5" md5sum -c "$x" md5sum: '23\ 5\ 6\ 7\ 8\ 9\ 10.md5': No such file or directory How do I fix... (1 Reply)
Discussion started by: soichiro
1 Replies

2. Shell Programming and Scripting

Need script for making files based on some conditions.

Hi All, I have a text file (code_data.txt) with the followig data. AMAR AB123456 XYZ KIRAN CB789 ABC RAJ CS78890 XYZ KAMESH A33535335 ABC KUMAR MD678894 MAT RITESH SR3535355... (26 Replies)
Discussion started by: ROCK_PLSQL
26 Replies

3. Shell Programming and Scripting

Compare files in directories with md5sum

And not to start. I can compare files, that's easy. The problem is that I compare files in a directory, and check if these files exist in another directory. The problem is that the file names are not the same. So I have to compare with "md5sum" or something similar. How I can do? All this in... (7 Replies)
Discussion started by: Jomeaide
7 Replies

4. Shell Programming and Scripting

Md5sum script

Hello, I need to download multiple files from an FTP server but occasionally they arrive in error so I need to perform an integrity check. I've been attempting to write a bash script that does the following: Downloads all files including those in sub directories Perform md5sum using... (4 Replies)
Discussion started by: shadyuk
4 Replies

5. Shell Programming and Scripting

PHP Script Help - Making links to files Clickable

Ok so I wrote a php script that outputs the below to users on a webpage. # Download: /home/content/d/i/v/divine1234/eBookDownloads/ScalpRemedy_jablaa12734.zip the php code that outputs the above is: echo ("<li>Download: $download_link</li>\n"); The thing is, I dont want... (1 Reply)
Discussion started by: SkySmart
1 Replies

6. Shell Programming and Scripting

Script to check MD5SUM on file

Hi, I currently have a shell script that takes an RPM and scp's it to a set of remote servers and installs it. What I would like to be able to do is make the script get the md5sum of the RPM locally (so get the md5sum of the rpm from where im running the script) and then scp the rpm to the... (0 Replies)
Discussion started by: tb1986
0 Replies

7. Shell Programming and Scripting

Making script show command (e.g. copy) being executed and variable substitution?

When script is running you only see when some of the commands are not successfull. Is there a way to see which command are executed and to show the substitution of variables as every line is executed ? (3 Replies)
Discussion started by: gr0124
3 Replies

8. UNIX for Dummies Questions & Answers

Making a copy of an Magneto Optical Disk

We are trying to make duplicates of some Magneto Optical Disks that were created in Irix 6.5. The disks are 2.3 gig and the using a scsi MOD drive. Is there possbily a disk copy like in dos or some simple script to do this - any help appreciated. Thanks (0 Replies)
Discussion started by: drew_holm
0 Replies

9. Shell Programming and Scripting

Running md5sum on a list of files

Hello, I would like to run md5sum on a list of files saved in a text file, and save the result in another file. (ie. md5sum `cat list.txt` > md5list.txt) I have tried several things, but I am always confronted to the same problem: some of the filenames have spaces. I have run sed on the... (5 Replies)
Discussion started by: SDelroen
5 Replies

10. Solaris

making copy of 0 level dump via ufsdump

Hi how do u make "copy" of o level dump taken via ufsdumo in solaris? To elaborate, imagine you have taken a 0 level dump via the following command ufsdump 0ulf /dev/rmt/1n / and then again execute the same command to take a second 0 level dump Now take an incremental dump ufsdump 1ulf... (2 Replies)
Discussion started by: vishalsngh
2 Replies
Login or Register to Ask a Question