Visit Our UNIX and Linux User Community


Find duplicate filenames and remove in different mount point


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Find duplicate filenames and remove in different mount point
# 1  
Old 09-15-2009
Find duplicate filenames and remove in different mount point

Hi Gurus,

Do any kind souls encounter have the same script as mentioned here.

Find and compare filenames in different mount point and remove duplicates.


Thanks a million!!!
wanna13e
# 2  
Old 09-15-2009
Give some examples, like between which you want to find duplicate, and what you wanted to remove ?
# 3  
Old 09-15-2009
As I understand your question, if you have any two (or more) instances of files with the same name, you want to remove all but one, irrespective of whether they have the same sizes, contents, etc. Please feel free to say no to this, if this is not what you intended.

As a matter of interest, when you are removing these files, how do you intend to determine which occurence is preserved and which are deleted?

Did you intend to keep any records?
# 4  
Old 09-15-2009
Thank you for taking time to read my post

Quote:
Originally Posted by thegeek
Give some examples, like between which you want to find duplicate, and what you wanted to remove ?
mount/volumeA/folderA/1234_A_54321.pdf
mount/volumeB/folderB/123_A_54321.pdf
mount/volumeC/folderC/123_A_54321.pdf

The script will remove the duplicate in volumeB and volumeC but keep file in volumeA.

Thank a million!
wanna13e.

---------- Post updated at 08:22 PM ---------- Previous update was at 08:20 PM ----------

Quote:
Originally Posted by markone
As I understand your question, if you have any two (or more) instances of files with the same name, you want to remove all but one, irrespective of whether they have the same sizes, contents, etc. Please feel free to say no to this, if this is not what you intended.

As a matter of interest, when you are removing these files, how do you intend to determine which occurence is preserved and which are deleted?

Did you intend to keep any records?
Yes, i intend to keep the records as shown in the example above in my post.


Thank a million!!!
wanna13e.
# 5  
Old 09-16-2009
lot of open source tools are available in sourceforge, as
fdupes & finddup and so on.
# 6  
Old 09-16-2009
I think fdupes use hashing (md5/sha) to determine if you have duplicate files. Way better than checking by filename.
# 7  
Old 09-16-2009
im not sure but i think diff command can do compare even if file is binary.
alternatively, you can create script and use diff to compare or follow what suggested above.

Previous Thread | Next Thread
Test Your Knowledge in Computers #41
Difficulty: Easy
In computer lingo, FAT stands for 'File Access Table'.
True or False?

8 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Mount Point is locked

Hi there, I have a mount point that is locked. How do I unlocked it? (1 Reply)
Discussion started by: alvinoo
1 Replies

2. UNIX for Beginners Questions & Answers

How to create a new mount point with 600GB and add 350 GBexisting mount point? IN AIX

How to create a new mount point with 600GB and add 350 GBexisting mount point Best if there step that i can follow or execute before i mount or add diskspace IN AIX Thanks (2 Replies)
Discussion started by: Thilagarajan
2 Replies

3. Shell Programming and Scripting

Script to find a mount point on multiple server

Hi, I have to find a mount point called... /opt/recall on multiple servers.. .Could someone give me a suggestion on how I could do this? (2 Replies)
Discussion started by: gartie
2 Replies

4. Shell Programming and Scripting

Find and remove duplicate record and print list

Gents, I needs to delete duplicate values and only get uniq values based in columns 2-27 Always we should keep the last record found... I need to store one clean file and other with the duplicate values removed. Input : S3033.0 7305.01 0 420123.8... (18 Replies)
Discussion started by: jiam912
18 Replies

5. UNIX for Dummies Questions & Answers

find & remove characters in filenames

I have a group of files in different directories with characters such as " ? : in the file names. How do I find these files and remove these characters on mass? Thanks (19 Replies)
Discussion started by: barrydocks
19 Replies

6. Shell Programming and Scripting

Remove Duplicate Filenames in 2 very large directories

Hello Gurus, O/S RHEL4 I have a requirement to compare two linux based directories for duplicate filenames and remove them. These directories are close to 2 TB each. I have tried running a: Prompt>diff -r data1/ data2/ I have tried this as well: jason@jason-desktop:~$ cat script.sh ... (7 Replies)
Discussion started by: jaysunn
7 Replies

7. UNIX for Dummies Questions & Answers

tar contains duplicate filenames

I have a problem where tar is somehow creating duplicate filenames when tarring a directory. Doing an ls on the directory does not show any duplicate filenames, yet when the directory is tarred, you can see that there are duplicates: bash-2.03# pwd /var/log/cricket bash-2.03# ls -1 | sort |... (2 Replies)
Discussion started by: dangral
2 Replies

8. UNIX for Dummies Questions & Answers

mount point

hi people, I'm trying to create a mount point, but am having no sucess at all, with the following: mount -F ufs /dev/dsk/diskname /newdirectory but i keep getting - mount-point /newdirectory doesn't exist. What am i doing wrong/missing? Thanks Rc (1 Reply)
Discussion started by: colesy
1 Replies

Featured Tech Videos