Sponsored Content
Operating Systems Linux Ubuntu Find duplicates among 2 directories Post 303031255 by nezabudka on Monday 25th of February 2019 04:51:10 AM
Old 02-25-2019
There is a tool that can determine the identity of files using the md5 sum.
Code:
apt install fdupes

Look at the listing
Code:
fdupes dir1/ dir2/

Interactive mode with a choice to remove
Code:
fdupes -d dir1/ dir2/

The following case is suitable for use in the script
All duplicates of the file will be deleted exclude only the first file (in order of sorting name files and then name dirs!) will be saved.
A simple way to change the directory with the saved file try to use -i option. It does not change the save directory, but in the reorganized sort order, the upper file may be in the folder you need
try
fdupes -i dir1/ dir2/
and then use
fdupes -Nd dir1/ dir2/
Well, before you delete something, be sure to read the man pages on the command and make training tests on its use.
This User Gave Thanks to nezabudka For This Post:
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Awk to find duplicates in 2nd field

I want to find duplicates in file on 2nd field i wrote this code: nawk '{a++} END{for i in a {if (a>1) print}}' temp Could not find whats wrong with this. Appreciate help (5 Replies)
Discussion started by: pinnacle
5 Replies

2. Shell Programming and Scripting

Shellscript to find duplicates according to size

I have a folder which in turn has numerous sub folders all containing pdf files with same file named in different ways. So I need a script if it can be written to find and print the duplicate files (That is files with same size) along with the respective paths. So I assume here that same file... (5 Replies)
Discussion started by: deaddevil
5 Replies

3. Shell Programming and Scripting

How to find 777 permisson is there or not for Directories and sub-directories

Hi All, I am Oracle Apps Tech guy, I have a requirement to find 777 permission is there or not for all Folders and Sub-folders Under APPL_TOP (Folder/directory) with below conditions i) the directory names should start with xx..... (like xxau,xxcfi,xxcca...etc) and exclude the directory... (11 Replies)
Discussion started by: gagan4599
11 Replies

4. Shell Programming and Scripting

Find duplicates in the first column of text file

Hello, My text file has input of the form abc dft45.xml ert rt653.xml abc ert57.xml I need to write a perl script/shell script to find duplicates in the first column and write it into a text file of the form... abc dft45.xml abc ert57.xml Can some one help me plz? (5 Replies)
Discussion started by: gameboy87
5 Replies

5. UNIX for Dummies Questions & Answers

sort and find duplicates for files with no white space

example data 5666700842511TAfmoham03151008075205999900000001000001000++ 5666700843130MAfmoham03151008142606056667008390315100005001 6666666663130MAfmoham03151008142606056667008390315100005001 I'd like to sort on position 10-14 where the characters are eq "130MA". Then based on positions... (0 Replies)
Discussion started by: mmarshall
0 Replies

6. UNIX for Dummies Questions & Answers

Using grep command to find the pattern of text in all directories and sub-directories.

Hi all, Using grep command, i want to find the pattern of text in all directories and sub-directories. e.g: if i want to search for a pattern named "parmeter", i used the command grep -i "param" ../* is this correct? (1 Reply)
Discussion started by: vinothrajan55
1 Replies

7. Shell Programming and Scripting

find numeric duplicates from 300 million lines....

these are numeric ids.. 222932017099186177 222932014385467392 222932017371820032 222932017409556480 I have text file having 300 millions of line as shown above. I want to find duplicates from this file. Please suggest the quicker way.. sort | uniq -d will... (3 Replies)
Discussion started by: pamu
3 Replies

8. Shell Programming and Scripting

Find All duplicates based on multiple keys

Hi All, Input.txt 123,ABC,XYZ1,A01,IND,I68,IND,NN 123,ABC,XYZ1,A01,IND,I67,IND,NN 998,SGR,St,R834,scot,R834,scot,NN 985,SGR0399,St,R180,T15,R180,T1,YY 985,SGR0399,St,R180,T15,R180,T1,NN 985,SGR0399,St,R180,T15,R180,T1,NN 2943,SGR?99,St,R68,Scot,R77,Scot,YY... (2 Replies)
Discussion started by: unme
2 Replies

9. Shell Programming and Scripting

Find duplicates in 2 & 3rd column and their ID

with below given format, I have been trying to find out all IDs for those entries with duplicate names in 2nd and 3rd columns and their count like how many time duplication happened for any name if any, 0.237788 Aaban Aahva 0.291066 Aabheer Aahlaad 0.845814 Aabid Aahan 0.152208 Aadam... (6 Replies)
Discussion started by: busyboy
6 Replies

10. UNIX for Beginners Questions & Answers

Find duplicates in file with line numbers

Hello All, This is a noob question. I tried searching for the answer but the answer found did not help me . I have a file that can have duplicates. 100 200 300 400 100 150 the number 100 is duplicated twice. I want to find the duplicate along with the line number. expected... (4 Replies)
Discussion started by: vatigers
4 Replies
scandir(3)						     Library Functions Manual							scandir(3)

NAME
scandir, alphasort - Scans or sorts directory contents LIBRARY
Standard C Library (libc.a) SYNOPSIS
#include <sys/types.h> #include <sys/dir.h> int scandir ( char *dir_name, struct dirent * (*name_list[ ]), int (*select) ( struct dirent *dir ), int (*compare) ( struct dirent **dir1, struct dirent **dir2 ) ); int alphasort ( struct dirent **dir1, struct dirent **dir2 ); PARAMETERS
Points to the directory name. Points to the array of pointers to directory entries. Points to a user-supplied function that is called by the scandir() function to select which entries to include in the array. Points to a user-supplied function that sorts the completed array. Points to a dirent structure. Points to a dirent structure. DESCRIPTION
The scandir() function reads the directory pointed to by the dir_name parameter. It then uses the malloc() function to create an array of pointers to directory entries. The scandir() function returns the number of entries in the array and, through the name_list parameter, a pointer to the array. The select parameter points to a user-supplied function that the scandir() function calls to select which entries to include in the array. The selection routine is passed a pointer to a directory entry and returns a nonzero value for a directory entry that is included in the array. If the select parameter is a null value, all directory entries are included. The compare parameter points to a user-supplied function that is passed to the qsort() function to sort the completed array. If the compare parameter is a null value, the array is not sorted. The memory allocated to the array can be deallocated by freeing each pointer in the array, and the array itself, with the free() function. The alphasort() function alphabetically compares the two dirent structures pointed to by the dir1 and dir2 parameters. This function can be passed as the compare parameter to either the scandir() function or the qsort() function. A user-supplied subroutine may also be used. RETURN VALUES
The scandir() function returns -1 if the directory cannot be opened for reading or if the malloc() function cannot allocate enough memory to hold all the data structures. If successful, the scandir() function returns the number of entries found. The alphasort() function returns the following values: Less than 0 (zero): The dirent structure pointed to by the dir1 parameter is lexi- cally less than the dirent structure pointed to by the dir2 parameter. 0 (zero): The dirent structures pointed to by the dir1 parameter and the dir2 parameter are equal. Greater than 0 (zero): The dirent structure pointed to by the dir1 parameter is lexically greater than the dirent structure pointed to by the dir2 parameter. RELATED INFORMATION
Functions: malloc(3), opendir(3), qsort(3) delim off scandir(3)
All times are GMT -4. The time now is 03:49 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy