Sponsored Content
Top Forums Shell Programming and Scripting Shell Script - find, recursively, all files that are duplicated Post 302360598 by methyl on Friday 9th of October 2009 11:50:41 AM
Old 10-09-2009
Inefficient but still took 10 mins for 60,000 files with 200 duplicates. Unsuitable for large numbers of duplicates.

Code:
#!/bin/ksh
find . -type f -print|while read F1
do
    basename "${F1}"
done | sort | uniq -d | while read F2
do
    find . -type f -name "${F2}" -exec ls -lad '{}' \;
done

 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

find and replace a search string recursively in files

Hi , I have a directory structure as dir and subdirectories and files under it and so on.now I need to find the files which contain the search string under every dir and subdir and replace . my search string is like searchstring=/a/b string to be replaced=/a/c/b please help. ... (7 Replies)
Discussion started by: mohanpadamata
7 Replies

2. Shell Programming and Scripting

Shell script to find files

Hi Experts, I am trying to write a shell script that should 1. Find files with name (ab030.txt,Ab030.TXT,AB030.TXT,ab030.TXT,AB030.txt) 2. If any of the above found, rename it to AB030.TXT Thanks. (4 Replies)
Discussion started by: welldone
4 Replies

3. Shell Programming and Scripting

Need to find recursively all shell script in the /xyz directory

Pls. advise how to find or used grep recursively all shell script files. Some files doesnt have a .sh or .ksh extension name. find / -name "*" |xargs grep bin |grep sh ?? TIA (1 Reply)
Discussion started by: budz26
1 Replies

4. Shell Programming and Scripting

Recursively *.ext files using find

HI, Getting the syntax error " find: missing conjunction" for the below code D1_DIR=/x/y/z D1_NAME=file_name FILE_DIR=pset for file in `find ${D1_DIR}/${D1_NAME} -name "*\.${FILE_DIR}" /dev/null {} \;` do echo $file done #Trying to find all the files with *.pset... (5 Replies)
Discussion started by: cvsanthosh
5 Replies

5. Shell Programming and Scripting

Shell script to copy particular file from directories recursively

I have directory path in which there are several sub directories. In all these sub dir there will be one env.cnf file. I want to copy this env.cnf file from each sub dir's and place them in destination path by creating same filename as sub dir_env.cnf. After copying env.cnf files from source... (4 Replies)
Discussion started by: Optimus81
4 Replies

6. Shell Programming and Scripting

Script to compare files recursively using sdiff

Hi All, I have been surfing to get some idea on how to compare same files from two different paths. one path will have oldfiles directory and another path will have newfiles directory. Each main directories will have sub-directories in them and each sub-directories inturn will have... (3 Replies)
Discussion started by: Optimus81
3 Replies

7. Shell Programming and Scripting

Help on case to call recursively in UNIX Shell Script

Hi, I am New to Unix Shell Scripting basically, i need some help in achieving a case statement in Shell script to call recursively That is if case having like 1 2 3 4 options , if user inputs 1 and gets executed case should ask for options again but user should not input the same input value 1,... (7 Replies)
Discussion started by: karthikram
7 Replies

8. Shell Programming and Scripting

How to recursively /usr/bin/find only readonly files?

I'm having trouble because, for some reason, cp -R missed a few files. And so did xcopy/s. Since I'm running Cygwin on Win10, I decided to see if robocopy would be more effective. The trouble is someone, maybe xcopy/s or cp -R dutifully set certain files to be read only so when I try a... (6 Replies)
Discussion started by: siegfried
6 Replies

9. Shell Programming and Scripting

Find Large Files Recursively From Specific Directory

Hi. I found many scripts in the web of achieving this. But I like to use this one find /EDWH-DMT03 -xdev -size +10000 -exec ls -la {} \;|sort -n -k 5 > LARGE.rst But the problem is, why it still list out files with 89 bytes as the output? Is there anything wrong with the command? My... (7 Replies)
Discussion started by: aimy
7 Replies

10. UNIX for Beginners Questions & Answers

Find and removing the old files and zipping the files using shell script

Hi, I am trying to removing the old files which were older than 10 days and same g zipping the files using the shell script. script was return as follows. find /jboss7_homes/JBOSS7/SKYLIV??/SKYLIV??_CRM/jboss-eap-7.0/standalone/log -mtime +10 -type f | xargs rm -f find /cer_skyliv??/log... (6 Replies)
Discussion started by: venkat918
6 Replies
ExtUtils::Command(3pm)					 Perl Programmers Reference Guide				    ExtUtils::Command(3pm)

NAME
ExtUtils::Command - utilities to replace common UNIX commands in Makefiles etc. SYNOPSIS
perl -MExtUtils::Command -e cat files... > destination perl -MExtUtils::Command -e mv source... destination perl -MExtUtils::Command -e cp source... destination perl -MExtUtils::Command -e touch files... perl -MExtUtils::Command -e rm_f files... perl -MExtUtils::Command -e rm_rf directories... perl -MExtUtils::Command -e mkpath directories... perl -MExtUtils::Command -e eqtime source destination perl -MExtUtils::Command -e test_f file perl -MExtUtils::Command -e test_d directory perl -MExtUtils::Command -e chmod mode files... ... DESCRIPTION
The module is used to replace common UNIX commands. In all cases the functions work from @ARGV rather than taking arguments. This makes them easier to deal with in Makefiles. Call them like this: perl -MExtUtils::Command -e some_command some files to work on and NOT like this: perl -MExtUtils::Command -e 'some_command qw(some files to work on)' For that use Shell::Command. Filenames with * and ? will be glob expanded. FUNCTIONS cat cat file ... Concatenates all files mentioned on command line to STDOUT. eqtime eqtime source destination Sets modified time of destination to that of source. rm_rf rm_rf files or directories ... Removes files and directories - recursively (even if readonly) rm_f rm_f file ... Removes files (even if readonly) touch touch file ... Makes files exist, with current timestamp mv mv source_file destination_file mv source_file source_file destination_dir Moves source to destination. Multiple sources are allowed if destination is an existing directory. Returns true if all moves succeeded, false otherwise. cp cp source_file destination_file cp source_file source_file destination_dir Copies sources to the destination. Multiple sources are allowed if destination is an existing directory. Returns true if all copies succeeded, false otherwise. chmod chmod mode files ... Sets UNIX like permissions 'mode' on all the files. e.g. 0666 mkpath mkpath directory ... Creates directories, including any parent directories. test_f test_f file Tests if a file exists. Exits with 0 if it does, 1 if it does not (ie. shell's idea of true and false). test_d test_d directory Tests if a directory exists. Exits with 0 if it does, 1 if it does not (ie. shell's idea of true and false). dos2unix dos2unix files or dirs ... Converts DOS and OS/2 linefeeds to Unix style recursively. SEE ALSO
Shell::Command which is these same functions but take arguments normally. AUTHOR
Nick Ing-Simmons "ni-s@cpan.org" Maintained by Michael G Schwern "schwern@pobox.com" within the ExtUtils-MakeMaker package and, as a separate CPAN package, by Randy Kobes "r.kobes@uwinnipeg.ca". perl v5.18.2 2014-01-06 ExtUtils::Command(3pm)
All times are GMT -4. The time now is 04:56 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy