Sponsored Content
Top Forums Shell Programming and Scripting Shell Script - find, recursively, all files that are duplicated Post 302360598 by methyl on Friday 9th of October 2009 11:50:41 AM
Old 10-09-2009
Inefficient but still took 10 mins for 60,000 files with 200 duplicates. Unsuitable for large numbers of duplicates.

Code:
#!/bin/ksh
find . -type f -print|while read F1
do
    basename "${F1}"
done | sort | uniq -d | while read F2
do
    find . -type f -name "${F2}" -exec ls -lad '{}' \;
done

 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

find and replace a search string recursively in files

Hi , I have a directory structure as dir and subdirectories and files under it and so on.now I need to find the files which contain the search string under every dir and subdir and replace . my search string is like searchstring=/a/b string to be replaced=/a/c/b please help. ... (7 Replies)
Discussion started by: mohanpadamata
7 Replies

2. Shell Programming and Scripting

Shell script to find files

Hi Experts, I am trying to write a shell script that should 1. Find files with name (ab030.txt,Ab030.TXT,AB030.TXT,ab030.TXT,AB030.txt) 2. If any of the above found, rename it to AB030.TXT Thanks. (4 Replies)
Discussion started by: welldone
4 Replies

3. Shell Programming and Scripting

Need to find recursively all shell script in the /xyz directory

Pls. advise how to find or used grep recursively all shell script files. Some files doesnt have a .sh or .ksh extension name. find / -name "*" |xargs grep bin |grep sh ?? TIA (1 Reply)
Discussion started by: budz26
1 Replies

4. Shell Programming and Scripting

Recursively *.ext files using find

HI, Getting the syntax error " find: missing conjunction" for the below code D1_DIR=/x/y/z D1_NAME=file_name FILE_DIR=pset for file in `find ${D1_DIR}/${D1_NAME} -name "*\.${FILE_DIR}" /dev/null {} \;` do echo $file done #Trying to find all the files with *.pset... (5 Replies)
Discussion started by: cvsanthosh
5 Replies

5. Shell Programming and Scripting

Shell script to copy particular file from directories recursively

I have directory path in which there are several sub directories. In all these sub dir there will be one env.cnf file. I want to copy this env.cnf file from each sub dir's and place them in destination path by creating same filename as sub dir_env.cnf. After copying env.cnf files from source... (4 Replies)
Discussion started by: Optimus81
4 Replies

6. Shell Programming and Scripting

Script to compare files recursively using sdiff

Hi All, I have been surfing to get some idea on how to compare same files from two different paths. one path will have oldfiles directory and another path will have newfiles directory. Each main directories will have sub-directories in them and each sub-directories inturn will have... (3 Replies)
Discussion started by: Optimus81
3 Replies

7. Shell Programming and Scripting

Help on case to call recursively in UNIX Shell Script

Hi, I am New to Unix Shell Scripting basically, i need some help in achieving a case statement in Shell script to call recursively That is if case having like 1 2 3 4 options , if user inputs 1 and gets executed case should ask for options again but user should not input the same input value 1,... (7 Replies)
Discussion started by: karthikram
7 Replies

8. Shell Programming and Scripting

How to recursively /usr/bin/find only readonly files?

I'm having trouble because, for some reason, cp -R missed a few files. And so did xcopy/s. Since I'm running Cygwin on Win10, I decided to see if robocopy would be more effective. The trouble is someone, maybe xcopy/s or cp -R dutifully set certain files to be read only so when I try a... (6 Replies)
Discussion started by: siegfried
6 Replies

9. Shell Programming and Scripting

Find Large Files Recursively From Specific Directory

Hi. I found many scripts in the web of achieving this. But I like to use this one find /EDWH-DMT03 -xdev -size +10000 -exec ls -la {} \;|sort -n -k 5 > LARGE.rst But the problem is, why it still list out files with 89 bytes as the output? Is there anything wrong with the command? My... (7 Replies)
Discussion started by: aimy
7 Replies

10. UNIX for Beginners Questions & Answers

Find and removing the old files and zipping the files using shell script

Hi, I am trying to removing the old files which were older than 10 days and same g zipping the files using the shell script. script was return as follows. find /jboss7_homes/JBOSS7/SKYLIV??/SKYLIV??_CRM/jboss-eap-7.0/standalone/log -mtime +10 -type f | xargs rm -f find /cer_skyliv??/log... (6 Replies)
Discussion started by: venkat918
6 Replies
shell_builtins(1)						   User Commands						 shell_builtins(1)

NAME
shell_builtins, case, for, foreach, function, if, repeat, select, switch, until, while - shell command interpreter built-in commands DESCRIPTION
The shell command interpreters csh(1), ksh(1), ksh93(1), and sh(1) have special built-in commands. The commands case, for, foreach, func- tion, if, repeat, select, switch, until, and while are commands in the syntax recognized by the shells. They are described in the Commands section of the manual pages of the respective shells. In ksh93(1), fc, hash, stop, suspend, times, and type are aliases by default. In ksh93, the following built-ins are bound to the /bin pathname by default and are invoked if the pathname search encounters an executable command of that name in the /bin or /usr/bin directory: cat, chown, getconf, head, mkdir, rmdir, tee, uniq, and wc. The remaining commands listed in the following table are built into the shells for reasons such as efficiency or data sharing between com- mand invocations. They are described on their respective manual pages. Command Shell ----------------------------------------------------------- ++**alias csh, ksh, ksh93 bg csh, ksh, ksh93, sh +*break csh, ksh, ksh93, sh builtin ksh93 case csh, ksh, ksh93, sh cat ksh93 cd csh, ksh, ksh93, sh chdir csh, sh chown ksh93 command ksh93 +*continue csh, ksh, ksh93, sh dirs csh disown ksh93 echo csh, ksh, ksh93, sh +*eval csh, ksh, ksh93, sh +*exec csh, ksh, ksh93, sh +*exit csh, ksh, ksh93, sh ++**export ksh, ksh93, sh false ksh, ksh93 fc ksh, ksh93 fg csh, ksh, ksh93, sh for ksh, ksh93, sh foreach csh function ksh, ksh93 getconf ksh93 getopts ksh, ksh93, sh glob csh goto csh hash ksh, ksh93, sh hashstat csh head ksh93 hist ksh93 history csh if csh, ksh, ksh93, sh jobs csh, ksh, ksh93, sh kill csh, ksh, ksh93, sh let ksh, ksh93, limit csh login csh, ksh, ksh93, sh logout csh mkdir ksh93 nice csh +*newgrp ksh, ksh93, sh nohup csh notify csh onintr csh popd csh print ksh, ksh93 printf ksh93 pushd csh pwd ksh, ksh93, sh read ksh, ksh93, sh ++**readonly ksh, ksh93, sh rehash csh repeat csh +*return ksh, ksh93, sh select ksh, ksh93 +set csh, ksh, ksh93, sh setenv csh shift csh, ksh, ksh93, sh sleep ksh93 source csh stop csh, ksh, ksh93, sh suspend csh, ksh, sh switch csh tee ksh93 test ksh, ksh93, sh time csh *times ksh, ksh93, sh *+trap ksh, ksh93, sh true ksh, ksh93 type ksh, ksh93, sh ++**typeset ksh, ksh93 ulimit ksh, ksh93, sh umask csh, ksh, ksh93, sh +unalias csh, ksh, ksh93 unhash csh uniq ksh93 unlimit csh +unset csh, ksh, ksh93, sh unsetenv csh until ksh, ksh93, sh *wait csh, ksh, ksh93, sh whence ksh, ksh93 while csh, ksh, ksh93, sh Bourne Shell, sh, Special Commands Input/output redirection is now permitted for these commands. File descriptor 1 is the default output location. When Job Control is enabled, additional Special Commands are added to the shell's environment. In addition to these built-in reserved command words, sh also uses: : No effect; the command does nothing. A zero exit code is returned. .filename Read and execute commands from filename and return. The search path specified by PATH is used to find the directory contain- ing filename. C shell, csh Built-in commands are executed within the C shell. If a built-in command occurs as any component of a pipeline except the last, it is exe- cuted in a subshell. In addition to these built-in reserved command words, csh also uses: : Null command. This command is interpreted, but performs no action. Korn Shell, ksh, Special Commands Input/Output redirection is permitted. Unless otherwise indicated, the output is written on file descriptor 1 and the exit status, when there is no syntax error, is zero. Commands that are preceded by one or two * (asterisks) are treated specially in the following ways: 1. Variable assignment lists preceding the command remain in effect when the command completes. 2. I/O redirections are processed after variable assignments. 3. Errors cause a script that contains them to abort. 4. Words, following a command preceded by ** that are in the format of a variable assignment, are expanded with the same rules as a variable assignment. This means that tilde substitution is performed after the = sign and word splitting and file name genera- tion are not performed. In addition to these built-in reserved command words, ksh also uses: * : [ arg ... ] The command only expands parameters. * .file [ arg ... ] Read the complete file then execute the commands. The commands are executed in the current shell environment. The search path specified by PATH is used to find the directory containing file. If any arguments arg are specified, they become the positional parameters. Otherwise, the positional parameters are unchanged. The exit status is the exit status of the last command executed. the loop termination test. Korn Shell, ksh93, Special Commands Input/Output redirection is permitted. Unless otherwise indicated, the output is written on file descriptor 1 and the exit status, when there is no syntax error, is zero. Except for :, true, false, echo, newgrp, and login, all built-in commands accept -- to indicate end of options. They also interpret the option --man as a request to display the manual page onto standard error and -? as a help request which prints a usage message on standard error. Commands that are preceded by one or two + are treated specially in the following ways: 1. Variable assignment lists preceding the command remain in effect when the command completes. 2. I/O redirections are processed after variable assignments. 3. Errors cause a script that contains them to abort. 4. They are not valid function names. 5. Words, following a command preceded by ++ that are in the format of a variable assignment, are expanded with the same rules as a variable assignment. This means that tilde substitution is performed after the = sign and field splitting and file name genera- tion are not performed. In addition to these built-in reserved command words, ksh93 also uses: : [ arg ... ] The command only expands parameters. .name [ arg ... ] If name is a function defined with the function name reserved word syntax, the function is executed in the cur- rent environment (as if it had been defined with the name() syntax.) Otherwise if name refers to a file, the file is read in its entirety and the commands are executed in the current shell environment. The search path specified by PATH is used to find the directory containing the file. If any arguments arg are specified, they become the positional parameters while processing the . command and the original positional parameters are restored upon completion. Otherwise the positional parameters are unchanged. The exit status is the exit status of the last command executed. SEE ALSO
Intro(1), alias(1), break(1), builtin(1), cd(1), chmod(1), csh(1), disown(1), echo(1), exec(1), exit(1), find(1), getoptcvt(1), getopts(1), glob(1), hash(1), history(1), jobs(1), kill(1), ksh(1), ksh93(1), let(1), limit(1), login(1), logout(1), newgrp(1), nice(1), nohup(1), print(1), printf(1), pwd(1), read(1), readonly(1), set(1), sh(1), shift(1), sleep(1), suspend(1), test(1)test(1), test(1B), time(1), times(1), trap(1), typeset(1), umask(1), wait(1), chdir(2), chmod(2), creat(2), umask(2), getopt(3C), profile(4), environ(5) SunOS 5.11 20 Nov 2007 shell_builtins(1)
All times are GMT -4. The time now is 08:49 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy