Sponsored Content
Top Forums Shell Programming and Scripting getting rid of duplicate files Post 92339 by jim mcnamara on Friday 9th of December 2005 11:27:14 AM
Old 12-09-2005
Start with something like this to find actual duplicated names.
Then use the file to find the paths to get full file names.
Code:
find /path -print -exec basename {} \; | awk 'arr[$0]++' > file

 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Renaming multiple files, to get rid of extension

I have a good script to rename multiple files, but what's the best way I can remove some text from multiple filenames? Say I have a directory with 35 files with a .XLS at the end, how can I rename them to remove the .XLS but keep everything the same, without having to mv manually. Thanks. (6 Replies)
Discussion started by: nj78
6 Replies

2. UNIX for Dummies Questions & Answers

Getting rid of files with no ownership

I am in the process of learning how to do system administration (just on my own Linux machine) and have been working with the find command. One of the things I tried was find / -nouser -o -nogroup I redirected the output of my find query into a text file, and when I did a wc -l on it, it... (1 Reply)
Discussion started by: kermit
1 Replies

3. Shell Programming and Scripting

Finding Duplicate files

How do you delete and and find duplicate files? (1 Reply)
Discussion started by: Jicom4
1 Replies

4. Shell Programming and Scripting

Getting Rid of Having to Write to Flat Files

Ok, so i've been having to write to flat files lately and then making my script read information from the flat file and then work off of that. i dont want to keep doing that because i believe it creates a mess. i like to keep my work all to one script instead of having that one script... (7 Replies)
Discussion started by: SkySmart
7 Replies

5. Shell Programming and Scripting

Find duplicate files

What utility do you recommend for simply finding all duplicate files among all files? (4 Replies)
Discussion started by: kiasas
4 Replies

6. UNIX for Dummies Questions & Answers

how to get rid of last _ in the files name?

ex: I have list of files in a folder. abc_def_geh_.txt abc_.txt abc_def_geh_12345_.txt ab134c_d345345ef_444geh_12345_.txt i need to rename all files to get rid of the _ before .txt result should look like this: abc_def_geh.txt abc.txt abc_def_geh_12345.txt... (2 Replies)
Discussion started by: lv99
2 Replies

7. Shell Programming and Scripting

Remove duplicate files

Hi, In a directory, e.g. ~/corpus is a lot of files and subdirectories. Some of the files are named: 12345___PP___0902___AA.txt 12346___PP___0902___AA. txt 12347___PP___0902___AA. txt The amount of files varies. I need to keep the highest (12347___PP___0902___AA. txt) and remove... (5 Replies)
Discussion started by: corfuitl
5 Replies

8. Shell Programming and Scripting

Duplicate files

Hi Gents, I have 1 files as seen below. 44571009 100 42381900 101 23482389 102 44571009 103 28849007 104 28765648 105 25689908 106 28765648 107 42381900 108 44571009 109 17298799 110 44571009 111 I would like to get something like it 44571009 100 103 109 111 (3 Replies)
Discussion started by: jiam912
3 Replies

9. Shell Programming and Scripting

Trying to get rid of a duplicate output line...

Hi folks, I'm trying to work on a script that will grab a router interface report and generate the numbers of "in use" and "un-used" ports per device. Right now, I've got a cut down of the report as follows: sing /usr/apps/siteName/etc/DCAFT-9K.cmds for send text Connecting using... (11 Replies)
Discussion started by: Marc G
11 Replies

10. Shell Programming and Scripting

Finds all duplicate files

Hi, How would you write bash script that given a directory as an argument and finds all duplicate files (with same contents - by using bytewise comparison) there and prints their names? (6 Replies)
Discussion started by: elior
6 Replies
FIND2PERL(1)						 Perl Programmers Reference Guide					      FIND2PERL(1)

NAME
find2perl - translate find command lines to Perl code SYNOPSIS
find2perl [paths] [predicates] | perl DESCRIPTION
find2perl is a little translator to convert find command lines to equivalent Perl code. The resulting code is typically faster than running find itself. "paths" are a set of paths where find2perl will start its searches and "predicates" are taken from the following list. "! PREDICATE" Negate the sense of the following predicate. The "!" must be passed as a distinct argument, so it may need to be surrounded by whitespace and/or quoted from interpretation by the shell using a backslash (just as with using find(1)). "( PREDICATES )" Group the given PREDICATES. The parentheses must be passed as distinct arguments, so they may need to be surrounded by whitespace and/or quoted from interpretation by the shell using a backslash (just as with using find(1)). "PREDICATE1 PREDICATE2" True if _both_ PREDICATE1 and PREDICATE2 are true; PREDICATE2 is not evaluated if PREDICATE1 is false. "PREDICATE1 -o PREDICATE2" True if either one of PREDICATE1 or PREDICATE2 is true; PREDICATE2 is not evaluated if PREDICATE1 is true. "-follow" Follow (dereference) symlinks. The checking of file attributes depends on the position of the "-follow" option. If it precedes the file check option, an "stat" is done which means the file check applies to the file the symbolic link is pointing to. If "-follow" option follows the file check option, this now applies to the symbolic link itself, i.e. an "lstat" is done. "-depth" Change directory traversal algorithm from breadth-first to depth-first. "-prune" Do not descend into the directory currently matched. "-xdev" Do not traverse mount points (prunes search at mount-point directories). "-name GLOB" File name matches specified GLOB wildcard pattern. GLOB may need to be quoted to avoid interpretation by the shell (just as with using find(1)). "-iname GLOB" Like "-name", but the match is case insensitive. "-path GLOB" Path name matches specified GLOB wildcard pattern. "-ipath GLOB" Like "-path", but the match is case insensitive. "-perm PERM" Low-order 9 bits of permission match octal value PERM. "-perm -PERM" The bits specified in PERM are all set in file's permissions. "-type X" The file's type matches perl's "-X" operator. "-fstype TYPE" Filesystem of current path is of type TYPE (only NFS/non-NFS distinction is implemented). "-user USER" True if USER is owner of file. "-group GROUP" True if file's group is GROUP. "-nouser" True if file's owner is not in password database. "-nogroup" True if file's group is not in group database. "-inum INUM" True file's inode number is INUM. "-links N" True if (hard) link count of file matches N (see below). "-size N" True if file's size matches N (see below) N is normally counted in 512-byte blocks, but a suffix of "c" specifies that size should be counted in characters (bytes) and a suffix of "k" specifies that size should be counted in 1024-byte blocks. "-atime N" True if last-access time of file matches N (measured in days) (see below). "-ctime N" True if last-changed time of file's inode matches N (measured in days, see below). "-mtime N" True if last-modified time of file matches N (measured in days, see below). "-newer FILE" True if last-modified time of file matches N. "-print" Print out path of file (always true). If none of "-exec", "-ls", "-print0", or "-ok" is specified, then "-print" will be added implicitly. "-print0" Like -print, but terminates with instead of . "-exec OPTIONS ;" exec() the arguments in OPTIONS in a subprocess; any occurrence of {} in OPTIONS will first be substituted with the path of the current file. Note that the command "rm" has been special-cased to use perl's unlink() function instead (as an optimization). The ";" must be passed as a distinct argument, so it may need to be surrounded by whitespace and/or quoted from interpretation by the shell using a backslash (just as with using find(1)). "-ok OPTIONS ;" Like -exec, but first prompts user; if user's response does not begin with a y, skip the exec. The ";" must be passed as a distinct argument, so it may need to be surrounded by whitespace and/or quoted from interpretation by the shell using a backslash (just as with using find(1)). "-eval EXPR" Has the perl script eval() the EXPR. "-ls" Simulates "-exec ls -dils {} ;" "-tar FILE" Adds current output to tar-format FILE. "-cpio FILE" Adds current output to old-style cpio-format FILE. "-ncpio FILE" Adds current output to "new"-style cpio-format FILE. Predicates which take a numeric argument N can come in three forms: * N is prefixed with a +: match values greater than N * N is prefixed with a -: match values less than N * N is not prefixed with either + or -: match only values equal to N SEE ALSO
find, File::Find. perl v5.16.2 2013-08-25 FIND2PERL(1)
All times are GMT -4. The time now is 04:42 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy