Sponsored Content
Top Forums Shell Programming and Scripting getting rid of duplicate files Post 92388 by moxxx68 on Friday 9th of December 2005 04:58:21 PM
Old 12-09-2005
find ./path -print -exec basename {} \; | awk -v var=arr 'arr[$0]++;i >= 2; i$var' | xargs mv --target-directory=test..


this worked to a certain extent although I am getting some error
messages with the cp command and the basename is contingious to
all names of the same type (ex, file.txt-{1,2,3,4}) and I am not
sure if this gives the exact result as far as leaving one file
although i tried diff two test directories against each other it
seemed real close.. please leave affirmation of any correct syntax
used (if any?)..
moxxx68
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Renaming multiple files, to get rid of extension

I have a good script to rename multiple files, but what's the best way I can remove some text from multiple filenames? Say I have a directory with 35 files with a .XLS at the end, how can I rename them to remove the .XLS but keep everything the same, without having to mv manually. Thanks. (6 Replies)
Discussion started by: nj78
6 Replies

2. UNIX for Dummies Questions & Answers

Getting rid of files with no ownership

I am in the process of learning how to do system administration (just on my own Linux machine) and have been working with the find command. One of the things I tried was find / -nouser -o -nogroup I redirected the output of my find query into a text file, and when I did a wc -l on it, it... (1 Reply)
Discussion started by: kermit
1 Replies

3. Shell Programming and Scripting

Finding Duplicate files

How do you delete and and find duplicate files? (1 Reply)
Discussion started by: Jicom4
1 Replies

4. Shell Programming and Scripting

Getting Rid of Having to Write to Flat Files

Ok, so i've been having to write to flat files lately and then making my script read information from the flat file and then work off of that. i dont want to keep doing that because i believe it creates a mess. i like to keep my work all to one script instead of having that one script... (7 Replies)
Discussion started by: SkySmart
7 Replies

5. Shell Programming and Scripting

Find duplicate files

What utility do you recommend for simply finding all duplicate files among all files? (4 Replies)
Discussion started by: kiasas
4 Replies

6. UNIX for Dummies Questions & Answers

how to get rid of last _ in the files name?

ex: I have list of files in a folder. abc_def_geh_.txt abc_.txt abc_def_geh_12345_.txt ab134c_d345345ef_444geh_12345_.txt i need to rename all files to get rid of the _ before .txt result should look like this: abc_def_geh.txt abc.txt abc_def_geh_12345.txt... (2 Replies)
Discussion started by: lv99
2 Replies

7. Shell Programming and Scripting

Remove duplicate files

Hi, In a directory, e.g. ~/corpus is a lot of files and subdirectories. Some of the files are named: 12345___PP___0902___AA.txt 12346___PP___0902___AA. txt 12347___PP___0902___AA. txt The amount of files varies. I need to keep the highest (12347___PP___0902___AA. txt) and remove... (5 Replies)
Discussion started by: corfuitl
5 Replies

8. Shell Programming and Scripting

Duplicate files

Hi Gents, I have 1 files as seen below. 44571009 100 42381900 101 23482389 102 44571009 103 28849007 104 28765648 105 25689908 106 28765648 107 42381900 108 44571009 109 17298799 110 44571009 111 I would like to get something like it 44571009 100 103 109 111 (3 Replies)
Discussion started by: jiam912
3 Replies

9. Shell Programming and Scripting

Trying to get rid of a duplicate output line...

Hi folks, I'm trying to work on a script that will grab a router interface report and generate the numbers of "in use" and "un-used" ports per device. Right now, I've got a cut down of the report as follows: sing /usr/apps/siteName/etc/DCAFT-9K.cmds for send text Connecting using... (11 Replies)
Discussion started by: Marc G
11 Replies

10. Shell Programming and Scripting

Finds all duplicate files

Hi, How would you write bash script that given a directory as an argument and finds all duplicate files (with same contents - by using bytewise comparison) there and prints their names? (6 Replies)
Discussion started by: elior
6 Replies
File::Basename(3pm)					 Perl Programmers Reference Guide				       File::Basename(3pm)

NAME
File::Basename - Parse file paths into directory, filename and suffix. SYNOPSIS
use File::Basename; ($name,$path,$suffix) = fileparse($fullname,@suffixlist); $name = fileparse($fullname,@suffixlist); $basename = basename($fullname,@suffixlist); $dirname = dirname($fullname); DESCRIPTION
These routines allow you to parse file paths into their directory, filename and suffix. NOTE: "dirname()" and "basename()" emulate the behaviours, and quirks, of the shell and C functions of the same name. See each function's documentation for details. If your concern is just parsing paths it is safer to use File::Spec's "splitpath()" and "splitdir()" methods. It is guaranteed that # Where $path_separator is / for Unix, for Windows, etc... dirname($path) . $path_separator . basename($path); is equivalent to the original path for all systems but VMS. "fileparse" my($filename, $directories, $suffix) = fileparse($path); my($filename, $directories, $suffix) = fileparse($path, @suffixes); my $filename = fileparse($path, @suffixes); The "fileparse()" routine divides a file path into its $directories, $filename and (optionally) the filename $suffix. $directories contains everything up to and including the last directory separator in the $path including the volume (if applicable). The remainder of the $path is the $filename. # On Unix returns ("baz", "/foo/bar/", "") fileparse("/foo/bar/baz"); # On Windows returns ("baz", 'C:fooar', "") fileparse('C:fooaraz'); # On Unix returns ("", "/foo/bar/baz/", "") fileparse("/foo/bar/baz/"); If @suffixes are given each element is a pattern (either a string or a "qr//") matched against the end of the $filename. The matching portion is removed and becomes the $suffix. # On Unix returns ("baz", "/foo/bar/", ".txt") fileparse("/foo/bar/baz.txt", qr/.[^.]*/); If type is non-Unix (see "fileparse_set_fstype") then the pattern matching for suffix removal is performed case-insensitively, since those systems are not case-sensitive when opening existing files. You are guaranteed that "$directories . $filename . $suffix" will denote the same location as the original $path. "basename" my $filename = basename($path); my $filename = basename($path, @suffixes); This function is provided for compatibility with the Unix shell command basename(1). It does NOT always return the file name portion of a path as you might expect. To be safe, if you want the file name portion of a path use "fileparse()". "basename()" returns the last level of a filepath even if the last level is clearly directory. In effect, it is acting like "pop()" for paths. This differs from "fileparse()"'s behaviour. # Both return "bar" basename("/foo/bar"); basename("/foo/bar/"); @suffixes work as in "fileparse()" except all regex metacharacters are quoted. # These two function calls are equivalent. my $filename = basename("/foo/bar/baz.txt", ".txt"); my $filename = fileparse("/foo/bar/baz.txt", qr/Q.txtE/); Also note that in order to be compatible with the shell command, "basename()" does not strip off a suffix if it is identical to the remaining characters in the filename. "dirname" This function is provided for compatibility with the Unix shell command dirname(1) and has inherited some of its quirks. In spite of its name it does NOT always return the directory name as you might expect. To be safe, if you want the directory name of a path use "fileparse()". Only on VMS (where there is no ambiguity between the file and directory portions of a path) and AmigaOS (possibly due to an implementation quirk in this module) does "dirname()" work like "fileparse($path)", returning just the $directories. # On VMS and AmigaOS my $directories = dirname($path); When using Unix or MSDOS syntax this emulates the dirname(1) shell function which is subtly different from how "fileparse()" works. It returns all but the last level of a file path even if the last level is clearly a directory. In effect, it is not returning the directory portion but simply the path one level up acting like "chop()" for file paths. Also unlike "fileparse()", "dirname()" does not include a trailing slash on its returned path. # returns /foo/bar. fileparse() would return /foo/bar/ dirname("/foo/bar/baz"); # also returns /foo/bar despite the fact that baz is clearly a # directory. fileparse() would return /foo/bar/baz/ dirname("/foo/bar/baz/"); # returns '.'. fileparse() would return 'foo/' dirname("foo/"); Under VMS, if there is no directory information in the $path, then the current default device and directory is used. "fileparse_set_fstype" my $type = fileparse_set_fstype(); my $previous_type = fileparse_set_fstype($type); Normally File::Basename will assume a file path type native to your current operating system (ie. /foo/bar style on Unix, fooar on Windows, etc...). With this function you can override that assumption. Valid $types are "MacOS", "VMS", "AmigaOS", "OS2", "RISCOS", "MSWin32", "DOS" (also "MSDOS" for backwards bug compatibility), "Epoc" and "Unix" (all case-insensitive). If an unrecognized $type is given "Unix" will be assumed. If you've selected VMS syntax, and the file specification you pass to one of these routines contains a "/", they assume you are using Unix emulation and apply the Unix syntax rules instead, for that function call only. SEE ALSO
dirname(1), basename(1), File::Spec perl v5.18.2 2014-01-06 File::Basename(3pm)
All times are GMT -4. The time now is 05:06 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy