Sponsored Content
Top Forums Shell Programming and Scripting getting rid of duplicate files Post 92362 by moxxx68 on Friday 9th of December 2005 01:57:49 PM
Old 12-09-2005
too many occurances to rm manually!
will this work...

find ./path -iname "basename" | awk 'BEGIN{arr[$0]++}{var=[$0];var > 0; var++}END{i=[var++]}' | xargs mv --target-directory=dup-dir

%rm dup-dir

looks like it would work if i could use an array to parse the
occurances starting with occurance 1 instead of 0... obviously
this could be in any directory but the way the tree is configured
it doesn't really matter as long as i have one occurance left!

could please use some help...
moxxx68
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Renaming multiple files, to get rid of extension

I have a good script to rename multiple files, but what's the best way I can remove some text from multiple filenames? Say I have a directory with 35 files with a .XLS at the end, how can I rename them to remove the .XLS but keep everything the same, without having to mv manually. Thanks. (6 Replies)
Discussion started by: nj78
6 Replies

2. UNIX for Dummies Questions & Answers

Getting rid of files with no ownership

I am in the process of learning how to do system administration (just on my own Linux machine) and have been working with the find command. One of the things I tried was find / -nouser -o -nogroup I redirected the output of my find query into a text file, and when I did a wc -l on it, it... (1 Reply)
Discussion started by: kermit
1 Replies

3. Shell Programming and Scripting

Finding Duplicate files

How do you delete and and find duplicate files? (1 Reply)
Discussion started by: Jicom4
1 Replies

4. Shell Programming and Scripting

Getting Rid of Having to Write to Flat Files

Ok, so i've been having to write to flat files lately and then making my script read information from the flat file and then work off of that. i dont want to keep doing that because i believe it creates a mess. i like to keep my work all to one script instead of having that one script... (7 Replies)
Discussion started by: SkySmart
7 Replies

5. Shell Programming and Scripting

Find duplicate files

What utility do you recommend for simply finding all duplicate files among all files? (4 Replies)
Discussion started by: kiasas
4 Replies

6. UNIX for Dummies Questions & Answers

how to get rid of last _ in the files name?

ex: I have list of files in a folder. abc_def_geh_.txt abc_.txt abc_def_geh_12345_.txt ab134c_d345345ef_444geh_12345_.txt i need to rename all files to get rid of the _ before .txt result should look like this: abc_def_geh.txt abc.txt abc_def_geh_12345.txt... (2 Replies)
Discussion started by: lv99
2 Replies

7. Shell Programming and Scripting

Remove duplicate files

Hi, In a directory, e.g. ~/corpus is a lot of files and subdirectories. Some of the files are named: 12345___PP___0902___AA.txt 12346___PP___0902___AA. txt 12347___PP___0902___AA. txt The amount of files varies. I need to keep the highest (12347___PP___0902___AA. txt) and remove... (5 Replies)
Discussion started by: corfuitl
5 Replies

8. Shell Programming and Scripting

Duplicate files

Hi Gents, I have 1 files as seen below. 44571009 100 42381900 101 23482389 102 44571009 103 28849007 104 28765648 105 25689908 106 28765648 107 42381900 108 44571009 109 17298799 110 44571009 111 I would like to get something like it 44571009 100 103 109 111 (3 Replies)
Discussion started by: jiam912
3 Replies

9. Shell Programming and Scripting

Trying to get rid of a duplicate output line...

Hi folks, I'm trying to work on a script that will grab a router interface report and generate the numbers of "in use" and "un-used" ports per device. Right now, I've got a cut down of the report as follows: sing /usr/apps/siteName/etc/DCAFT-9K.cmds for send text Connecting using... (11 Replies)
Discussion started by: Marc G
11 Replies

10. Shell Programming and Scripting

Finds all duplicate files

Hi, How would you write bash script that given a directory as an argument and finds all duplicate files (with same contents - by using bytewise comparison) there and prints their names? (6 Replies)
Discussion started by: elior
6 Replies
MAKEHISTORY(8)						      System Manager's Manual						    MAKEHISTORY(8)

NAME
makehistory - tools to recover Usenet history database. SYNOPSIS
makehistory [ -A oldtmp ] [ -a active ] [ -b ] [ -f filename ] [ -i ] [ -n ] [ -o ] [ -r ] [ -s size ] [ -T tmpdir ] [ -u [ -v ] ] DESCRIPTION
Makehistory rebuilds the history(5) text file and the associated dbz(3) database. The default name of the text file is /var/lib/news/his- tory; to specify a different name, use the ``-f'' flag. Makehistory scans the active(5) file to determine which newsgroup directories within the spool directory, /var/spool/news, should be scanned. (If a group is removed, but its spool directory still exists, makehistory will ignore it.) The program reads each file found and writes a history line for it. After the text file is written, makehistory will build the dbz database. OPTIONS
-A If the ``-A'' flag is used then the argument given is the pathname makehistory can use to store a copy of the history file as it's being built. It will be appended to, so existing data will not be lost (and so should be valid history entries). -a If the ``-a'' flag is given then the argument is the active file to use rather than the default one of /var/lib/news/active. -b If the ``-b'' flag is used, then makehistory will remove any articles that do not have valid Message-ID headers in them. -f If the ``-f'' flag is used, then the database files are named file.dir and file.pag. If the ``-f'' flag is not used, then a tempo- rary link to the name history.n is made and the database files are written as history.n.pag and history.n.dir. -o If the ``-o'' flag is used, then the link is not made and any existing history files are overwritten. If the old database exists, makehistory will use it to determine the size of the new database. -i To ignore the old database use the ``-i'' flag. Using the ``-o'' flag implies the ``-i'' flag. -s The program will also ignore any old database if the ``-s'' flag is used to specify the approximate number of entries in the new database. Accurately specifying the size is an optimization that will create a more efficient database. (The size should be the estimated eventual size of the file, typically the size of the old file.) For more information, see the discussion of dbzfresh and dbzsize in dbz(3). -u If the ``-u'' flag is given, then makehistory assumes that innd is running. It will pause the server while scanning, and then send ``addhist'' commands (see ctlinnd(8)) to the server for any article that is not found in the dbz database. The command ``makehis- tory -bu'' is useful after a system crash, to delete any mangled articles and bring the article database back into a more consistent state. -v If the ``-v'' flag is used with the ``-u'' flag, then makehistory will put a copy of all added lines on its standard output. -n To scan the spool directory without rebuilding the dbz files, use the ``-n'' flag. If used with ``-u'', the server will not be paused while scanning. -r To just build the dbz files from an existing text file, use the ``-r'' flag. The ``-i'' or ``-s'' flags can be useful if there are no valid dbz files to use. -T Makehistory needs to create a temporary file that contains one line for each article it finds, which can become very large. This file is created in the /var/spool/news/in.coming/tmp directory. The ``TMPDIR'' environment variable may be used to specify a dif- ferent directory. Alternatively, the ``-T'' flag may be used to specify a temporary directory. In addition, the sort(1) that is invoked during the build writes large temporary files (often to /var/tmp but see your system manpages). If the ``-T'' flag is used, then the flag and its value will be passed to sort. On most systems this will change the temporary directory that sort uses. if used, this flag and its value will be passed on to the sort(1) command that is invoked during the build. EXAMPLES
A typical way to use this program is with the following /bin/sh commands: ctlinnd throttle "Rebuilding history file" cd /var/lib/news if makehistory -n -f history.n ; then : else echo Error creating history file! exit 1 fi # The following line can be used to retain expired history # It is not necessary for the history file to be sorted. # awk 'NF==2 { print; }' <history >>history.n # View history file for mistakes. if makehistory -r -s `wc -l <history` -f history.n; then mv history.n history mv history.n.dir history.dir mv history.n.pag history.pag fi ctlinnd go '' BUGS AND LIMITATIONS
Makehistory does not handle symbolic links. If the news spool area is split across multiple partitions, the following commands should probably be run before the database is regenerated: cd /var/spool/news find . -type l -name '[1-9]*' -print | xargs -t rm Make sure to run the command on all the appropriate partitions! HISTORY
Written by Rich $alz <rsalz@uunet.uu.net> for InterNetNews. This is revision 1.3, dated 1996/11/26. SEE ALSO
active(5), ctlinnd(8), dbz(3), filechan(8), history(5), innd(8), newsfeeds(5), makeactive(8), newsrequeue(8). MAKEHISTORY(8)
All times are GMT -4. The time now is 05:19 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy