Sponsored Content
Top Forums Shell Programming and Scripting Remove duplicate files in same directory Post 302391935 by coolatt on Wednesday 3rd of February 2010 01:14:44 AM
Old 02-03-2010
Remove duplicate files in same directory

Hi all.

Am doing continuous backup of mailboxes using rsync.
So whenever a new mail arrives it is automatically copied on backup server.
When a new mail arrives it is named as xyz:2, when it is read by the email client an S is appended xyz:2,S
Eventually , 2 copies of the same file exist on the backup server with different names while on the Mailserver only xyz:2,S exists.

e.g
on mail server:
xyz:2,RS

on backup server:
xyz:2,
xyz:2,S
xyz:2,RS

So in 1 directory i can have 3 copies of file xyz & 2 copies of file abc.
Can anyone help how i can remove the oldest files(xyz:2,) and keep only the most recent one on the backup server ?

ThanksSmilie
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

remove duplicate files in a directory

Hi ppl. I have to check for duplicate files in a directory . the directory has following files /the/folder /containing/the/file a1.yyyymmddhhmmss a1.yyyyMMddhhmmss b1.yyyymmddhhmmss b2.yyyymmddhhmmss c.yyyymmddhhmmss d.yyyymmddhhmmss d.yyyymmddhhmmss where the date time stamp can be... (1 Reply)
Discussion started by: asinha63
1 Replies

2. Shell Programming and Scripting

script that detects duplicate files in directory

I need help with a script which accepts one argument and goes through all the files under a directory and prints a list of possible duplicate files As its output, it prints zero or more lines, each one containing a space-separated list of filenames. All the files listed on one line have the same... (1 Reply)
Discussion started by: trueman82
1 Replies

3. Shell Programming and Scripting

remove all duplicate lines from all files in one folder

Hi, is it possible to remove all duplicate lines from all txt files in a specific folder? This is too hard for me maybe someone could help. lets say we have an amount of textfiles 1 or 2 or 3 or... maximum 50 each textfile has lines with text. I want all lines of all textfiles... (8 Replies)
Discussion started by: lowmaster
8 Replies

4. Shell Programming and Scripting

Remove duplicate files based on text string?

Hi I have been struggling with a script for removing duplicate messages from a shared mailbox. I would like to search for duplicate messages based on the “Message-ID” string within the messages files. I have managed to find the duplicate “Message-ID” strings and (if I would like) delete... (1 Reply)
Discussion started by: spangberg
1 Replies

5. Shell Programming and Scripting

Remove Duplicate Files On Remote Servers

Hello, I wrote a basic script that works however I am was wondering if it could be sped up. I am comparing files over ssh to remove the file from the source server directory if a match occurs. Please Advise me on my mistakes. #!/bin/bash for file in `ls /export/home/podcast2/"$1" ` ; do ... (5 Replies)
Discussion started by: jaysunn
5 Replies

6. Shell Programming and Scripting

perl/shell need help to remove duplicate lines from files

Dear All, I have multiple files having number of records, consist of more than 10 columns some column values are duplicate and i want to remove these duplicate values from these files. Duplicate values may come in different files.... all files laying in single directory.. Need help to... (3 Replies)
Discussion started by: arvindng
3 Replies

7. Shell Programming and Scripting

[uniq + awk?] How to remove duplicate blocks of lines in files?

Hello again, I am wanting to remove all duplicate blocks of XML code in a file. This is an example: input: <string-array name="threeItems"> <item>item1</item> <item>item2</item> <item>item3</item> </string-array> <string-array name="twoItems"> <item>item1</item> <item>item2</item>... (19 Replies)
Discussion started by: raidzero
19 Replies

8. Shell Programming and Scripting

Remove duplicate files

Hi, In a directory, e.g. ~/corpus is a lot of files and subdirectories. Some of the files are named: 12345___PP___0902___AA.txt 12346___PP___0902___AA. txt 12347___PP___0902___AA. txt The amount of files varies. I need to keep the highest (12347___PP___0902___AA. txt) and remove... (5 Replies)
Discussion started by: corfuitl
5 Replies

9. Windows & DOS: Issues & Discussions

Remove duplicate lines from text files.

So, I have text files, one "fail.txt" And one "color.txt" I now want to use a command line (DOS) to remove ANY line that is PRESENT IN BOTH from each text file. Afterwards there shall be no duplicate lines. (1 Reply)
Discussion started by: pasc
1 Replies

10. Shell Programming and Scripting

Remove all but newest two files (Not a duplicate post)

TARGET_DIR='/media/andy/MAXTOR_SDB1/Ubuntu_Mate_18.04/' REGEX='{4}-{2}-{2}_{2}:{2}' # regular expression that match to: date '+%Y-%m-%d_%H:%M' LATEST_FILE="$(ls "$TARGET_DIR" | egrep "^${REGEX}$" | tail -1)" find "$TARGET_DIR" ! -name "$LATEST_FILE" -type f -regextype egrep -regex... (7 Replies)
Discussion started by: drew77
7 Replies
nisserver(1M)						  System Administration Commands					     nisserver(1M)

NAME
nisserver - set up NIS+ servers SYNOPSIS
/usr/lib/nis/nisserver -r [-x] [-f] [-v] [-Y] [ -d NIS+_domain] [ -g NIS+_groupname] [-l network_passwd] /usr/lib/nis/nisserver -M [-x] [-f] [-v] [-Y] -d NIS+_domain [ -g NIS+_groupname] [ -h NIS+_server_host] /usr/lib/nis/nisserver -R [-x] [-f] [-v] [-Y] [ -d NIS+_domain] [ -h NIS+_server_host] DESCRIPTION
The nisserver shell script can be used to set up a root master, non-root master, and replica NIS+ server with level 2 security (DES). If other authentication mechanisms are configured with nisauthconf(1M), nisserver will set up a NIS+ server using those mechanisms. nisauth- conf(1M) should be used before nisserver. When setting up a new domain, this script creates the NIS+ directories (including groups_dir and org_dir) and system table objects for the domain specified. It does not populate the tables. nispopulate(1M) must be used to populate the tables. OPTIONS
-d NIS+_domain Specifies the name for the NIS+ domain. The default is your local domain. -f Forces the NIS+ server setup without prompting for confirmation. -g NIS+_groupname Specifies the NIS+ group name for the new domain. This option is not valid with -R option. The default group is admin.<domain>. -h NIS+_server_host Specifies the hostname for the NIS+ server. It must be a valid host in the local domain. Use a fully qualified hostname (for example, hostx.xyz.sun.com.) to specify a host outside of your local domain. This option is only used for setting up non-root master or replica servers. The default for non-root master server setup is to use the same list of servers as the parent domain. The default for replica server setup is the local hostname. -l network_password Specifies the network password with which to create the credentials for the root master server. This option is only used for master root server setup (-r option). If this option is not specified, the script prompts you for the login password. -M Sets up the specified host as a master server. Make sure that rpc.nisd(1M) is running on the new master server before this command is executed. -R Sets up the specified host as a replica server. Make sure that rpc.nisd is running on the new replica server. -r Sets up the server as a root master server. Use the -R option to set up a root replica server. -v Runs the script in verbose mode. -x Turns the echo mode on. The script just prints the commands that it would have executed. Note that the commands are not actually executed. The default is off. -Y Sets up a NIS+ server with NIS-compatibility mode. The default is to set up the server without NIS-compatibility mode. USAGE
Use the first synopsis of the command (-r) to set up a root master server. To run the command, you must be logged in as super-user on the server machine. Use the second synopsis of the command (-M) to set up a non-root master server for the specified domain. To run the command, you must be logged in as a NIS+ principal on a NIS+ machine and have write permission to the parent directory of the domain that you are setting up. The new non-root master server machine must already be an NIS+ client (see nisclient(1M)) and have the rpc.nisd(1M) daemon running. Use the third synopsis of the command (-R) to set up a replica server for both root and non-root domains. To run the command, you must be logged in as a NIS+ principal on a NIS+ machine and have write permission to the parent directory of the domain that you are replicating. The new non-root replica server machine must already be an NIS+ client and have the rpc.nisd daemon running. EXAMPLES
Example 1: Setting up Servers To set up a root master server for domain sun.com.: root_server# /usr/lib/nis/nisserver -r -d sun.com. For the following examples make sure that the new servers are NIS+ clients and that rpc.nisd is running on these hosts before executing nisserver. To set up a replica server for the sun.com. domain on host sunreplica: root_server# /usr/lib/nis/nisserver -R -d sun.com. -h sunrep To set up a non-root master server for domain xyz.sun.com. on host sunxyz with the NIS+ groupname as admin-mgr.xyz.sun.com.: root_server# /usr/lib/nis/nisserver -M -d xyz.sun.com. -h sunxyz -g admin-mgr.xyz.sun.com. To set up a non-root replica server for domain xyz.sun.com. on host sunabc: sunxyz# /usr/lib/nis/nisserver -R -d xyz.sun.com. -h sunabc ATTRIBUTES
See attributes(5) for descriptions of the following attributes: +-----------------------------+-----------------------------+ | ATTRIBUTE TYPE | ATTRIBUTE VALUE | +-----------------------------+-----------------------------+ |Availability |SUNWcsu | +-----------------------------+-----------------------------+ SEE ALSO
nis+(1), nisgrpadm(1), nismkdir(1), nisaddcred(1M), nisauthconf(1M), nisclient(1M), nisinit(1M), nispopulate(1M), nisprefadm(1M), nis- setup(1M), rpc.nisd(1M), attributes(5) NOTES
NIS+ might not be supported in future releases of the SolarisTM Operating Environment. Tools to aid the migration from NIS+ to LDAP are available in the Solaris 9 operating environment. For more information, visit http://www.sun.com/directory/nisplus/transition.html. SunOS 5.10 13 Dec 2001 nisserver(1M)
All times are GMT -4. The time now is 02:09 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy