Sponsored Content
Top Forums Shell Programming and Scripting Remove Duplicate Files On Remote Servers Post 302402254 by tokland on Tuesday 9th of March 2010 08:54:25 AM
Old 03-09-2010
Quote:
Originally Posted by jaysunn
I am comparing files over ssh to remove the file from the source server directory if a match occurs.
You are connecting to the server for each file, this is slooooow; you should connect only one time. There are lots of ways of doing it, that's one:

Code:
ssh server ls remotedir | ( cd localdir && xargs -d"\n" rm )

Anyway, this whole idea is a bit weird. If this involves syncing files, rsync is the right tool.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

remove duplicate files in a directory

Hi ppl. I have to check for duplicate files in a directory . the directory has following files /the/folder /containing/the/file a1.yyyymmddhhmmss a1.yyyyMMddhhmmss b1.yyyymmddhhmmss b2.yyyymmddhhmmss c.yyyymmddhhmmss d.yyyymmddhhmmss d.yyyymmddhhmmss where the date time stamp can be... (1 Reply)
Discussion started by: asinha63
1 Replies

2. Shell Programming and Scripting

remove all duplicate lines from all files in one folder

Hi, is it possible to remove all duplicate lines from all txt files in a specific folder? This is too hard for me maybe someone could help. lets say we have an amount of textfiles 1 or 2 or 3 or... maximum 50 each textfile has lines with text. I want all lines of all textfiles... (8 Replies)
Discussion started by: lowmaster
8 Replies

3. Shell Programming and Scripting

Remove duplicate files based on text string?

Hi I have been struggling with a script for removing duplicate messages from a shared mailbox. I would like to search for duplicate messages based on the “Message-ID” string within the messages files. I have managed to find the duplicate “Message-ID” strings and (if I would like) delete... (1 Reply)
Discussion started by: spangberg
1 Replies

4. Shell Programming and Scripting

Remove duplicate files in same directory

Hi all. Am doing continuous backup of mailboxes using rsync. So whenever a new mail arrives it is automatically copied on backup server. When a new mail arrives it is named as xyz:2, when it is read by the email client an S is appended xyz:2,S Eventually , 2 copies of the same file exist on... (7 Replies)
Discussion started by: coolatt
7 Replies

5. Shell Programming and Scripting

perl/shell need help to remove duplicate lines from files

Dear All, I have multiple files having number of records, consist of more than 10 columns some column values are duplicate and i want to remove these duplicate values from these files. Duplicate values may come in different files.... all files laying in single directory.. Need help to... (3 Replies)
Discussion started by: arvindng
3 Replies

6. Shell Programming and Scripting

how to delete files on two remote servers simultaneously?

dear all, i'm preparing a script which can do these actions : 1. stop remove server's certain service 2. clean the files on remote servers simultaneously (because lots of files need to be deleted) 3. after files/logs are removed, restart the service again i'm stuck on how to clean remote... (4 Replies)
Discussion started by: tiger2000
4 Replies

7. Shell Programming and Scripting

Remove duplicate files

Hi, In a directory, e.g. ~/corpus is a lot of files and subdirectories. Some of the files are named: 12345___PP___0902___AA.txt 12346___PP___0902___AA. txt 12347___PP___0902___AA. txt The amount of files varies. I need to keep the highest (12347___PP___0902___AA. txt) and remove... (5 Replies)
Discussion started by: corfuitl
5 Replies

8. UNIX for Dummies Questions & Answers

Help with Copying files between two remote servers

Hi All, Please help me for a shell. I am a New to unix I am trying to DB dump file from one server and copying it to another server. From My Local ServerA connecting to remote ServerB using ssh and taking dump of a instance. That Dump file i need to copy to ServerC. I am able to connect... (6 Replies)
Discussion started by: maddyd2k
6 Replies

9. Windows & DOS: Issues & Discussions

Remove duplicate lines from text files.

So, I have text files, one "fail.txt" And one "color.txt" I now want to use a command line (DOS) to remove ANY line that is PRESENT IN BOTH from each text file. Afterwards there shall be no duplicate lines. (1 Reply)
Discussion started by: pasc
1 Replies

10. Shell Programming and Scripting

Remove all but newest two files (Not a duplicate post)

TARGET_DIR='/media/andy/MAXTOR_SDB1/Ubuntu_Mate_18.04/' REGEX='{4}-{2}-{2}_{2}:{2}' # regular expression that match to: date '+%Y-%m-%d_%H:%M' LATEST_FILE="$(ls "$TARGET_DIR" | egrep "^${REGEX}$" | tail -1)" find "$TARGET_DIR" ! -name "$LATEST_FILE" -type f -regextype egrep -regex... (7 Replies)
Discussion started by: drew77
7 Replies
vzmigrate(8)							    Containers							      vzmigrate(8)

NAME
vzmigrate - migrate a container between two OpenVZ servers SYNOPSIS
vzmigrate [-r|--remove-area yes|no] [--ssh=ssh_options] [--rsync=rsync_options] [--keep-dst] [--online] [-v] destination_address CTID DESCRIPTION
This utility is used to migrate a container from one (source) Hardware Node (HN) to another (destination) HN. The utility can migrate either stopped or running container. For a stopped container, simple CT private area transfer is performed (rsync(1) is used for file transfer). For running containers, migration may be offline (default) or online. This program uses ssh as a transport layer. You will need to put ssh public key to destination node and be able to connect to node without entering password. OPTIONS
-r, --remove-area yes | no Whether to remove a container area on source HN for the successfully migrated container. Default is yes. --ssh=options Additional options that will be passed to ssh while establishing connection to destination HN. --rsync=options Additional options that will be passed to rsync(8). You may add options like -z to enable data compression if you are migrating over a slow link. --keep-dst Do not clean synced destination container private area in case of some error. It makes sense to use this option on big container migration to avoid syncing container private area again in case some error (on container stop for example) occurs during first migration attempt. --online Perform online (zero-downtime) migration: during the migration the container hangs for a while and after the migration it continues working as though nothing has happened. -v Verbose mode. Causes vzmigrate to print debugging messages about its progress. Multiple -v options increase the verbosity. The maximum is 3. EXAMPLES
Migration of CT 101 to 192.168.1.130 with downtime: vzmigrate 192.168.1.130 101 Online migration of CT 102 to 192.168.1.130: vzmigrate --online 192.168.1.130 102 EXIT STATUS
0 EXIT_OK Command completed successfully. 1 EXIT_USAGE Bad command line options. 2 EXIT_VE_STOPPED Container is stopped. 4 EXIT_CONNECT Can't connect to destination (source) HN. 6 EXIT_COPY Container private area copying/moving failed. 7 EXIT_VE_START Can't start or restore destination CT. 8 EXIT_VE_STOP Can't stop or checkpoint source CT. 9 EXIT_EXISTS Container already exists on destination HN. 10 EXIT_NOTEXIST Container does not exists on source HN. 12 EXIT_IP_INUSE You attempt to migrate CT which IP address(es) are already in use on the destination node. 13 EXIT_QUOTA Operation with CT quota failed. SEE ALSO
rsync(1). COPYRIGHT
Copyright (C) 2001-2010, Parallels, Inc. Licensed under GNU GPL. OpenVZ 28 Jun 2011 vzmigrate(8)
All times are GMT -4. The time now is 11:18 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy