Sponsored Content
Top Forums Shell Programming and Scripting Remove Duplicate Files On Remote Servers Post 302402254 by tokland on Tuesday 9th of March 2010 08:54:25 AM
Old 03-09-2010
Quote:
Originally Posted by jaysunn
I am comparing files over ssh to remove the file from the source server directory if a match occurs.
You are connecting to the server for each file, this is slooooow; you should connect only one time. There are lots of ways of doing it, that's one:

Code:
ssh server ls remotedir | ( cd localdir && xargs -d"\n" rm )

Anyway, this whole idea is a bit weird. If this involves syncing files, rsync is the right tool.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

remove duplicate files in a directory

Hi ppl. I have to check for duplicate files in a directory . the directory has following files /the/folder /containing/the/file a1.yyyymmddhhmmss a1.yyyyMMddhhmmss b1.yyyymmddhhmmss b2.yyyymmddhhmmss c.yyyymmddhhmmss d.yyyymmddhhmmss d.yyyymmddhhmmss where the date time stamp can be... (1 Reply)
Discussion started by: asinha63
1 Replies

2. Shell Programming and Scripting

remove all duplicate lines from all files in one folder

Hi, is it possible to remove all duplicate lines from all txt files in a specific folder? This is too hard for me maybe someone could help. lets say we have an amount of textfiles 1 or 2 or 3 or... maximum 50 each textfile has lines with text. I want all lines of all textfiles... (8 Replies)
Discussion started by: lowmaster
8 Replies

3. Shell Programming and Scripting

Remove duplicate files based on text string?

Hi I have been struggling with a script for removing duplicate messages from a shared mailbox. I would like to search for duplicate messages based on the “Message-ID” string within the messages files. I have managed to find the duplicate “Message-ID” strings and (if I would like) delete... (1 Reply)
Discussion started by: spangberg
1 Replies

4. Shell Programming and Scripting

Remove duplicate files in same directory

Hi all. Am doing continuous backup of mailboxes using rsync. So whenever a new mail arrives it is automatically copied on backup server. When a new mail arrives it is named as xyz:2, when it is read by the email client an S is appended xyz:2,S Eventually , 2 copies of the same file exist on... (7 Replies)
Discussion started by: coolatt
7 Replies

5. Shell Programming and Scripting

perl/shell need help to remove duplicate lines from files

Dear All, I have multiple files having number of records, consist of more than 10 columns some column values are duplicate and i want to remove these duplicate values from these files. Duplicate values may come in different files.... all files laying in single directory.. Need help to... (3 Replies)
Discussion started by: arvindng
3 Replies

6. Shell Programming and Scripting

how to delete files on two remote servers simultaneously?

dear all, i'm preparing a script which can do these actions : 1. stop remove server's certain service 2. clean the files on remote servers simultaneously (because lots of files need to be deleted) 3. after files/logs are removed, restart the service again i'm stuck on how to clean remote... (4 Replies)
Discussion started by: tiger2000
4 Replies

7. Shell Programming and Scripting

Remove duplicate files

Hi, In a directory, e.g. ~/corpus is a lot of files and subdirectories. Some of the files are named: 12345___PP___0902___AA.txt 12346___PP___0902___AA. txt 12347___PP___0902___AA. txt The amount of files varies. I need to keep the highest (12347___PP___0902___AA. txt) and remove... (5 Replies)
Discussion started by: corfuitl
5 Replies

8. UNIX for Dummies Questions & Answers

Help with Copying files between two remote servers

Hi All, Please help me for a shell. I am a New to unix I am trying to DB dump file from one server and copying it to another server. From My Local ServerA connecting to remote ServerB using ssh and taking dump of a instance. That Dump file i need to copy to ServerC. I am able to connect... (6 Replies)
Discussion started by: maddyd2k
6 Replies

9. Windows & DOS: Issues & Discussions

Remove duplicate lines from text files.

So, I have text files, one "fail.txt" And one "color.txt" I now want to use a command line (DOS) to remove ANY line that is PRESENT IN BOTH from each text file. Afterwards there shall be no duplicate lines. (1 Reply)
Discussion started by: pasc
1 Replies

10. Shell Programming and Scripting

Remove all but newest two files (Not a duplicate post)

TARGET_DIR='/media/andy/MAXTOR_SDB1/Ubuntu_Mate_18.04/' REGEX='{4}-{2}-{2}_{2}:{2}' # regular expression that match to: date '+%Y-%m-%d_%H:%M' LATEST_FILE="$(ls "$TARGET_DIR" | egrep "^${REGEX}$" | tail -1)" find "$TARGET_DIR" ! -name "$LATEST_FILE" -type f -regextype egrep -regex... (7 Replies)
Discussion started by: drew77
7 Replies
ssh-http-proxy-connect(1)					   User Commands					 ssh-http-proxy-connect(1)

NAME
ssh-http-proxy-connect - Secure Shell proxy for HTTP SYNOPSIS
/usr/lib/ssh/ssh-http-proxy-connect [-h http_proxy_host] [-p http_proxy_port] connect_host connect_port DESCRIPTION
A proxy command for ssh(1) that uses HTTP CONNECT. Typical use is where connections external to a network are only allowed via a proxy web server. OPTIONS
The following options are supported: -h http_proxy_host Specifies the proxy web server through which to connect. Overrides the HTTPPROXY and http_proxy environment vari- ables if they are set. -p http_proxy_port Specifies the port on which the proxy web server runs. If not specified, port 80 is assumed. Overrides the HTTP- PROXYPORT and http_proxy environment variables if they are set. OPERANDS
The following operands are supported: http_proxy_host The host name or IP address (IPv4 or IPv6) of the proxy. http_proxy_port The numeric port number to connect to on http_proxy_host. connect_host The name of the remote host to which the proxy web server is to connect you. connect_port The numeric port number of the proxy web server to connect you to on http_proxy_host. EXAMPLES
The recommended way to use a proxy connection command is to configure the ProxyCommand in ssh_config(4) (see Example 1 and Example 2). Example 3 shows how the proxy command can be specified on the command line when running ssh(1). Example 1: Setting the proxy from the environment The following example uses ssh-http-proxy-connect in ssh_config(4) when the proxy is set from the environment: Host playtime.foo.com ProxyCommand /usr/lib/ssh/ssh-http-proxy-connect playtime.foo.com 22 Example 2: Overriding proxy environment variables The following example uses ssh-http-proxy-connect in ssh_config(4) to override (or if not set) proxy environment variables: Host playtime.foo.com ProxyCommand /usr/lib/ssh/ssh-http-proxy-connect -h webcache -p 8080 playtime.foo.com 22 Example 3: Using the command line The following example uses ssh-http-proxy-connect from the ssh(1) command line: example$ ssh -o'ProxyCommand="/usr/lib/ssh/ssh-http-proxy-connect -h webcache -p 8080 playtime.foo.com 22"' playtime.foo.com ENVIRONMENT VARIABLES
HTTPPROXY Takes the http_proxy_host operand to specify the default proxy host. Overrides http_proxy if both are set. HTTPPROXYPORT Takes the http_proxy_port operand to specify the default proxy port. Ignored if HTTPPROXY is not set. http_proxy URL format for specifying proxy host and port. EXIT STATUS
The following exit values are returned: 0 Successful completion. 1 An error occurred. ATTRIBUTES
See attributes(5) for descriptions of the following attributes: +-----------------------------+-----------------------------+ | ATTRIBUTE TYPE | ATTRIBUTE VALUE | +-----------------------------+-----------------------------+ |Availability |SUNWsshu | +-----------------------------+-----------------------------+ |Interface Stability |Stable | +-----------------------------+-----------------------------+ SEE ALSO
ssh(1), ssh-socks5-proxy-connect(1), ssh_config(4), attributes(5) SunOS 5.10 24 Oct 2001 ssh-http-proxy-connect(1)
All times are GMT -4. The time now is 05:59 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy