Sponsored Content
Top Forums UNIX for Beginners Questions & Answers Automate splitting of files , scp files as each split completes and combine files on target server Post 303033428 by malaika on Friday 5th of April 2019 07:50:49 AM
Old 04-05-2019
Automate splitting of files , scp files as each split completes and combine files on target server

i use the split command to split a one terabyte backup file into 10 chunks of 100 GB each. The files are split one after the other. While the files is being split, I will like to scp the files one after the other as soon as the previous one completes, from server A to Server B. Then on server B , once all the ten files have been received, i wll like to combine (cat them). How can i automate the whole process.
I want to save time and not wait for the completion of the splitting of all the ten 1000 GB files before I scp, I want to scp as each file completes.
Can anyone help with how to automate this process ? Thanks
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to combine to flat files which are there in remote server ?

:mad: I have to write a script in informatica, which is used to combine two remote flat files (3 Replies)
Discussion started by: srinivasaraju.m
3 Replies

2. Shell Programming and Scripting

Splitting input files into multiple files through AWK command

Hi, I needs to split *.txt files from single directory depends on the some mutltiple input values. i have wrote the code like below for file in *.txt do grep -i -h "value1|value2" $file > $file; done. My requirment is more input values needs to be given in grep; let us say 50... (3 Replies)
Discussion started by: arund_01
3 Replies

3. Shell Programming and Scripting

Shell script to transfer the files from source to target server.

I need to write a shell script to transfer the files every hour from source - target server. The cron job should be running every hour and shouldn't copy already copied files to the remote server ? I was able to write intial script but not able to get the logic for (in the next run it should... (12 Replies)
Discussion started by: radhirk
12 Replies

4. Shell Programming and Scripting

copy files from remote server (B) to target server (A)?

Hi All, what is the comand to log off the remote server? I have 2 servers A, B. I need to find all files older than 7 days on server B and copy over to server A. My logic is: login the remote server: ================= ssh hostB cd /data/test find . -mtime -7 -ls | awk '{print... (4 Replies)
Discussion started by: Beginer0705
4 Replies

5. Shell Programming and Scripting

scp files from other server with check

Hi all, I am new to this forum,and it has been really a great pleasure to be a part of this forum:) well,i m trying to write a shell script where i m transfering a set of files from other server to my own server. for this i m using scp... (1 Reply)
Discussion started by: prasson_ibm
1 Replies

6. Shell Programming and Scripting

script for Copying files from one server to another using scp

Hi Scripting experts, I am new to the unix scripting. Please help me out for solving the condition given below I am trying to develop a script for Copying files which are getting generated in server A to server B using scp. In serverA files are generating as for eg abc1.txt, abc2.txt,... (5 Replies)
Discussion started by: rohithji
5 Replies

7. Shell Programming and Scripting

combine multiple files by column into one files already sorted!

I have multiple files; each file contains a certain data in a column view simply i want to combine all those files into one file in columns example file1: a b c d file 2: 1 2 3 4 file 3: G (4 Replies)
Discussion started by: ahmedamro
4 Replies

8. Shell Programming and Scripting

scp files that are 3 days older from remote server-

hello, i am trying to get a list of files to be scped from the remote server by running the below in my local unix server ( note - there is a passwordless connectivity setup made between the local and remote server) and, we use KSH. --- ssh $scp_host "find /a/b/c/*/ -iname "$remote_file"" >... (4 Replies)
Discussion started by: billpeter3010
4 Replies

9. Shell Programming and Scripting

Need shell script to compare directories and delete files on target server

Hello, I need help in writing the shell script for below mentioned case. There are 2 servers(server A, server B). A cronjob syncs files between these 2 servers. Existing script is copying files from A to B. This is done using the command rsync. However, the files are not deleted... (2 Replies)
Discussion started by: SravaniVedam11
2 Replies

10. UNIX for Beginners Questions & Answers

scp of multiple files to remote server

Hello, I would like to scp multiple files to a remote server that requires a password for the process to be completed. I have 30 folders (x_1, x_2 ... x_30), each containing 25 files. What I want to do is scp 1 out of the 25 files (file called bvals) for all my folders to a remote server and... (3 Replies)
Discussion started by: nasia.m
3 Replies
split(1)							   User Commands							  split(1)

NAME
split - split a file into pieces SYNOPSIS
split [-linecount | -l linecount] [-a suffixlength] [file [name]] split [-b n | nk | nm] [-a suffixlength] [file [name]] DESCRIPTION
The split utility reads file and writes it in linecount-line pieces into a set of output-files. The name of the first output-file is name with aa appended, and so on lexicographically, up to zz (a maximum of 676 files). The maximum length of name is 2 characters less than the maximum filename length allowed by the filesystem. See statvfs(2). If no output name is given, x is used as the default (output-files will be called xaa, xab, and so forth). OPTIONS
The following options are supported: -linecount | -l linecount Number of lines in each piece. Defaults to 1000 lines. -a suffixlength Uses suffixlength letters to form the suffix portion of the filenames of the split file. If -a is not specified, the default suffix length is 2. If the sum of the name operand and the suffixlength option-argument would create a filename exceeding NAME_MAX bytes, an error will result; split will exit with a diagnostic message and no files will be created. -b n Splits a file into pieces n bytes in size. -b nk Splits a file into pieces n*1024 bytes in size. -b nm Splits a file into pieces n*1048576 bytes in size. OPERANDS
The following operands are supported: file The path name of the ordinary file to be split. If no input file is given or file is -, the standard input will be used. name The prefix to be used for each of the files resulting from the split operation. If no name argument is given, x will be used as the prefix of the output files. The combined length of the basename of prefix and suffixlength cannot exceed NAME_MAX bytes. See OPTIONS. USAGE
See largefile(5) for the description of the behavior of split when encountering files greater than or equal to 2 Gbyte ( 2^31 bytes). ENVIRONMENT VARIABLES
See environ(5) for descriptions of the following environment variables that affect the execution of split: LANG, LC_ALL, LC_CTYPE, LC_MES- SAGES, and NLSPATH. EXIT STATUS
The following exit values are returned: 0 Successful completion. >0 An error occurred. ATTRIBUTES
See attributes(5) for descriptions of the following attributes: +-----------------------------+-----------------------------+ | ATTRIBUTE TYPE | ATTRIBUTE VALUE | +-----------------------------+-----------------------------+ |Availability |SUNWesu | +-----------------------------+-----------------------------+ |CSI |Enabled | +-----------------------------+-----------------------------+ |Interface Stability |Committed | +-----------------------------+-----------------------------+ |Standard |See standards(5). | +-----------------------------+-----------------------------+ SEE ALSO
csplit(1), statvfs(2), attributes(5), environ(5), largefile(5), standards(5) SunOS 5.11 16 Apr 1999 split(1)
All times are GMT -4. The time now is 01:54 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy