Visit Our UNIX and Linux User Community


Script to archive logs and sftp to another archive server


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Script to archive logs and sftp to another archive server
# 1  
Old 01-30-2018
Script to archive logs and sftp to another archive server

Requirement:

Under fuse application we have placeholders called containers;



Every container has their logs under:

Code:
<container1>/data/log/fuse.log
<container1>/data/log/fuse.log.1
<container1>/data/log/fuse.log.XX
   
<container2>/data/log/fuse.log
<container2>/data/log/fuse.log.1
<container2>/data/log/fuse.log.XX




Our requirement is to have the logs moved from the server to another archiver server using SFTP after compressing it.


We have 9 servers and each of them will get their logs stored in the archiver server in different paths.

Lets say,

1st node has 3 containers
The archiver script will make a directory in the archiver server and store the compressed files as below.

Code:
/<app>/Prod/Cluster1/<Node1>/<Container1>/<compressed logs>
/<app>/Prod/Cluster1/<Node1>/<Container2>/<compressed logs>
/<app>/Prod/Cluster1/<Node1>/<Container3>/<compressed logs>

Similarly,

2nd node has 3 containers
Code:
/<app>/Prod/Cluster1/<Node2>/<Container1>/<compressed logs>
/<app>/Prod/Cluster1/<Node2>/<Container2>/<compressed logs>
/<app>/Prod/Cluster1/<Node2>/<Container3>/<compressed logs>

Any suggestion on this would be of great help Smilie

---------- Post updated at 11:08 AM ---------- Previous update was at 10:41 AM ----------

We had this implemented using perl and ftp but now we need to implement shell script with sftp
Smilie

Last edited by Scott; 01-30-2018 at 03:33 AM.. Reason: Code tags
# 2  
Old 01-30-2018
So why can't you just use the perl script and modify it to use SFTP instead of FTP?
# 3  
Old 01-30-2018
There is not SFTP perl module supported by redhat as of now. Got it checked with Redhat support and hence cannot implement it. Smilie
# 4  
Old 01-30-2018
Can you show us how you call ftp from your Perl?
# 5  
Old 01-30-2018
Do you have a file with the nine servers' access info listed, including the number of containers?
Do you want all log files in a directory compressed (by which mechanism / tool?) into one single archive?
Do you mean "delete original files" when you say "logs moved"?
Where is that script going to run, on a central server not identical with either of the nine nor the archive server?
# 6  
Old 01-30-2018
@RudiC
Do you have a file with the nine servers' access info listed, including the number of containers?
Ans: The existing perl script called another file which contained path for the logs node wise.
So node 1 would have 2 files, file 1: perl script, file2: a txt file containing the log location of that node.

Do you want all log files in a directory compressed (by which mechanism / tool?) into one single archive?
Ans: As of now no compression is in place. We would like the script to compress the logs and then sftp post compression.

Do you mean "delete original files" when you say "logs moved"?
Ans: Yes, so as to save server space. Something like logs purging

Where is that script going to run, on a central server not identical with either of the nine nor the archive server?
Ans. Not on a central server. The script will run individually on each node and sftp it to the archive server.
# 7  
Old 01-30-2018
@rbatte1

Attached the existing perl script

Previous Thread | Next Thread
Test Your Knowledge in Computers #603
Difficulty: Medium
MySQL 8.0 does not maintain information about password history so you cannot enabling restrictions on reuse of previous passwords.
True or False?

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

SFTP file and archive only if its succedded

Hi All, I have a requirement where I have to archive the file as soon as I sFTP the file to remote location below is the code I am going with ls $FILE_NAME | while read FNAME do sftp -v -oport=nn $FTP_USER@$FTP_HOST <<EOF 2>&1 | tee -a ${LOG_FILE_NAME} cd ${TGT_FOLDER_NAME} lcd... (4 Replies)
Discussion started by: Pratik4891
4 Replies

2. Solaris

archive logs mount point space check script

I have the below shell script which is checking /archlog mount point space on cappire(solaris 10) server. When the space usage is above 80% it should e-mail. When i tested this script it is working as expected. -------------------------------------------------------------------------... (0 Replies)
Discussion started by: dreams5617
0 Replies

3. UNIX for Dummies Questions & Answers

Archive files that was already sftp'd/process

Hi, I need to modify my archive script to archive only files which is already sftp to the windows. We encounter some issue which the file is already archiving though it is not yet sftp'd to the Windows. Any answers.?Thanks! (1 Reply)
Discussion started by: sonja
1 Replies

4. Shell Programming and Scripting

Connect to Server B from Server A and Archive the file

Hi I need to connect to a ServerB from Server A eg: Server A : 172.20.273.51, un: xxx, pwd: xxx Server B: 172.20.273.51, un:yyy, pwd: yyy Need to copy the files(with name starts with IME*) from /grid/pc/IME* to /grid/pc/archive What will be the command or script to... (3 Replies)
Discussion started by: vsmeruga
3 Replies

5. Shell Programming and Scripting

Archive log Pull script from one server to another server

Hi all, Iam looking out for a shell script which pulls archive from Server A to Server B. And at the same time, we dont want the archives which are already pulled in Server B to be removed from Server A. Please help me on this, I have been trying on this for a quiet few time. ... (3 Replies)
Discussion started by: vivi.raghav
3 Replies

6. Shell Programming and Scripting

Extracting from archive | compressing to new archive

Hi there, I have one huge archive (it's a system image). I need sometime to create smaller archives with only one or two file from my big archive. So I'm looking for a command that extracts files from an archive and pipe them to another one. I tried the following : tar -xzOf oldarchive.tgz... (5 Replies)
Discussion started by: chebarbudo
5 Replies

7. Shell Programming and Scripting

simple script to alert of archive logs filling

Hi all. I am not a DBA. But I do have responsibility for making sure the archive logs dont fill up and cause the database. This happend the other day while I was absent (sick) and I got a good ticking off for it. Needless to say I dont want this happen! Could anyone lend a hand to a... (8 Replies)
Discussion started by: Incremental
8 Replies

8. UNIX for Advanced & Expert Users

how to archive logs older than 5 days & then delete them?

My code is tar -cvf logs.tar `find /usr/openv/logs/512*.log -mtime +2` && find *.log* -mtime +2 -exec rm {} \; this gives me output as: tar: Missing filenames:confused: (1 Reply)
Discussion started by: timus1980
1 Replies

9. Solaris

Archive data accross solaris server

Can anyone tell me how to safely archive data across the network instead of tape in Solaris. Thanks Remi (1 Reply)
Discussion started by: Remi
1 Replies

Featured Tech Videos