Sponsored Content
Top Forums Shell Programming and Scripting Script to archive logs and sftp to another archive server Post 303012094 by Arjun Goswami on Tuesday 30th of January 2018 12:38:16 AM
Old 01-30-2018
Script to archive logs and sftp to another archive server

Requirement:

Under fuse application we have placeholders called containers;



Every container has their logs under:

Code:
<container1>/data/log/fuse.log
<container1>/data/log/fuse.log.1
<container1>/data/log/fuse.log.XX
   
<container2>/data/log/fuse.log
<container2>/data/log/fuse.log.1
<container2>/data/log/fuse.log.XX




Our requirement is to have the logs moved from the server to another archiver server using SFTP after compressing it.


We have 9 servers and each of them will get their logs stored in the archiver server in different paths.

Lets say,

1st node has 3 containers
The archiver script will make a directory in the archiver server and store the compressed files as below.

Code:
/<app>/Prod/Cluster1/<Node1>/<Container1>/<compressed logs>
/<app>/Prod/Cluster1/<Node1>/<Container2>/<compressed logs>
/<app>/Prod/Cluster1/<Node1>/<Container3>/<compressed logs>

Similarly,

2nd node has 3 containers
Code:
/<app>/Prod/Cluster1/<Node2>/<Container1>/<compressed logs>
/<app>/Prod/Cluster1/<Node2>/<Container2>/<compressed logs>
/<app>/Prod/Cluster1/<Node2>/<Container3>/<compressed logs>

Any suggestion on this would be of great help Smilie

---------- Post updated at 11:08 AM ---------- Previous update was at 10:41 AM ----------

We had this implemented using perl and ftp but now we need to implement shell script with sftp
Smilie

Last edited by Scott; 01-30-2018 at 03:33 AM.. Reason: Code tags
 

9 More Discussions You Might Find Interesting

1. Solaris

Archive data accross solaris server

Can anyone tell me how to safely archive data across the network instead of tape in Solaris. Thanks Remi (1 Reply)
Discussion started by: Remi
1 Replies

2. UNIX for Advanced & Expert Users

how to archive logs older than 5 days & then delete them?

My code is tar -cvf logs.tar `find /usr/openv/logs/512*.log -mtime +2` && find *.log* -mtime +2 -exec rm {} \; this gives me output as: tar: Missing filenames:confused: (1 Reply)
Discussion started by: timus1980
1 Replies

3. Shell Programming and Scripting

simple script to alert of archive logs filling

Hi all. I am not a DBA. But I do have responsibility for making sure the archive logs dont fill up and cause the database. This happend the other day while I was absent (sick) and I got a good ticking off for it. Needless to say I dont want this happen! Could anyone lend a hand to a... (8 Replies)
Discussion started by: Incremental
8 Replies

4. Shell Programming and Scripting

Extracting from archive | compressing to new archive

Hi there, I have one huge archive (it's a system image). I need sometime to create smaller archives with only one or two file from my big archive. So I'm looking for a command that extracts files from an archive and pipe them to another one. I tried the following : tar -xzOf oldarchive.tgz... (5 Replies)
Discussion started by: chebarbudo
5 Replies

5. Shell Programming and Scripting

Archive log Pull script from one server to another server

Hi all, Iam looking out for a shell script which pulls archive from Server A to Server B. And at the same time, we dont want the archives which are already pulled in Server B to be removed from Server A. Please help me on this, I have been trying on this for a quiet few time. ... (3 Replies)
Discussion started by: vivi.raghav
3 Replies

6. Shell Programming and Scripting

Connect to Server B from Server A and Archive the file

Hi I need to connect to a ServerB from Server A eg: Server A : 172.20.273.51, un: xxx, pwd: xxx Server B: 172.20.273.51, un:yyy, pwd: yyy Need to copy the files(with name starts with IME*) from /grid/pc/IME* to /grid/pc/archive What will be the command or script to... (3 Replies)
Discussion started by: vsmeruga
3 Replies

7. UNIX for Dummies Questions & Answers

Archive files that was already sftp'd/process

Hi, I need to modify my archive script to archive only files which is already sftp to the windows. We encounter some issue which the file is already archiving though it is not yet sftp'd to the Windows. Any answers.?Thanks! (1 Reply)
Discussion started by: sonja
1 Replies

8. Solaris

archive logs mount point space check script

I have the below shell script which is checking /archlog mount point space on cappire(solaris 10) server. When the space usage is above 80% it should e-mail. When i tested this script it is working as expected. -------------------------------------------------------------------------... (0 Replies)
Discussion started by: dreams5617
0 Replies

9. Shell Programming and Scripting

SFTP file and archive only if its succedded

Hi All, I have a requirement where I have to archive the file as soon as I sFTP the file to remote location below is the code I am going with ls $FILE_NAME | while read FNAME do sftp -v -oport=nn $FTP_USER@$FTP_HOST <<EOF 2>&1 | tee -a ${LOG_FILE_NAME} cd ${TGT_FOLDER_NAME} lcd... (4 Replies)
Discussion started by: Pratik4891
4 Replies
GlusterFS(8)							   Gluster Inc. 						      GlusterFS(8)

NAME
GlusterFS - Clustered Filesystem. SYNOPSIS
glusterfsd [options] [mountpoint] DESCRIPTION
GlusterFS is a clustered file-system capable of scaling to several peta-bytes. It aggregates various storage bricks over Infiniband RDMA or TCP/IP interconnect into one large parallel network file system. Storage bricks can be made of any commodity hardware such as x86-64 server with SATA-II RAID and Infiniband HBA. GlusterFS is fully POSIX compliant FileSystem. On client side, it has dependency on FUSE package, on server side, it works seemlessly on different OSes. (Currently supported on GNU/Linux, Solaris). OPTIONS
Mandatory or optional arguments to long options are also mandatory or optional for any corresponding short options. Basic options -f, --volfile=VOLUME-FILE File to use as VOLUME-FILE [default:/etc/glusterfs/glusterfs.vol] -l, --log-file=LOGFILE File to use for logging [default:/var/log/glusterfs/glusterfs.log] -L, --log-level=LOGLEVEL Logging severity. Valid options are TRACE, DEBUG, INFO, WARNING, ERROR and CRITICAL [default: WARNING] -s, --volfile-server=SERVER Server to get the volume from. This option overrides --volfile option Advanced options --debug Run in debug mode. This option sets --no-daemon, --log-level to DEBUG and --log-file to console -N, --no-daemon Run in foreground --read-only Makes the filesystem read-only -p, --pid-file=PIDFILE File to use as pid file --volfile-id=KEY KEY of the volume file to be fetched from server --volfile-server-port=PORT Port number of volfile server --volfile-server-transport=TRANSPORT Transport type to get volume file from server [default: socket] --volume-name=VOLUME-NAME Volume name to be used for MOUNT-POINT [default: top most volume in VOLUME-FILE] --xlator-option=VOLUME-NAME.OPTION=VALUE Add/override a translator option for a volume with the specified value Fuse options --attribute-timeout=SECONDS Set attribute timeout to SECONDS for inodes in fuse kernel module [default: 1] --entry-timeout=SECONDS Set entry timeout to SECONDS in fuse kernel module [default: 1] --direct-io-mode=BOOL Enable/Disable direct-io mode in fuse module [default: enable] Miscellaneous Options -?, --help Give this help list --usage Give a short usage message -V, --version Print program version FILES
/etc/glusterfs/*.vol SEE ALSO
fusermount(1), mount.glusterfs(8), gluster(8) COPYRIGHT
Copyright(c) 2006-2011 Gluster, Inc. <http://www.gluster.com> 19 March 2010 Cluster Filesystem GlusterFS(8)
All times are GMT -4. The time now is 08:29 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy