Need advice on approach for script Post: 302603561

Sponsored Content

Top Forums Shell Programming and Scripting Need advice on approach for script Post 302603561 by msarro on Thursday 1st of March 2012 10:37:31 AM

03-01-2012

Registered User

Need advice on approach for script

Greetings all. I have a repository server which receives, without exhaggeration, several million files a week. The majority of these files are in .csv format, which means they're highly compressable. They are spread throughout numerous directories where there are configured monitoring utilities which parse them as they arrive. They parsers will only parse files as old as 7 days, so at day 8 I would like the files to be tarred/gzipped.

I would like to do this, and am trying to figure out the best approach. I could use find, which would go through and recursively tar/gzip everything it finds. However this doesn't move the files to a new location. Granted, I could use a second find command and move everything to a mirror of the repository directory structure.

I was also considering just using tar, creating an archive using the directory structure of the source data, setting it to remove the files, but I can't find a way in tar to remove only files older than N days old.

I was hoping I could get some suggestions on the best approach to take.

To illustrate:
Right now I have these directories:
/splunk/MSP/
/splunk/CEMP/
/splunk/CEMP/PBTS/
/splunk/CEMP/BTS/
/splunk/CEMP/cedar/
/splunk/CEMP/sbc/

and in each one there are several thousand files. Imagine there are files in each dir called a and b, where a is 8 days old, b is 1 day old.

In a perfect world I would have the script generate a single tar/gz file in a specified directory, which had the complete directory structure contained inside with only file a from each dir. After running the script, inside each directory only file b should remain in the actual repo.

Your thoughts and suggestions are appreciated.

msarro

View Public Profile for msarro

Find all posts by msarro

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

first script. need help and advice.

Hello everyone, This is my first post here and this is the first time I am using UNIX OS (Slackware). I find it really useful and powerful and would like to master it but as you may guess I am expreicing quite a few problems. I've been reading a few documentations about it and bash this week...

2. Shell Programming and Scripting

Advice on Script

I would like some advice on how to logically put together a script to handle a daily task of data gathering for the following problem. I have two files, file1 has 125,000 records that I cut and remove unwanted fields through scripts and cron. In file2, I have 25000 records that has the same...

3. Shell Programming and Scripting

Script Advice please?

Ok. I want to parse a log file and search only for denied traffic for the previous hour. The log looks like this: Jun 18 17:47:56 routername 36806: Jun 18 17:53:01.088: %SEC-6-IPACCESSLOG: list ingress-filter denied tcp 1.2.3.4(1234) -> 6.7.8.9(53), 4 packets I only really care about the...

4. Shell Programming and Scripting

Script Help/Advice

Alright, I feel like I have a pretty good basic knowledge of shell scripting, but this one is throwing me for a loop. I know I've seen something similar done with awk, but I couldn't find it with the search function. I've grepped through my log file and get results like this: --...

5. Shell Programming and Scripting

advice on shell script

Hello, I have this script running on cron every 20 minutes. By 12pm daily, our system is expecting all input files to be uploaded by the script. After this cutoff time, the script would still be running though, but i need some kind of alerts/logs to know which input files weren't received for...

6. Shell Programming and Scripting

Approach to writting a script

Hello all, I've just joined. I did a google search and your site came up, I had a look and thought I'd like to become a member. I'm from Ireland. I've written a few scripts before, but this new task has me foxed. I would like to figure out the best approach to achieving the following ...

7. Shell Programming and Scripting

Advice on script

Hi folks, I use following script:- #!/bin/sh # cd Linbread TODAY=`date +"%m%d"` DATA=`grep $TODAY linbread.dat` HOUR=`date +"%H"` if then TOD="Morning" elif then TOD="Afternoon" else TOD="Evening" fi echo $DATA | gawk -F"|" '{printf("%s\n\n%s",$2,$3)}' > $$tmp fold -s -w60...

8. SuSE

Write shell script using menu-driven approach to show various system

QUESTION: Write shell script using menu-driven approach to show various system configuration like 1) Currently logged user and his logname 2) Your current shell 3) Your home directory 4) Your current path setting 5) Your current working directory 6) Show Currently logged number of...

9. Shell Programming and Scripting

Script advice

All I have 2 parent directories - input and output. Each parent has multiple sub-directories...each sub-directory has multiple files. Each parent directory structure is a mirror image of itself I need to poll the imput directory and if a new file is found, encrypt the file, move the file to...

10. Shell Programming and Scripting

Help with approach and developing script

Hi- I need to develop a script for following scenario in AIX and K shell environment.I am from windows server background for most my career ,so please bear with me and advise suitable approach and technical assistance.Having said that I am aware of unix shell commands but never pput together at...

LEARN ABOUT DEBIAN

virt-tar-out

virt-tar-out(1) 					      Virtualization Support						   virt-tar-out(1)

NAME

       virt-tar-out - Pack a virtual machine disk image directory into a tarball.

SYNOPSIS

	virt-tar-out -a disk.img /dir files.tar

	virt-tar-out -d domain /dir files.tar

	virt-tar-out -d domain /dir - | gzip --best > files.tar.gz

DESCRIPTION

       "virt-tar-out" packs a virtual machine disk image directory into a tarball.

       The first parameter is the absolute path of the virtual machine directory.  The second parameter is the tar file to write.  Use "-" to
       write to standard output.

EXAMPLES

       Download the home directories from a guest:

	virt-tar-out -d MyGuest /home - | gzip --best > homes.tar.gz

JUST A SHELL SCRIPT WRAPPER AROUND GUESTFISH

       This command is just a simple shell script wrapper around the guestfish(1) "tar-out" command.  For anything more complex than a trivial
       copy, you are probably better off using guestfish directly.

OPTIONS

       Since the shell script just passes options straight to guestfish, read guestfish(1) to see the full list of options.

SEE ALSO

       guestfish(1), virt-cat(1), virt-copy-in(1), virt-copy-out(1), virt-edit(1), virt-tar-in(1), <http://libguestfs.org/>.

AUTHORS

       Richard W.M. Jones ("rjones at redhat dot com")

COPYRIGHT

       Copyright (C) 2011 Red Hat Inc.	<http://libguestfs.org/>

       This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by
       the Free Software Foundation; either version 2 of the License, or (at your option) any later version.

       This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of
       MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License for more details.

       You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation,
       Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA.

libguestfs-1.18.1						    2013-12-07							   virt-tar-out(1)