Hey everyone. First some background... our network routinely polls SNMP data from across a massive ecosystem three times a day. Depending on device revision, some data is useful, some isn't. The tar.gz file is named MTA_DATE_(1, 2, or 3).tar.gz based on which poll of the day it is. The files are then put in a folder on a file server where they are accessible via http at the end of the collection window.
My approach is to create three scripts, each to be run by cron to download the latest file at the end of its upload window. Each script is run 8 hours apart. The script downloads the file (1, 2, or 3), extracts the contents to a temporary folder, removes useless bloat data, and then retars/zips them, and finally copies them to their destination directory.
Here are the first two scripts:
1
Code:
#!/bin/bash
cd /var/splunk/MTA/
wget http://myserver.net/polldata_archives/MTA/MTA_`date +%Y``date +%m``date +%d`_1.tar.gz && tar xfz MTA_`date +%Y``date +%m``date +%d`_1.tar.gz && yes|rm MTA_`date +%Y``date +%m``date +%d`_1.tar.gz && find . -type f -size 0 -print|xargs rm && sed -i 's/TIMEOUT//g;s/null//g;s/Not Supported//g;s/Index Not Found//g' *.net && yes|mv *.net cleaned/
cd cleaned
tar -zcf MTA_`date +%Y``date +%m``date +%d`_1.tar.gz *.* && mv MTA_`date +%Y``date +%m``date +%d`_1.tar.gz /var/splunk/MTA/LOGARCH/ && yes|rm *.*
echo MTA_`date +%Y``date +%m``date +%d`_1.tar.gz >> /var/splunk/MTA/log
exit
2
Code:
#!/bin/bash
cd /var/splunk/MTA/
wget http://myserver.net/polldata_archives/MTA/MTA_`date +%Y``date +%m``date +%d`_2.tar.gz && tar xfz MTA_`date +%Y``date +%m``date +%d`_2.tar.gz && yes|rm MTA_`date +%Y``date +%m``date +%d`_2.tar.gz && find . -type f -size 0 -print|xargs rm && sed -i 's/TIMEOUT//g;s/null//g;s/Not Supported//g;s/Index Not Found//g' *.net && yes|mv *.net cleaned/
cd cleaned
tar -zcf MTA_`date +%Y``date +%m``date +%d`_2.tar.gz *.* && mv MTA_`date +%Y``date +%m``date +%d`_2.tar.gz /var/splunk/MTA/LOGARCH/ && yes|rm *.*
echo MTA_`date +%Y``date +%m``date +%d`_2.tar.gz >> /var/splunk/MTA/log
exit
The third script is basically the same as the first two however it doesn't work. The issue is that the last window doesn't finish until 12am, the next day. So, using the date command no longer works. How can I take the current date, subtract 1, and use that to create the new filename? If I need I can do it in python but I was hoping to stick entirely with BASH to reduce the need of additional software components.
Start before 8:00, 16:00, midnight, store the date-time in a variable, from date if not provided on the command line (for testing), and use bash date-time functions on the variable, not date. Then sleep a bit if that is too soon.
Get your shift at run start by (hour + 1) / 3, and use one script.
Have your producing systems make more friendly data for you?
Find tools and structure that allow more pipelined, parallel processing to get done sooner without a lot of intermediate file creation/destruction. Maybe tar -zcf file is slower than tar -f - | gzip > file ?
Using mv between fs is expensive (cp+rm), but in one fs is very fast (dir updates only).
As long as both scripts start before midnight, you only need to evaluate the date once at the start of the script.
See this illustration of storing the date in an environment variable then using it afterwards in a command.
Code:
YYYYMMDD="`date +%Y%m%d`"
echo "${YYYYMMDD}"
In fact you can then compose the whole filename in an environment variable and considerably shorten your subsequent command lines as well as making them easier to read.
For example:
Dear all,
I am working on script which call other shell scripts in a loop but problem is from second script am not able to come out.
Here is the snippet:-
#!/bin/bash
HSFILE=/root/Test/Components.txt
LOGFile=/opt/domain/AdminDomain/application/logs... (3 Replies)
Hi all, I'm wondering if you could give me some advice. I am new to scripting and am getting rather frustrated that i can get my script to call another script if certain criteria is met, via command line, but I cannot get the same script to work thru the cron jobs.
My first script monitors... (8 Replies)
Dear Members,
I have a table REQUESTS in Oracle which has an attribute REQUEST_ACTION. The entries in REQUEST_ACTION are like, ME, MD, ND, NE etc.
I would like to create a script which will will call other scripts based on the request action.
Can we directly read from the REQUEST_ACTION... (2 Replies)
Hi,
I have to write a program to compute the checksums of files
./script.sh
I wrote the program using bash and it took me forever since I am a beginner but it works very well.
I'm getting so close to the deadline and I realised today that actually I have to use normal Bourne shell... (3 Replies)
Hi all,
I have set up a cron job which calls another shell script shell script which in turn calls a Java process. The cron tab looks so.
0,30 7-18 * * 1-5 /u01/home/weblogic/brp/bin/checkstatus.sh >> /u01/home/weblogic/logs/checkstatus.log
The checkstatus.sh scripts looks like this.
... (4 Replies)
Hi,
I've written a script that runs on a Database server. It has to shutdown the Application server, do an Oracle Dump and then restart the Application server. Its been a long time since I wrote any shells scripts. Can you tell me if the scripts that I execute within my script will be executed... (3 Replies)
Hi,
First, let me explain the issue I am trying to solve. We have a lot of expect scripts with the duplicated send/expect commands. So, I'd like to be able to extract the duplicated code into the common scripts that can be used by other scripts.
Below is my test where I am trying to call... (0 Replies)
script that ask for "enter a file name" and removes that file
and asks for confirmation before deletion
if executed the output might look as
enter the filename you intent to deleted
remover file? Y
file deleterd
I knwo the comand I would use
find . -name *.* -ok rm {}\; I guess... can... (1 Reply)
12-07-2010
