Would appreciate a quick second set of eyes on a script (regarding doing things in the background)


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Would appreciate a quick second set of eyes on a script (regarding doing things in the background)
# 1  
Old 07-07-2010
Would appreciate a quick second set of eyes on a script (regarding doing things in the background)

What I'm trying to do is leave a tcpdump running all the time on a server, and at the end of every day kill it and start a new one. For some reason my boss doesn't want to set this up as a cron job, so I've come up with the following.:

Code:
#!/bin/bash

PCAPFILE=/tmp/mgmt.$(date "+%H.%M.%S.%m.%d").pcap

tcpdump -i eth0 -nn -vv -s 1500 net 192.168.5 -w $PCAPFILE   2>&1 & 
dumpPid=$!


TDATE=$(date --date='-1 days ago' "+%m.%d.%y")
DATE=$(date "+%m.%d.%y")


killDump ()
{
    kill $dumpPid
}


zipPcap ()
{
    gzip -c $1 > $1.gz && rm $1
}


    while true; do
        until [[ "$DATE" == "$TDATE" ]]; do
            sleep 1800
        done
        killDump
        zipPcap $PCAPFILE &
        PCAPFILE=/tmp/mgmt.$(date "+%H.%M.%S.%m.%d").pcap
        tcpdump -i eth0 -nn -vv -s 1500 net 192.168.5 -w $PCAPFILE   2>&1 & 
        dumpPid=$!
        TDATE=$(date --date='-1 days ago' "+%m.%d.%y")
        DATE=$(date "+%m.%d.%y")
    done

I've tested this on a small scale on my personal laptop, but I've made a minor change as highlighted above that I want to make sure shouldn't cause any problems.

Since by the end of the day the pcap is probably going to be rather large, I don't want the script to have to wait for the old one to finish zipping before it starts the next capture, so is running the zipPcap function in the background like that an acceptable way of kicking off the compression, and having the rest of the script move forwards?

I'm pretty confident this should be fine, but as I'm working without a lab to test on right now, I wanted to get a second set of eyes.
# 2  
Old 07-07-2010
Quote:
Originally Posted by DeCoTwc
Since by the end of the day the pcap is probably going to be rather large, I don't want the script to have to wait for the old one to finish zipping before it starts the next capture
How many CPU cores does the system have? On a one-core system there's no advantage to running multiple gzips in parallel, since each will be running half as fast. On a four-core you could run four. etc. There's also the question of disk speed -- your disk can probalby keep up with 1 running gzip easily, but how about 4, or 8? And having four different large files being written to disk simultaneously is a recipe for bad fragmentation. There's also the question of memory use, potentially unlimited if you have a huge number of files.

The point? Don't go too nuts. A few may help, dozens probably won't. You may also want to create the file in temp space then move it once complete, to avoid too much fragmentation on the main filesystem.
Quote:
so is running the zipPcap function in the background like that an acceptable way of kicking off the compression, and having the rest of the script move forwards?
You'll want a way to limit the number of processes created, create n processes for n cores then start waiting for individual processes before creating another.

I'd also note you might want a way to check whether a background compression succeeded or failed. Again, you can tell this by wait-ing for a specific background PID.
# 3  
Old 07-07-2010
I think this bit will loop forever. Neither variable changes within the loop.

Quote:
until [[ "$DATE" == "$TDATE" ]]; do
sleep 1800
done
# 4  
Old 07-07-2010
Also. Why not have pcap pipe its output through gzip in the first place, instead of afterwards in one big batch?
# 5  
Old 07-07-2010
Quote:
Originally Posted by methyl
I think this bit will loop forever. Neither variable changes within the loop.

I'm not sure what you mean, there are two loops, the while true loop and the until loop.

All the until loop does is every 30 minutes check if the dates match or not. Once they do, it does all the work in the while loop which includes updating the dates:

Code:
while true; do  
        until [[ "$DATE" == "$TDATE" ]]; do
            sleep 1800
        done
        killDump
        zipPcap $PCAPFILE &
        PCAPFILE=/tmp/mgmt.$(date "+%H.%M.%S.%m.%d").pcap
        tcpdump -i eth0 -nn -vv -s 1500 net 192.168.5 -w $PCAPFILE   2>&1 & 
        dumpPid=$!
        TDATE=$(date --date='-1 days ago' "+%m.%d.%y")
        DATE=$(date "+%m.%d.%y")
    done

Every time the red part completes, it does the blue part which includes updating the variables that red relies on.

---------- Post updated at 12:56 PM ---------- Previous update was at 12:44 PM ----------

Quote:
Originally Posted by Corona688
How many CPU cores does the system have? On a one-core system there's no advantage to running multiple gzips in parallel, since each will be running half as fast. On a four-core you could run four. etc. There's also the question of disk speed -- your disk can probalby keep up with 1 running gzip easily, but how about 4, or 8? And having four different large files being written to disk simultaneously is a recipe for bad fragmentation. There's also the question of memory use, potentially unlimited if you have a huge number of files.

The point? Don't go too nuts. A few may help, dozens probably won't. You may also want to create the file in temp space then move it once complete, to avoid too much fragmentation on the main filesystem. You'll want a way to limit the number of processes created, create n processes for n cores then start waiting for individual processes before creating another.

I'd also note you might want a way to check whether a background compression succeeded or failed. Again, you can tell this by wait-ing for a specific background PID.
Well my main concern isn't having multiple zips going on at the same time. I mean it should only be zipping the file once a day, and I highly doubt that by the time the next zip comes around the last one will still be running.

My concern is that I don't want the next instance of tcpdump to have to wait for the zip to complete. I wanted the zip to run in the background while the next tcpdump starts.

I decided to do some more testing on a VM on my laptop, and I noticed that unlike what I wanted, the tcpdump was waiting for the zip to complete before starting again, so I moved the ampersand inside the zipPcap function, and now it seems to be working. I'm watching the files get created in real time, and when one stops writing 2 more files appear, the .gz, and the new .pcap. Once the .gz is done, the original file gets deleted.

My testing version:

Code:
#!/bin/bash

PCAPFILE=/tmp/mgmt.$(date "+%H.%M.%S.%m.%d").pcap

tcpdump -i eth0 -nn -vv -s 1500 -w $PCAPFILE   2>&1 & 
dumpPid=$!
i=1
killnum=3

TDATE=$(date --date='-1 days ago' "+%m.%d.%y")
DATE=$(date "+%m.%d.%y")


killDump ()
{
    kill $dumpPid
}


zipPcap ()
{
    gzip -c $1 > $1.gz && rm $1 &
}


    while true; do
        until [[ "$i" == "$killnum" ]]; do
            sleep 30
	    let i=i+1
	    echo $i
	    echo $dumpPid
        done
        killDump
        zipPcap $PCAPFILE 
        PCAPFILE=/tmp/mgmt.$(date "+%H.%M.%S.%m.%d").pcap
        tcpdump -i eth0 -nn -vv -s 1500  -w $PCAPFILE   2>&1 & 
        dumpPid=$!
        TDATE=$(date --date='-1 days ago' "+%m.%d.%y")
        DATE=$(date "+%m.%d.%y")
	i=0
    done

Oh, and to answer your question, the box I'm working on has 16 cores...


So far as zipping the pcap in real time...honestly, I didn't know that was an option.
# 6  
Old 07-07-2010
The red part never completes because nothing changes $DATE or $TDATE within the loop.

This might work (untested in bash):

Code:
       until [[ "$DATE" == "$TDATE" ]]; do
            sleep 1800
            DATE=$(date "+%m.%d.%y")
        done

# 7  
Old 07-07-2010
---------- Post updated at 05:57 PM ---------- Previous update was at 01:07 PM ----------

[/COLOR]Just an update, this is the version I wound up going with. It seems to be working fine, but I suppose I won't really know for sure until after 1230

Code:
#!/bin/bash
######################################################
# Program:      networkCap.sh
# Date Created: 7 July 2010
# Date Updated: NA
# Developer:    G B (Support Manager) && D D (Support Engineer)
# Description:  runs tcpdump on management netwok and automatically starts a new capture each day and zips the old one
######################################################

PCAPFILE=/tmp/mgmt.$(date "+%H.%M.%S.%m.%d").pcap

tcpdump -i eth0 -nnvv -s 1500 net 10.248.89 -w $PCAPFILE   2>&1 &
dumpPid=$!


TDATE=$(date --date='-1 days ago' "+%m.%d.%y")
DATE=$(date "+%m.%d.%y")


killDump ()
{
    kill $dumpPid
}


zipPcap ()
{
    gzip -c $1 > $1.gz && rm $1 &
}


    while true; do
        until [[ "$DATE" == "$TDATE" ]]; do
            sleep 1800
            DATE=$(date "+%m.%d.%y")
        done
        killDump
        zipPcap $PCAPFILE
        PCAPFILE=/tmp/mgmt.$(date "+%H.%M.%S.%m.%d").pcap
        tcpdump -i eth0 -nnvv -s 1500 net 10.248.89 -w $PCAPFILE   2>&1 &
        dumpPid=$!
        TDATE=$(date --date='-1 days ago' "+%m.%d.%y")
        DATE=$(date "+%m.%d.%y")
    done


Last edited by DeCoTwc; 07-07-2010 at 11:36 PM.. Reason: Finally realized just how wrong I was about methyl's advice and fixed things
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to set background image

Dear Friends I want to, set the background of mail as image/colorful using scripting. I try by send mail command but it is not working.Please give me suggestion on this. (1 Reply)
Discussion started by: AhmedLakadkutta
1 Replies

2. Shell Programming and Scripting

Shell script that will do two things at once

I am trying to script up a build of my system from source, and the first couple of steps requires me to do a pull from a CVS mirror three times, and then the script begins. What I'd like is to be able to do one CVS pull, then start the build of userland, and while userland is building, pull down... (1 Reply)
Discussion started by: brakeb
1 Replies

3. UNIX for Dummies Questions & Answers

Quick question about set number

In my .exrc file I have line numbers turned on but it adds an indent. I don't like this, is there a way to have the line numbers at the left edge of my terminal instead of indented? Here's my .exrc 1 set ignorecase noslowopen report=0 autoindent showmatch showmode nu 2 set... (4 Replies)
Discussion started by: ebadamageplan
4 Replies

4. Shell Programming and Scripting

how to set background color in Unix terminal

Hi All, how do I set in .profile file Unix terminal background color = BLUE ? Please advice me. :confused: (2 Replies)
Discussion started by: raghur77
2 Replies

5. Shell Programming and Scripting

Broke Perl Script Second pair of eyes NET::FTPSSL

Hi all, Let me first start out by saying I'm a perl newbie and hope somebody can help, for the life of me I can't figure out why my script will not find and download a remote file via FTPSSL. What it's supposed to do is find the latest file named... (4 Replies)
Discussion started by: Styles
4 Replies

6. Programming

Another set of eyes

Original code used fwrite instead of putc. What is expected is that the destination file will be written to. Instead I end up with a zero length file. I'm sure there is something simple I'm missing. tia. #include <stdio.h> #include <stdlib.h> #include <string.h> int main(int argc, char... (6 Replies)
Discussion started by: ramen_noodle
6 Replies

7. Shell Programming and Scripting

Need some help with this script -- extra eyes

I have two issues with this script. 1. I cannot seem to get my counters to count correctly. 2. My function to eject to CAP1 or CAP2 is hung in a loop and doens't exit back to the previous function. I would like to be able to select which cap to eject to . Each cap holds only 40 tapes, so when one... (15 Replies)
Discussion started by: gzs553
15 Replies

8. UNIX for Advanced & Expert Users

How to set background colours for a cygwin console

Hi, I need to set the background colors for cygwin console, when I do ssh to production boxes. This should be done through commands.. Please suggest me asap. Thanks in advance. (3 Replies)
Discussion started by: praveen_b744
3 Replies

9. Shell Programming and Scripting

set schedule to run a script at background while logout

Hi, How can I run a script at 9:00am and 6:00pm everyday? Can I run it at background while I logout my account? Please help!! Many Thanks!! (1 Reply)
Discussion started by: happyv
1 Replies

10. UNIX for Dummies Questions & Answers

set background/foreground color in .profile

I am using a telnet session (VT100) and need to modify my .profile so that it will set the color of the telnet session. I am not using Xterm (ie: can't use .Xdefaults). I am able to change the colors via menu's but need to preset in .profile. Is this possible??? Can't find anything at all on how... (3 Replies)
Discussion started by: dvella
3 Replies
Login or Register to Ask a Question