Home Man
Today's Posts

Tell your work related tech stories and share experiences here. Share your successes and failures and other "war stories" in this forum.

Right On Time, Somewhere

war stories

Login to Reply

Thread Tools Search this Thread
# 1  
Old 01-12-2012
Right On Time, Somewhere

Things like this are teaching me a greater appreciation for network time...

The huge gap between the speed of a satellite connection and its tiny bandwidth allowance means either being completely draconian or having to chide people to not abuse it all the time. We're somewhere inbetween, if a customer downloads too much they'll be slowed down temporarily, but not shut off; if the situation continues, we may phonecall and investigate.

We had a situation where a small community had massively overused their satellite connection pretty much collectively, going over not only our limits but our provider's limits, causing the entire satellite connection to be throttled. We needed to shut the connection down for a few hours before the satellite modem would let go.

We planned it, set a time, and warned our customers. How it worked was very simple -- two entries in root's crontab. At noon that day, the first one would run 'ifconfig eth1 down', taking the community offline but leaving me in communication with the server. At 5pm that day, it would run '/sbin/reboot'.

The server clock had drifted far more than I'd anticipated in the months since its last boot and clock-set, and the shutdown happened one hour early.
# 2  
Old 01-13-2012
Couldn't even blame it on Daylight Savings Time. Did you install ntp?
# 3  
Old 01-13-2012
Testing it in dev first. If I mess up a production system, it's hours of driving and sometimes a bit of sledding to fix them in person, especially at this time of year.
# 4  
Old 01-16-2012
Thanks for sharing your story. It's very true that most of the times we do not bother to check the time of the clock before scheduling stuffs.

We maintain IT infrastructure for a big pharma company. For any SLA (service level agreement) breach, my employer has to pay a real big amount of money to the client. Now that's been told, once my colleague had to schedule a maintenance on an AIX server. We have a procedure to do that. There's a lot of approvals from service delivery managers of both the client and our company required. After getting those, this guy went on scheduling the reboot of the machine in maintenance mode in cron a day before. The next day, I got a call from IT Incident management people saying a server is down before it's scheduled maintenance window. It happened around 20 minutes before the scheduled time. We had to raise a severity for this. Upon checking the root cause of this later, we found somehow the server was failing to sync with the NTP server and the clock was going 20 minutes faster than the actual time.

And yes, because of all these, we breached the SLA!
The Following User Says Thank You to admin_xor For This Useful Post:
nmatt00 (01-30-2012)
Login to Reply

« Previous Thread | Next Thread »
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Calculate Time diff in milli milliseconds(Time format : HH:MM:SS,NNN) Raza Ali Shell Programming and Scripting 6 03-06-2018 08:55 AM
Find gaps in time data and replace missing time value and column 2 value by interpolation in awk malandisa Programming 4 01-16-2015 10:11 AM
Convert UTC time into current UNIX sever time zone mohanalakshmi Shell Programming and Scripting 5 12-11-2013 07:19 PM
Adding time to date time in UNIX shell scipting codehelp04 Shell Programming and Scripting 3 12-03-2012 04:31 AM
modifying date and time and time zone on solaris 5.10 with (redundant server) veritas barry1946 Solaris 1 09-20-2012 12:28 AM
How to get time duration between two human readable time stamp in Unix? ford99 Shell Programming and Scripting 3 07-19-2011 09:52 AM
Shell script to convert epoch time to real time aismann Shell Programming and Scripting 4 04-25-2011 10:09 PM
Convert Epoch Time to Standard Date and Time & Vice Versa DrivesMeCrazy Shell Programming and Scripting 5 02-07-2009 12:40 AM
How To Provide Time Sync Using Nts-150 Time Server On Unix Network? pesty UNIX for Advanced & Expert Users 2 03-22-2007 02:20 AM

All times are GMT -4. The time now is 02:55 AM.

Unix & Linux Forums Content Copyrightę1993-2018. All Rights Reserved.
Show Password