Unix/Linux Go Back    


War Stories Tell your work related tech stories and share experiences here. Share your successes and failures and other "war stories" in this forum.

Right On Time, Somewhere

War Stories


Closed    
 
Thread Tools Search this Thread Display Modes
    #1  
Old Unix and Linux 01-12-2012   -   Original Discussion by Corona688
Corona688 Corona688 is offline Forum Staff  
Mead Rotor
 
Join Date: Aug 2005
Last Activity: 21 November 2017, 3:22 PM EST
Location: Saskatchewan
Posts: 22,518
Thanks: 1,154
Thanked 4,273 Times in 3,946 Posts
Right On Time, Somewhere

Things like this are teaching me a greater appreciation for network time...

The huge gap between the speed of a satellite connection and its tiny bandwidth allowance means either being completely draconian or having to chide people to not abuse it all the time. We're somewhere inbetween, if a customer downloads too much they'll be slowed down temporarily, but not shut off; if the situation continues, we may phonecall and investigate.

We had a situation where a small community had massively overused their satellite connection pretty much collectively, going over not only our limits but our provider's limits, causing the entire satellite connection to be throttled. We needed to shut the connection down for a few hours before the satellite modem would let go.

We planned it, set a time, and warned our customers. How it worked was very simple -- two entries in root's crontab. At noon that day, the first one would run 'ifconfig eth1 down', taking the community offline but leaving me in communication with the server. At 5pm that day, it would run '/sbin/reboot'.

The server clock had drifted far more than I'd anticipated in the months since its last boot and clock-set, and the shutdown happened one hour early.
Sponsored Links
    #2  
Old Unix and Linux 01-13-2012   -   Original Discussion by Corona688
jgt's Unix or Linux Image
jgt jgt is offline Forum Advisor  
Registered User
 
Join Date: Apr 2007
Last Activity: 21 November 2017, 6:58 AM EST
Location: 44.21.48N 80.50.15W
Posts: 2,002
Thanks: 2
Thanked 272 Times in 255 Posts
Couldn't even blame it on Daylight Savings Time. Linux Did you install ntp?
Sponsored Links
    #3  
Old Unix and Linux 01-13-2012   -   Original Discussion by Corona688
Corona688 Corona688 is offline Forum Staff  
Mead Rotor
 
Join Date: Aug 2005
Last Activity: 21 November 2017, 3:22 PM EST
Location: Saskatchewan
Posts: 22,518
Thanks: 1,154
Thanked 4,273 Times in 3,946 Posts
Testing it in dev first. If I mess up a production system, it's hours of driving and sometimes a bit of sledding to fix them in person, especially at this time of year.
    #4  
Old Unix and Linux 01-16-2012   -   Original Discussion by Corona688
admin_xor's Unix or Linux Image
admin_xor admin_xor is offline
Registered User
 
Join Date: Jun 2011
Last Activity: 3 May 2015, 12:12 PM EDT
Posts: 452
Thanks: 13
Thanked 80 Times in 76 Posts
Linux Thanks for sharing your story. It's very true that most of the times we do not bother to check the time of the clock before scheduling stuffs.

We maintain IT infrastructure for a big pharma company. For any SLA (service level agreement) breach, my employer has to pay a real big amount of money to the client. Now that's been told, once my colleague had to schedule a maintenance on an AIX server. We have a procedure to do that. There's a lot of approvals from service delivery managers of both the client and our company required. After getting those, this guy went on scheduling the reboot of the machine in maintenance mode in cron a day before. The next day, I got a call from IT Incident management people saying a server is down before it's scheduled maintenance window. It happened around 20 minutes before the scheduled time. We had to raise a severity for this. Upon checking the root cause of this later, we found somehow the server was failing to sync with the NTP server and the clock was going 20 minutes faster than the actual time.

And yes, because of all these, we breached the SLA! Linux
The Following User Says Thank You to admin_xor For This Useful Post:
nmatt00 (01-30-2012)
Sponsored Links
Closed

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Linux More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
How to get time duration between two human readable time stamp in Unix? ford99 Shell Programming and Scripting 3 07-19-2011 10:52 AM
Shell script to convert epoch time to real time aismann Shell Programming and Scripting 4 04-25-2011 11:09 PM
Convert Epoch Time to Standard Date and Time & Vice Versa DrivesMeCrazy Shell Programming and Scripting 5 02-07-2009 01:40 AM
How To Provide Time Sync Using Nts-150 Time Server On Unix Network? pesty UNIX for Advanced & Expert Users 2 03-22-2007 03:20 AM



All times are GMT -4. The time now is 05:01 PM.