Right On Time, Somewhere


 
Thread Tools Search this Thread
The Lounge War Stories Right On Time, Somewhere
# 1  
Old 01-12-2012
Right On Time, Somewhere

Things like this are teaching me a greater appreciation for network time...

The huge gap between the speed of a satellite connection and its tiny bandwidth allowance means either being completely draconian or having to chide people to not abuse it all the time. We're somewhere inbetween, if a customer downloads too much they'll be slowed down temporarily, but not shut off; if the situation continues, we may phonecall and investigate.

We had a situation where a small community had massively overused their satellite connection pretty much collectively, going over not only our limits but our provider's limits, causing the entire satellite connection to be throttled. We needed to shut the connection down for a few hours before the satellite modem would let go.

We planned it, set a time, and warned our customers. How it worked was very simple -- two entries in root's crontab. At noon that day, the first one would run 'ifconfig eth1 down', taking the community offline but leaving me in communication with the server. At 5pm that day, it would run '/sbin/reboot'.

The server clock had drifted far more than I'd anticipated in the months since its last boot and clock-set, and the shutdown happened one hour early.
# 2  
Old 01-13-2012
Couldn't even blame it on Daylight Savings Time. Smilie Did you install ntp?
# 3  
Old 01-13-2012
Testing it in dev first. If I mess up a production system, it's hours of driving and sometimes a bit of sledding to fix them in person, especially at this time of year.
# 4  
Old 01-16-2012
Smilie Thanks for sharing your story. It's very true that most of the times we do not bother to check the time of the clock before scheduling stuffs.

We maintain IT infrastructure for a big pharma company. For any SLA (service level agreement) breach, my employer has to pay a real big amount of money to the client. Now that's been told, once my colleague had to schedule a maintenance on an AIX server. We have a procedure to do that. There's a lot of approvals from service delivery managers of both the client and our company required. After getting those, this guy went on scheduling the reboot of the machine in maintenance mode in cron a day before. The next day, I got a call from IT Incident management people saying a server is down before it's scheduled maintenance window. It happened around 20 minutes before the scheduled time. We had to raise a severity for this. Upon checking the root cause of this later, we found somehow the server was failing to sync with the NTP server and the clock was going 20 minutes faster than the actual time.

And yes, because of all these, we breached the SLA! Smilie
This User Gave Thanks to admin_xor For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

8 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Calculate Time diff in milli milliseconds(Time format : HH:MM:SS,NNN)

Hi All, I have one file which contains time for request and response. I want to calculate time difference in milliseconds for each line. This file can contain 10K lines. Sample file with 4 lines. for first line. Request Time: 15:23:45,255 Response Time: 15:23:45,258 Time diff... (6 Replies)
Discussion started by: Raza Ali
6 Replies

2. Programming

Find gaps in time data and replace missing time value and column 2 value by interpolation in awk

Dear all, I am kindly seeking assistance on the following issue. I am working with data that is sampled every 0.05 hours (that is 3 minutes intervals) here is a sample data from the file 5.00000 15.5030 5.05000 15.6680 5.10000 16.0100 5.15000 16.3450 5.20000 16.7120 5.25000... (4 Replies)
Discussion started by: malandisa
4 Replies

3. Shell Programming and Scripting

Convert UTC time into current UNIX sever time zone

Hi guys thanks for the help for my previous posts.Now i have a requirement that i download a XMl file which has UTC time stamp.I need to convert UTC time into Unix server timezone. For ex if the time zone of unix server is CDT then i need to convert into CDT.whatever may be the system time... (5 Replies)
Discussion started by: mohanalakshmi
5 Replies

4. Solaris

modifying date and time and time zone on solaris 5.10 with (redundant server) veritas

I have a cluster of two Solaris server (veritas cluster). one working and the other is standby I am going to change the date on them , and am looking for a secure solution as it is giving an important service. my opinion is that the active one doesn't need to be restarted (if I don't change the... (1 Reply)
Discussion started by: barry1946
1 Replies

5. Shell Programming and Scripting

How to calculate time difference between start and end time of a process!

Hello All, I have a problem calculating the time difference between start and end timings...! the timings are given by 24hr format.. Start Date : 08/05/10 12:55 End Date : 08/09/10 06:50 above values are in mm/dd/yy hh:mm format. Now the thing is, 7th(08/07/10) and... (16 Replies)
Discussion started by: smarty86
16 Replies

6. Shell Programming and Scripting

Convert Epoch Time to Standard Date and Time & Vice Versa

Hi guys, I know that this topic has been discuss numerous times, and I have search the net and this forum for it. However, non able to address the problem I faced so far. I am on Solaris Platform and unable to install additional packages like the GNU date and gawk to make use of their... (5 Replies)
Discussion started by: DrivesMeCrazy
5 Replies

7. Shell Programming and Scripting

Start time/end time and status of crontab job

Is there anyway to get the start time and end time / status of a crontab job which was just completed? Of course, we know the start time of the crontab job since we are scheduling. But I would like to know process start and time recorded somewhere or can be fetched from a command like 'ps'. ... (3 Replies)
Discussion started by: thambi
3 Replies

8. UNIX for Advanced & Expert Users

How To Provide Time Sync Using Nts-150 Time Server On Unix Network?

can anybody tel lme,how to instal NTS -150 on a unix network,it needs some patch to fetch time frm serve,,?? (2 Replies)
Discussion started by: pesty
2 Replies
Login or Register to Ask a Question