Login or Register to Ask a Question and Join Our Community


Solaris

System date sudden change

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Operating Systems Solaris System date sudden change
# 1  
Old 06-17-2010
System date sudden change
Hi there,

We have a Solaris 10 machine which has been up and running for more than 400 days. A strange behaviour happened. The system date defaulted to epoch timestamp. Oracle stopped and application failed causing management to parade. We managed to reset the date. All other servers and including the problem one are synced with ntp server.

My question is how to investigate the sudden change in system date?
What caused it to go haywire? There is nothing in the logs to indicate anyone modified anything.

Below is an excerpt of messages:

Code:
Jun 16 03:40:50 serversun xmsTrigger: [ID 839465 user.notice]
xmsTrigger(18788) NOTICE: Path ptx:0 prx:0 ltx:3 lrx:0 'MMX_REDIRECT_serversun' 
Jun 16 03:40:50 serversun xmsTrigger: [ID 839465 user.notice]
xmsTrigger(18788) NOTICE: Path ptx:0 prx:0 ltx:0 lrx:1 'MMX_REDIRECT_serversun4' 
Jun 16 03:40:50 serversun xmsTrigger: [ID 839465 user.notice]
xmsTrigger(18788) NOTICE: Path ptx:0 prx:0 ltx:0 lrx:1 'MMX_REDIRECT_serversun1' 
Jun 16 03:40:50 serversun xmsTrigger: [ID 839465 user.notice]
xmsTrigger(18788) NOTICE: Path ptx:0 prx:0 ltx:0 lrx:1 'MMX_REDIRECT_serversun2' 
Jun 16 03:40:51 serversun xmsTrigger: [ID 839465 user.notice]
xmsTrigger(18788) NOTICE: Maximum SMS/s rate in the last period: 2 Jun 16 03:40:54 serversun sua_if: [ID 761583 user.error] sua_if(18789) ERROR =11 TcSccpCommonInterface.cc564:  Unrecognised Transaction ID received (29480), TCAP Type (101) Jun 16 03:40:54 serversun sua_if: [ID 761583 user.error] sua_if(18789) ERROR =11 TcSccpCommonInterface.cc564:  Unrecognised Transaction ID received (29480), TCAP Type (101) Jun 16 03:41:04 serversun sua_if: [ID 761583 user.error] sua_if(18789) ERROR =11 TcSccpCommonInterface.cc564:  Unrecognised Transaction ID received (28438), TCAP Type (101) Jun 16 03:41:04 serversun sua_if: [ID 761583 user.error] sua_if(18789) ERROR =11 TcSccpCommonInterface.cc564:  Unrecognised Transaction ID received (28438), TCAP Type (101) Jun 16 03:41:06 serversun xmsTrigger: [ID 839465 user.notice]
xmsTrigger(18788) NOTICE: Maximum SMS/s rate in the last period: 2 Jun 16 03:41:17 serversun last message repeated 1 time Jun 16 03:41:26 serversun xmsTrigger: [ID 839465 user.notice]
xmsTrigger(18788) NOTICE: Your system is running within allowed limits (maximum concurrent calls: 1000, currently: 1; maximum SMS/sec: 1000, currently 2) Jun 16 03:41:27 serversun xmsTrigger: [ID 839465 user.notice]
xmsTrigger(18788) NOTICE: Maximum SMS/s rate in the last period: 2 Jun 16 03:41:47 serversun last message repeated 2 times Nov  5 22:17:32 serversun diameterBeClient: [ID 839465 user.notice]
diameterBeClient(18795) NOTICE: {970020} DCD DIAMETER_MIN_LATENCY in the last period for host sunsvr was 3748 Nov  5 22:17:32 serversun diameterBeClient: [ID 839465 user.notice]
diameterBeClient(18795) NOTICE: {970021} DCD DIAMETER_MAX_LATENCY in the last period for host sunsvr was 114060 Nov  5 22:17:32 serversun diameterBeClient: [ID 839465 user.notice]
diameterBeClient(18795) NOTICE: {970022} DCD DIAMETER_AVERAGE_LATENCY in the last period for host sunsvr was 7036 Nov  5 22:17:32 serversun oraPStoreCleaner: [ID 839465 user.notice]
oraPStoreCleaner(2103) NOTICE: Shutting down from daemon mode.
Nov  5 22:17:33 serversun oraPStoreCleaner: [ID 761583 user.error]
oraPStoreCleaner(2103) ERROR Oracle=-3113 oraConnect.pc@231: delete  from cmn_persistent_store  where date_inserted<=TO_DATE(:b0,'Y: ORA-03113:
end-of-file on communication channel                       
Nov  5 22:17:33 serversun oraPStoreCleaner: [ID 761583 user.error]
oraPStoreCleaner(2103) ERROR Oracle=-3113 oraConnect.pc@231: delete  from cmn_persistent_store  where date_inserted<=TO_DATE(:b0,'Y: ORA-03113:
end-of-file on communication channel                       
Nov  5 22:17:33 serversun UssdMfileD: [ID 438867 user.warning]
UssdMfileD(2105) WARNING OEXEC, OEXN=3113 cmnMfileDaemonApi.cc@612: ORA-03113:
end-of-file on communication channel
Nov  5 22:17:33 serversun UssdMfileD: [ID 438867 user.warning]
UssdMfileD(2106) WARNING OEXEC, OEXN=3113 cmnMfileDaemonApi.cc@612: ORA-03113:
end-of-file on communication channel
Nov  5 22:17:33 serversun slee_acs: [ID 675952 user.error] slee_acs(18780)
ERROR: CCS LoadService: Error getting account data (by CLI) Nov  5 22:17:33 serversun slee_acs: [ID 675952 user.error] slee_acs(18780)
ERROR: CCS LoadService: Error getting account data (by CLI) Nov  5 22:17:33 serversun slee_acs: [ID 675952 user.error] slee_acs(18780)
ERROR: ORACLE error--
Nov  5 22:17:33 serversun slee_acs: [ID 675952 user.error] slee_acs(18780)
ERROR: ORACLE error--
Nov  5 22:17:33 serversun slee_acs: [ID 675952 user.error] slee_acs(18780)
ERROR: ORA-03114: not connected to ORACLE Nov  5 22:17:33 serversun slee_acs: [ID 675952 user.error] slee_acs(18780)
ERROR: ORA-03114: not connected to ORACLE Nov  5 22:17:33 serversun slee_acs: [ID 953149 user.warning]
slee_acs(18780) WARNING: acsChassisPerformLoad: CCS - no call plan found (CLN '502181071854383', CPN '60182672937') Nov  5 22:17:33 serversun slee_acs: [ID 675952 user.error] slee_acs(18780)
ERROR: CCS LoadService: Error getting account data (by CLI) Nov  5 22:17:33 serversun slee_acs: [ID 675952 user.error] slee_acs(18780)
ERROR: CCS LoadService: Error getting account data (by CLI) Nov  5 22:17:33 serversun slee_acs: [ID 675952 user.error] slee_acs(18780)
ERROR: ORACLE error--
Nov  5 22:17:33 serversun slee_acs: [ID 675952 user.error] slee_acs(18780)
ERROR: ORACLE error--
Nov  5 22:17:33 serversun slee_acs: [ID 675952 user.error] slee_acs(18780)
ERROR: ORA-03114: not connected to ORACLE Nov  5 22:17:33 serversun slee_acs: [ID 675952 user.error] slee_acs(18780)
ERROR: ORA-03114: not connected to ORACLE Nov  5 22:17:33 serversun slee_acs: [ID 953149 user.warning]
slee_acs(18780) WARNING: acsChassisPerformLoad: CCS - no call plan found (CLN '502181071736437', CPN '60189628114') Nov  5 22:17:33 serversun slee_acs: [ID 675952 user.error] slee_acs(18780)
ERROR: CCS LoadService: Error getting account data (by CLI) Nov  5 22:17:33 serversun slee_acs: [ID 675952 user.error] slee_acs(18780)
ERROR: CCS LoadService: Error getting account data (by CLI) Nov  5 22:17:33 serversun slee_acs: [ID 675952 user.error] slee_acs(18780)
ERROR: ORACLE error--
Nov  5 22:17:33 serversun slee_acs: [ID 675952 user.error] slee_acs(18780)
ERROR: ORACLE error--
Nov  5 22:17:33 serversun slee_acs: [ID 675952 user.error] slee_acs(18780)
ERROR: ORA-03114: not connected to ORACLE Nov  5 22:17:33 serversun slee_acs: [ID 675952 user.error] slee_acs(18780)
ERROR: ORA-03114: not connected to ORACLE Nov  5 22:17:33 serversun slee_acs: [ID 953149 user.warning]
slee_acs(18780) WARNING: acsChassisPerformLoad: CCS - no call plan found (CLN '502181072000755', CPN '60182648809') Jan  1 00:00:01 serversun diameterBeClient: [ID 839465 user.notice]
diameterBeClient(18795) NOTICE: {970020} DCD DIAMETER_MIN_LATENCY in the last period for host sunsvr was 694 Jan  1 00:00:01 serversun diameterBeClient: [ID 839465 user.notice]
diameterBeClient(18795) NOTICE: {970021} DCD DIAMETER_MAX_LATENCY in the last period for host sunsvr was 9492 Jan  1 00:00:01 serversun diameterBeClient: [ID 839465 user.notice]
diameterBeClient(18795) NOTICE: {970022} DCD DIAMETER_AVERAGE_LATENCY in the last period for host sunsvr was 2910 Jan  1 00:00:01 serversun xmsTrigger: [ID 839465 user.notice]
xmsTrigger(18788) NOTICE: Your system is running within allowed limits (maximum concurrent calls: 1000, currently: 1; maximum SMS/sec: 1000, currently 1) Jan  1 00:00:01 serversun slee_acs: [ID 953149 user.warning]
slee_acs(18780) WARNING: ccs_getGlobalCapabilities: Error retrieving capabilities.
Jan  1 00:00:01 serversun sua_if: [ID 761583 user.error] sua_if(18789) ERROR =11 TcSccpCommonInterface.cc564:  Unrecognised Transaction ID received (32614), TCAP Type (101) Jan  1 00:00:01 serversun sua_if: [ID 761583 user.error] sua_if(18789) ERROR =11 TcSccpCommonInterface.cc564:  Unrecognised Transaction ID received (32614), TCAP Type (101) Jan  1 00:00:01 serversun oraPStoreCleaner: [ID 675952 user.error]
oraPStoreCleaner(11491) ERROR: Could not connect to the database Jan  1 00:00:01 serversun oraPStoreCleaner: [ID 675952 user.error]
oraPStoreCleaner(11491) ERROR: Could not connect to the database Jan  1 00:00:02 serversun slee_acs: [ID 953149 user.warning]
slee_acs(18781) WARNING: ccs_getGlobalCapabilities: Error retrieving capabilities.
Jan  1 00:00:02 serversun oraPStoreCleaner: [ID 675952 user.error]
oraPStoreCleaner(11497) ERROR: Could not connect to the database Jan  1 00:00:02 serversun oraPStoreCleaner: [ID 675952 user.error]
oraPStoreCleaner(11497) ERROR: Could not connect to the database Jan  1 00:00:02 serversun oraPStoreCleaner: [ID 675952 user.error]
oraPStoreCleaner(11503) ERROR: Could not connect to the database Jan  1 00:00:02 serversun oraPStoreCleaner: [ID 675952 user.error]
oraPStoreCleaner(11503) ERROR: Could not connect to the database ....

Can anyone tell where to start?

Thanks for your help.
# 2  
Old 06-17-2010
What model server is it?
# 3  
Old 06-17-2010
Hi.

Could an admin have modified the date (using date(1). That could certainly screw up Oracle if going backwards. A corrupt /etc/utmp file would show very long uptime, but not sure if that would cause Oracle any problems (and in any case would correct itself after a reboot).
Last edited by Scott; 06-17-2010 at 04:53 PM..
# 4  
Old 06-18-2010
Thank you for the responses.
@garskoci
The model is:
SunOS ..Generic_120011-14 sun4u sparc SUNW,Sun-Fire-V440

@scottn
"Could an admin have modified the date " No.
The usual system logs doesn't show any useful information.
# 5  
Old 06-18-2010
possible causes:

someone might have logged onto this system and changed the time :

the battery might have gone wild on sys-board

the application has caused this weird behavior
# 6  
Old 06-19-2010
is your system using any NTP setup?
# 7  
Old 06-19-2010
Quote:
Originally Posted by incredible
is your system using any NTP setup?
Good question.

Yes. there are many other servers syncing with the same NTP server.
Unfortunately this one misbehaved. I am suspecting the application
which may have back dated data as transaction id in the TCAP packet. Well, it's a wild guess.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

How to change existing date to current date in a filename?

Suppose i have a list of files in a directory as mentioned below 1. Shankar_04152019_ny.txt 2. Gopi_shan_03122019_mi.txt 3. Siva_mourya_02242019_nd.txt .. . . . . 1000 . Jiva_surya_02282019_nd.txt query : At one shot i want to modify the above all filenames present in one path with... (4 Replies)
Discussion started by: Shankar455
4 Replies

2. Answers to Frequently Asked Questions

Compare date in .txt with system date and remove if it's lesser than system date

I m working on shell scripting and I m stuck where in my .txt file there is column as expiry date and I need to compare that date with system date and need to remove all the rows where expiry date is less than system date and create a new .txt with update. (1 Reply)
Discussion started by: Stuti
1 Replies

3. UNIX for Beginners Questions & Answers

Compare date in .txt with system date and remove if it's lesser than system date

Can someone help me with the code wherein there is a file f1.txt with different column and 34 column have expiry date and I need to get that and compare with system date and if expiry date is <system date remove those rows and other rows should be moved to new file f2.txt . I don't want to delete... (2 Replies)
Discussion started by: Stuti
2 Replies

4. Shell Programming and Scripting

Change the content of files but not change the date

I have 100 files in a directory , all the files have a word "error" and they are created in different date . Now I would like to change the word from "error" to "warning" , and keep the date of the files ( that means do not change the file creation date after change the word ) , can advise what can... (7 Replies)
Discussion started by: ust3
7 Replies

5. Shell Programming and Scripting

Change the content of files but not change the date

I have 100 files in a directory , all the files have a word "error" and they are created in different date . Now I would like to change the word from "error" to "warning" , and keep the date of the files ( that means do not change the file creation date after change the word ) , can advise what can... (0 Replies)
Discussion started by: ust3
0 Replies

6. UNIX for Dummies Questions & Answers

Delete a row from a file if one column containing a date is greater than the current system date

Hello gurus, I am hoping someone can help me with the required code/script to make this work. I have the following file with records starting at line 4: NETW~US60~000000000013220694~002~~IT~USD~2.24~20110201~99991231~01~01~20101104~... (4 Replies)
Discussion started by: chumsky
4 Replies

7. UNIX for Advanced & Expert Users

Change of System Date format

I have this T5220 server with Solaris 10 installed on to it. Now when I fire the command "date" the o/p I get is as follows : bash-3.00# date Tuesday, February 9, 2010 7:25:24 PM IST whereas I want the date to appear in this format: bash-3.00# date Tue Feb 9 19:26:06 IST 2010 ... (3 Replies)
Discussion started by: ksahay@123
3 Replies

8. Solaris

change system date in solaris

dears any one know how to change system date in solaris (10 Replies)
Discussion started by: murad.jaber
10 Replies

9. UNIX for Dummies Questions & Answers

Move A File With Same Date,don't Change The Desitination Dir Date

Assume, I created one file three years back and I like to move the file to some other directory with the old date (Creation date)? Is it possible? Explain? (1 Reply)
Discussion started by: jee.ku2
1 Replies

10. AIX

Change System Date

Hi, anybody know how to change the system date which the year is greater than 2037? is it posibble been done under aix machine?? thanks :) (1 Reply)
Discussion started by: Ricole
1 Replies
Login or Register to Ask a Question