Today (Saturday) We will make some minor tuning adjustments to MySQL.

You may experience 2 up to 10 seconds "glitch time" when we restart MySQL. We expect to make these adjustments around 1AM Eastern Daylight Saving Time (EDT) US.


Sudden application crash in servers


Login or Register to Reply

 
Thread Tools Search this Thread
# 1  
Sudden application crash in servers

Hi,

This weekend there was a sudden application crash in the server.
I did not know where to start to investigate the problem, so I first looked into the /var/adm/syslog/syslog.log, and this was what I found :

Code:
Dec 17 00:38:02 L28bi01 sshd[126]: error: accept: No buffer space available
Dec 17 00:38:02 L28bi01 sshd[24333]: error: setsockopt SO_KEEPALIVE: Invalid argument
Dec 17 00:38:07 L28bi01 sshd[24379]: error: setsockopt SO_KEEPALIVE: Invalid argument
Dec 17 00:38:21 L28bi01 sshd[24445]: error: PAM: No account present for user for illegal user UlGLXBTX from 10.61.1.55
Dec 17 00:38:21 L28bi01 sshd[24447]: error: PAM: No account present for user for illegal user anonymous from 10.61.1.55
Dec 17 00:38:26 L28bi01 sshd[24511]: error: PAM: No account present for user for illegal user guest from 10.61.1.55
Dec 17 00:38:27 L28bi01 sshd[24515]: error: PAM: No account present for user for illegal user IyoYLEnT from 10.61.1.55
Dec 17 00:38:28 L28bi01 sshd[24517]: error: PAM: No account present for user for illegal user shelladmin from 10.61.1.55
Dec 17 00:38:31 L28bi01 sshd[24524]: error: PAM: Authentication failed for root from 10.61.1.55
Dec 17 00:38:31 L28bi01 sshd[24525]: error: PAM: No account present for user for illegal user netscreen from 10.61.1.55
Dec 17 00:38:33 L28bi01 sshd[24528]: error: PAM: No account present for user for illegal user admin from 10.61.1.55
Dec 17 00:38:38 L28bi01 sshd[24534]: error: PAM: Authentication failed for root from 10.61.1.55
Dec 17 00:38:58 L28bi01 sshd[24542]: error: PAM: No account present for user for illegal user admin1 from 10.61.1.55
Dec 17 00:39:06 L28bi01 sshd[24552]: error: PAM: No account present for user for illegal user admin from 10.61.1.55
Dec 17 00:39:18 L28bi01 sshd[24561]: error: PAM: No account present for user for illegal user emailswitch from 10.61.1.55
Dec 17 00:39:22 L28bi01 sshd[24584]: error: PAM: No account present for user for illegal user product from 10.61.1.55
Dec 17 00:39:23 L28bi01 sshd[24599]: error: PAM: No account present for user for illegal user admin from 10.61.1.55
Dec 17 00:39:27 L28bi01 sshd[24621]: error: PAM: Authentication failed for root from 10.61.1.55
Dec 17 00:39:29 L28bi01 sshd[24626]: error: PAM: No account present for user for illegal user n3ssus from 10.61.1.55
Dec 17 00:39:31 L28bi01 sshd[24632]: error: PAM: Authentication failed for root from 10.61.1.55
Dec 17 00:41:01 L28bi01 sshd[126]: error: accept: No buffer space available
Dec 17 00:41:01 L28bi01 sshd[25366]: error: setsockopt SO_KEEPALIVE: Invalid argument
Dec 17 00:41:55 L28bi01 sshd[26128]: error: PAM: No account present for user for illegal user cisco from 10.61.1.55
Dec 17 00:42:00 L28bi01 sshd[26134]: error: PAM: No account present for user for illegal user Cisco from 10.61.1.55
Dec 17 00:42:02 L28bi01 sshd[26142]: error: PAM: No account present for user for illegal user admin from 10.61.1.55
Dec 17 00:42:04 L28bi01 sshd[26175]: error: PAM: No account present for user for illegal user  from 10.61.1.55
Dec 17 00:42:10 L28bi01 sshd[26254]: error: PAM: No account present for user for illegal user manage from 10.61.1.55
Dec 17 00:42:15 L28bi01 sshd[26273]: error: PAM: No account present for user for illegal user monitor from 10.61.1.55
Dec 17 00:42:19 L28bi01 sshd[26280]: error: PAM: No account present for user for illegal user ftp from 10.61.1.55
Dec 17 00:42:54 L28bi01 sshd[26792]: error: PAM: No account present for user for illegal user Fortimanager_Access from 10.61.1.55
Dec 17 00:42:54 L28bi01 sshd[26791]: error: PAM: No account present for user for illegal user nessus_oJgOWh46 from 10.61.1.55
Dec 17 00:42:56 L28bi01 sshd[26791]: error: PAM: No account present for user for illegal user nessus_oJgOWh46 from 10.61.1.55
Dec 17 00:43:27 L28bi01 sshd[26926]: error: setsockopt SO_KEEPALIVE: Invalid argument

The error that is most related to this problem is "No buffer space available".
When I googled this error, there was no solid solution, some say memory pressure, and some say check the kernel value "tcp_conn_request_max" but I do not see this value present at all in the server.

However, the application logs present this error :

Code:
File: data.c, Line: 2963, Time: 2017.12.17 00:36:56, RC: -23

Text:    CL_receive_message failed

Error during 'read'

  System error: Connection timed out

 

File: data.c, Line: 2963, Time: 2017.12.17 00:37:46, RC: -23

Text:    CL_receive_message failed

Error during 'read'

  System error: Connection timed out

 

File: data.c, Line: 2963, Time: 2017.12.17 00:37:46, RC: -23

Text:    CL_receive_message failed

Error during 'read'

  System error: Connection timed out

 

File: data.c, Line: 825, Time: 2017.12.17 00:38:52, RC: -28

Text:

Connection between client and server was terminated

 

File: data.c, Line: 918, Time: 2017.12.17 00:38:52, RC: -28

Text:

Connection between client and server was terminated

 

File: data.c, Line: 3564, Time: 2017.12.17 00:43:27, RC: -20

Text:

Socket option error

  System error: Invalid argument

 

File: dta_ids.c, Line: 4027, Time: 2017.12.17 00:43:27, RC: 0

Text:    DaTA shutting down: ids clients finished

 

 

File: dta_ids.c, Line: 4052, Time: 2017.12.17 00:43:28, RC: 0

Text:    DaTA shutting down: std clients finished

 

 

File: dta_ids.c, Line: 4078, Time: 2017.12.17 00:43:31, RC: 0

Text:    DaTA shutting down: file queues synchronized

Could this be a network issue?
How do I investigate this problem, I need to know the RCA of it. Please help.
# 2  
Q:
Are you using (in your site, that is..) a "security" device thats "scans" for possible security issues?
# 3  
Possibly the system has run out of network buffer resources. Has the load/number of users been steadily increasing?

Network configurations determine the number of buffers available for network packets (of various different sizes) arriving and departing, and also the maximum number of connections.
# 4  
Exactly and some security scanning devices though said "non intrusive" manage to get you in such embarrassing situation...Why do these messages occurs?:
Code:
No account present for user for illegal user admin from 10.61.1.55

What is behind that IP address?...
# 5  
You need to tell us all what hardware this is, what O/S you are running and which version. Give us a clue!!

EDIT: Okay, you've posted to the HP-UX forum, got that much.
# 6  
Tons of login attempts to various users would make me very suspicious, even though they seem to occur from the local area network. Even more as they include four failed attempts for root.
Identify the machine that attempts come from and check it for malware.
# 7  
Hi,

I have seen almost exactly this before, it was a product called "Foundstone" - the Wintel team had deployed this straight out of the box and it caused mayhem on the Unix Estate.

I would think that this one of two things.
  1. You have a security breach and you're going to have a significant issue.
  2. Or from what I can see there is some kind of Scanner running and it needs to be configured or stopped before it starts locking accounts and causing other issues.

I would be tempted to speak to the other teams and find out what has changed. Also watch for it happening on a regular basis "Weekly, Monthly etc".

It could also be that someone has got something to evaluate and don't understand the implications.

As advised earlier in the thread, find the machine and beat the user up - you have an excuse!

Regards

Gull04
Login or Register to Reply

|
Thread Tools Search this Thread
Search this Thread:
Advanced Search

8 More Discussions You Might Find Interesting

1. Red Hat

XCB crash while Video Rendering in multithreded application

I encounter the following crash on RHEL 7.0 when I run a multithreaded video rendering application using GLFW and OpenGL. OpenGL version is 2.1 and MESA version is 9.3.0 Following is the back trace of the multi-threaded program I am working on:... (0 Replies)
Discussion started by: anuachin
0 Replies

2. Solaris

T4-1/Solaris 11 and sudden 50% performance drop

We have a SPARC T4-1 server, running Solaris 11, and it's doing some pretty extensive parsing on roughly 100GB data set. All was well still few weeks ago, when I was testing the performance, I was reaching rougly 50minute calculation times, and it was more or less expected performance. Now... (0 Replies)
Discussion started by: julumme
0 Replies

3. Programming

Client/Server Socket Application - Preventing Client from quitting on server crash

Problem - Linux Client/Server Socket Application: Preventing Client from quitting on server crash Hi, I am writing a Linux socket Server and Client using TCP protocol on Ubuntu 9.04 x64. I am having problem trying to implement a scenario where the client should keep running even when the... (2 Replies)
Discussion started by: varun.nagpaal
2 Replies

4. Programming

socket system call can not succedd right after application crash.

hello all, I have developed a server application in C for ulinux kernel 2.6.It works very fine; creating a socket, binding it to a port, listening for incoming sockets and accepting them ,all finish without any error. But there is a problem regarding application crash.After an intentionally... (1 Reply)
Discussion started by: Sedighzadeh
1 Replies

5. Shell Programming and Scripting

Difference between development and Production unix servers for a application??

Hi all I am running a major script of my application in development for implementing code changes for process improvement in time. The script runs in production once in a month . It takes 8 hours 30 mins in Production server . what surprice me is , when I run the same script in development server... (9 Replies)
Discussion started by: sakthifire
9 Replies

6. UNIX for Advanced & Expert Users

Application Servers Installation and Users

Hi, When most of the server applications get installed, they create their own user. I believe this is to not use the "root" account. For example, Apache when installed creates a user called "apache". And the directories which it uses are all owned by this user. This seems to be the... (2 Replies)
Discussion started by: srikanths
2 Replies

7. UNIX for Dummies Questions & Answers

Application servers, proper usage

Is it common in the Unix/Linux environment to install compute intensive applications on a Server system and have the client machines download the executables into memory at runtime to run locally? This model seems taxing to the network, and as I understand, has been largely abandoned in the... (1 Reply)
Discussion started by: jonwillog
1 Replies

8. UNIX for Dummies Questions & Answers

sudden poweroff ??

Hi I am using unix for last few days. Here is my problem during boot the machine stop giving video signal and I don't know what's happening. When I ping (during boot) it from another machine it comes alive then goes out. The power on the CPU is on all the time. please help. (9 Replies)
Discussion started by: santosh1981
9 Replies

Featured Tech Videos