UNIX for Dummies Questions & Answers

I have recently been assigned to help maintain some testing software on two UNIX stations. The stations are connected to instruments that supply power, signals, and take measurements from a box. Both test stations are basically the same. I was given a crash course (read a little training) a couple of weeks ago, introducing me to the proprietary software environment that test the boxes, and a little UNIX information. The UNIX stations, called homer and lisa, are stand-alone machines, that are networked together. I can type “rusers” and see both computers, and think I have been able to ftp files from one to another. The crash course came as there was a problem with the testing software. The training was supplied by another software engineer who had written and maintained the testing software until about 5 years ago, and has moved onto other projects. During the time he was training me, he was fixing the big problems, and giving me some little things to do. He was working on lisa. I was working on homer. We made copies of the testing software to make the changes. This is the main difference between stations: what copies have been made. I asked my trainer what he did before he left, and this is what he told me:
The last thing I attempted on lisa was to make a backup of the system using the tape drive. I tried a command in the "tar" utility that tells it to stop after so many blocks and it is suppose instruct you to insert another tape into the drive. When I came back the next day, the "tar" had aborted and displayed an error message. I would have thought that rebooting the tester would clear any problems that the "tar" had created. If nothing else seems to work, I would look up the "tar" commands and see if there is any command that "resets" the tar utility.

The next time we logged on and then off lisa, we started getting a message

Code:

--Notice: alloc: /:file system full

I was given the suggestion of trying “df –k” which gave these results: (ignore the exact numbers. I took a screen shot with my phone camera, and am just trying to read the blurry results. The capacity % is correct. Other numbers are not exact, except for 0. I can tell that one.)

Code:

Filesystem Kbytes used avail capacity
/dev/dsk/c0t0d0s0 106xxx 105xxx 0 100%
/dev/dsk/c0t0d0s6 507xxx 365xxx 142xxx 74%
/proc 0 0 0 0%
Fd 0 0 0 0%
/dev/dsk/c0t0d0s7 7138xxx 5436xxx 1568xxx 78%
Swap 434xxx 72 434xxx 1%

I also did the “df –k” on homer, and it came up very similar. There was a difference of a few bytes, and maybe changed the capacity of the s6 and s7 a percentage point or so, but nothing major. So I am confused as to what the “file system full message” is telling me. Why isn’t it on both? What am I missing?
Being corrupted by Windows, I thought, maybe if I reboot the computer, it might reset the problem. I asked the technician who does the tests how to reboot, or restart UNIX, as I didn’t know. He didn’t either. We powered down the main tester, which is done in the main part of environment, logged out of UNIX. Then I pushed the power button. I guess there is a shutdown command I should have used, correct? I turned it back on, and saw some more messages during the boot-up process.

Code:

--Notice: realloccg /:file system full

And

Code:

--Oct 17 10:39:41 lisa syslogd: /var/admin/messages: no space left on device

Then I tried to login, and received this message:

Code:

--No utmpx entry: You must exec “login” from the lowest “shell”

From a little research, I see that there are different files for containing login and logout information. That is the utmp, utmpx, wtmp, and wtmpx, and a few others. According to my research, I can and should clear out the contents of these when they get to big. I was finally given a way to login, and these are the steps I went through, and what I did after I logged in.
Stop-a
Typed “boot –s”
Logged in as single user root, with the root password
Typed “df –k”
Typed “fsck –y /dev/dsk/c0t0d0s0, which fixed a problem.
<cntl> d
And tried to login again, and still received the “no utmpx entry” message. So I went through the above process again, until the <cntl> d. I stayed logged in and tried to look at the utmp, utmpx, wtmp, and wtmpx files. In the research, I saw that there are several directories to look:

Code:

-/dev/adm
-/usr/adm
-/var/adm

I don’t understand why there are at least three places to find these files? I did a “ls –l” in each of those directories, and everything looked the same. I wonder if they are the same directory, just being pointed to, not actually different places? Any comments? Maybe I will try to see this in a little while. Anyway, umpt was around 72, utmpx was about 10x that, at around 740, wmpt was around 7100 and wmptx was 74000. I can’t remember exact sizes, I do remember that they were about this magnitude different from each other. I think that I compared this to homer, and saw very similar numbers. So I am still confused as to why lisa gives me “file system full” messages and homer doesn’t.
So if you got through reading this, I will say, “URGENT!!” and “help me!!!!!” but of course, that is why I am here.
I will summarize my questions, in case you didn’t see them.

I think this is all my questions for now. I appreciate any help you can give me, as I am just sitting around banging my head against a wall trying to figure this out.

Thanks,
Brian

Moderator's Comments:

Use code tags please. You will get a PM with instructions to use them.

I'm sorry the font you use is too small for my poor eyes...
I will summarize a bit for I cannot read all your post (really too small for me...)
You had an issue you suspect coming from tar usage, yes?
/ is 100% full

Just here, if its a station I would suspect a big file being created in /dev where a default tape device should be..
You can only boot in single user if / is 100% full, so you are to remove unwanted files to recreate enough space for the system
I you have a /var that is full, so the same (cleanup) for the system needs place to write its logs otherwise the log files risk corruption
That will be the second part I suppose but until you have solved your / full, there is nothing else to do

I am sorry that it was so small. It didn't look that small when I first was writing it. I have increased the font size. I hope that helps.

I do think that some of the issue might be a tar problem. The root looks to be 100% full. My question comes because of confusion. My other station is almost the same, being 100% full on the root, but it doesn't have the same problems. But is my best bet just to delete the contents of the wtmp and the wtmpx, which I found to be 707464 and 7249920 respectively. Those are the big files. Is that the best way to start?

Thanks,
Brian

Well / can be full...but still have enough room for root user to cleanup before disaster...
e.g. normally with HFS you used to have 10% reserved ( full for all users except root...).
With vxfs full at 95% you would be unable to extend the file system for lack of space for reorg...
Now I doubt it being because of wtmp or wtmpx for you would most certainly be unable to connect if you were not root... and you would have system logs warning you about the near coming issue, all I can think of is look for a specific file especially if someone used tar to backup/archive and most common issue is someone misspelling the tape device path, I had long ago a very important server crash because of that... I was lucky finding in /dev a /dev/tape when the system had 2 devices: a spooltape device and a DAT device and the path should have been something starting by /dev/rmt/XXXX
/dev is not very big type

Code:

 du -sk /dev

if its more than 60MB then the culprit is hidden somewhere there... if /var is full then some cleaning is necessary to be able to use vi for a start (on HP anyway...) all the system logs need to be able to write!
Your system look like a solaris is it so?

For information solaris 10:

Code:

slo4:/dev $ du -sk .
118     .
slo4:/dev $ uname -r
5.10

This time you did a clean shutdown /reboot in single user mode, then (fsck) you did a file system check on the disk as a whole (I think - I've been using since 1994 only LVM...) because to speed up its access UNIX caches the file system and so cutting the current the way you did the first time let the poor thing with all its pending writes and files opened in a bad situation...

You asked about the system I am on. I was going to put it in the first post, and forgot there as well. Here it is: Sun OS 5.6, version generic [UNIX(R) System V release 4.0], August 1997.

I did the du -sk /dev, and saw that it is over 60 MB. Is there files I should stay away from cleaning? Should I also clean the contents of the utmp, utmpx, wtmp, and wtmpx? Is anything in those files really important for me to keep?

Thanks again.

So its a solaris 6... (wow I dont know if I can find something similar...)
If you read my previous post, /dev should be quite small... so I'm sure the culprit is there.. Let's not touch anything else for the moment.
Do you have a /dev/tape file ? What size?
What do you have in /dev/rmt ?