Facing file processing timeouts


 
Thread Tools Search this Thread
Top Forums UNIX for Advanced & Expert Users Facing file processing timeouts
# 1  
Old 11-12-2007
Facing file processing timeouts

Hi

In our ETL application we have used simple scripts to move & split the files. We are randomly facing timeouts on these jobs. It indicates we are running out of resources.

How should I confirm the resources are inadequate?

I know we have commands like vmstat & iostat which displays performance stats. Problem with them they mention some disks in their outputs, but I am not sure how the paths/dirs those files are being operated maps to those disks?

Any suggestion to track this?
# 2  
Old 11-12-2007
I love chasing performance issues Smilie Fisrt, what exactly operations you are doing ? What language is being used for these operations ? How big are the files, where do you see the timeouts, how powerful is your hardware ? What is your OS ?
Use "top" command to determine what's going on like :
"top -p PID-number, second-PID-number"
replace the PID-number with the respective PIDs of your scripts and you'll see where the resources are going.
# 3  
Old 11-12-2007
We have simple shell scripts which splits the file & move them.

The ETL application we are using is DataStage. Though we see timeouts at DataStage & reported by ControlM as well we had seen this issue with simple 'mv' commands.

OS we are using is AIX.

As you suggested ... If I type 'man top' I get message as manual entry is not installed.

How would you find out ... how powerful or configuration of hardware? Mostly I understand the problem surrounds to hardware limitation. But before I can comment on that I should provide numbers like we are have so many GBs of memory % of utilization is done ...
# 4  
Old 11-13-2007
use "topas" instead, that's why I asked about the OS.
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. IP Networking

Problems and doubts with sockets and timeouts

Hi! I need some help to understand a little bit more the behaviour about socket and TCP connections... Here is my problem I have a client and a server that were written in python. The server program wait until a message arrive and then print the message but if the message not arrive in a second... (2 Replies)
Discussion started by: Kovalevski
2 Replies

2. UNIX for Dummies Questions & Answers

awk - Rename output file, after processing, same as input file

I have one input file ABC.txt and one output DEF.txt. After the ABC is processed and created output, I want to rename ABC.txt to ABC.orig and DEF to ABC.txt. Currently when I am doing this, it does not process the input file as it cannot read and write to the same file. How can I achieve this? ... (12 Replies)
Discussion started by: High-T
12 Replies

3. UNIX for Dummies Questions & Answers

Facing difficulties to untar a tra.gz file

Hi , I trying to cp a files to one location and trying to untar there,the tar.gz files get copied to the specified location but wont untar. Here is what i did if cp /path/of/the/file.tar.gz /path/to/the/file/file.tar.gz then tar -zxvf /path/to/the/file/file.tar.gz echo "FILE... (2 Replies)
Discussion started by: vikatakavi
2 Replies

4. Programming

awk processing / Shell Script Processing to remove columns text file

Hello, I extracted a list of files in a directory with the command ls . However this is not my computer, so the ls functionality has been revamped so that it gives the filesizes in front like this : This is the output of ls command : I stored the output in a file filelist 1.1M... (5 Replies)
Discussion started by: ajayram
5 Replies

5. Shell Programming and Scripting

Timeouts in expect script

Hi all. I have an expect script which for walks across servers, checks freespace and provides sorted list of biggest files #!/usr/bin/expect -- set timeout 600 stty -echo log_user 0 spawn -noecho sudo -u introot /home/introot/bin/twssh expect root send_user "Entering \r" send "uname\r"... (15 Replies)
Discussion started by: urello
15 Replies

6. Shell Programming and Scripting

How to make parallel processing rather than serial processing ??

Hello everybody, I have a little problem with one of my program. I made a plugin for collectd (a stats collector for my servers) but I have a problem to make it run in parallel. My program gathers stats from logs, so it needs to run in background waiting for any new lines added in the log... (0 Replies)
Discussion started by: Samb95
0 Replies

7. Shell Programming and Scripting

Facing problems when FTPING file from server to local (Very Urgent)

Hi, when i am trying to FTP one file (test.txt) from Unix server to my local machine then i am getting the following error, as shown below ftp> binary 200 Type set to I. ftp> put C:\FTP\test.txt C:FTPtest.txt: No such file or directory ftp> The red color text is the error. Can... (3 Replies)
Discussion started by: sunitachoudhury
3 Replies

8. UNIX for Dummies Questions & Answers

Device Timeouts on Unix Server

Hi I am brand new to the forum I have very limited understanding of unix, but am very computer literate. At work we have a unix server which has been the bane of our lives. We have stopped being able to access the network drives and the screen has lots of device timeouts. I have learned how to do... (1 Reply)
Discussion started by: scovell01
1 Replies

9. Cybersecurity

UNIX Network timeouts

Hi, can anyone point me in the right direction regarding a UNIX setting that determines when a network connection will timeout? I am getting network timeouts and I would like to know if there is a setting in UNIX 11i OS that I can modify to increase the time limit. Thank you (4 Replies)
Discussion started by: Allano
4 Replies
Login or Register to Ask a Question