Sponsored Content
Top Forums Programming fwrite takes extremely long time Post 77188 by jim mcnamara on Tuesday 5th of July 2005 03:57:39 PM
Old 07-05-2005
Real time I/O processing usually uses the aio calls. These let you make a write call while you are doing something else. Normally these processes run at special priority, so they can get the CPU whenever they need it.

As for IPC - interprocess communication - shared memory is very fast. The idea of message queues is also very good because you can send messages without regard to how fast they are being read. As long as they do get read....

However there is a limit to I/O throughput.

It actually sounds like your system is I/O bound, if you know that term. If you have 900ms waits on I/O completion, you are going to have a problem no matter what design you implement. That is about 13 x 30 x 30ms "sleep" cycles. Which will create another condition requiring a disk write by the kernel as soon as it comes back from a write. Every one of your writes (assuming an 8 byte double and all 10000 array elements are written) is 100 X 100 X 8 bytes = 80 thousand bytes every 30ms.

13 requests ~ 1MB of data every 900ms. This is on top of all the other system I/O.
You may want to have a sysadmin look at monitoring I/O for you. You could write to several different low I/O volume filesystems simultaneously, for example.

The kernel normally manages I/O requests -it lets them pile up for a while, then it writes all of the pending requests to disk. There are ways to reconfigure the kernel for real time I/O considerations, but it alters the system for everybody else. You can't do this to a multiuser production box without causing problems.

You might want to get a PC with a really high throughput disk I/O subsystem, maybe one that supports realtime streaming, put Linux on it, and go from there. You could also consider doing something to reduce I/O like compressing the data first. If compression gets you a lot.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

shell script takes long time to complete

Hi all, I wrote this shell script to validate filed numbers for input file. But it take forever to complete validation on a file. The average speed is like 9mins/MB. Can anyone tell me how to improve the performance of a shell script? Thanks (12 Replies)
Discussion started by: ozzman
12 Replies

2. Linux

it takes long time to login on server

Hi, I am trying to login using ssh on Red Hat Linux 5 server, The password appears immediately but after I enter the password it takes about 90 seconds to login completely. Please suggest what changes require? Regards, Manoj (4 Replies)
Discussion started by: manoj.solaris
4 Replies

3. UNIX for Dummies Questions & Answers

time how long it takes to load a module

Hello, like the title says, how can i measure the time it takes to load a module in Linux, and how how can i measure the time it takes to load a statically compiled module. /Best Regards Olle ---------- Post updated at 01:13 PM ---------- Previous update was at 11:54 AM ---------- For... (0 Replies)
Discussion started by: ollebanan
0 Replies

4. UNIX for Dummies Questions & Answers

Changing Password process takes a long time

We are running unix. After a reboot of the server we have found that changing password takes a long time. if type in passwd "username" you can type in the 1st instance of the password , press enter , then it will wait for about 3 minutes before bringing up the confirm password line typing it in... (4 Replies)
Discussion started by: AIXlewis
4 Replies

5. Shell Programming and Scripting

sort takes a long time

Dear experts I have a 200MG text file in this format: text \tab number I try to sort using options -fd and it takes very long! is that normal or I can speed it up in some ways? I dont want to split the file since this one is already splitted. I use this command: sort -fd file >... (12 Replies)
Discussion started by: voolek
12 Replies

6. Programming

What happens fwrite/fread at the same time?

Hello, I have a question about what happens when I copy the file which is being written by another process on Solaris 9/SPARC, UFS file system. in particular, I want to know what happens while some process is reading the file using fread or mmap, another process try to write something on the... (4 Replies)
Discussion started by: wipe3out
4 Replies

7. Solaris

Re-sync Taking Extremely Long.

It's almost 3 days now and my resync/re-attach is only at 80%. Is there something I can check in Solaris 10 that would be causing the degradation. It's only a standby machine. My live system completed in 6hrs. (9 Replies)
Discussion started by: ravzter
9 Replies

8. UNIX and Linux Applications

database takes long time to process

Hi, we currently having a issue where when we send jobs to the server for the application lawson, it is taking a very long time to complete. here are the last few lines of the database log. 2012-09-18-10.35.55.707279-240 E244403536A576 LEVEL: Warning PID : 950492 ... (1 Reply)
Discussion started by: techy1
1 Replies

9. Shell Programming and Scripting

Wget takes a long time to complete

Hi, I wish to check the return value for wget $url. However, some urls are designed to take 45 minutes or more to return. All i need to check if the URL can be reached or not using wget. How can i get wget to return the value in a few seconds ? (8 Replies)
Discussion started by: mohtashims
8 Replies

10. Shell Programming and Scripting

Transfer file from a server takes long time

It takes 6 hrs for a 90 GB zip file that i am copying / transferring from serverA onto serverB. scp user1@serverA:/opt/setup/cash.zip . Output: cash.zip 21% 19GB 4.7MB/s 4:11:46 ETA uname -a SunOS serverB 5.11 11.2 sun4v sparc sun4vCan you please suggest if i could do... (11 Replies)
Discussion started by: mohtashims
11 Replies
cpipe(1)							Programmer's Manual							  cpipe(1)

NAME
cpipe - copy stdin to stdout while counting bytes and reporting progress SYNOPSIS
cpipe [-b bsize] [-vt] [-vr] [-vw] [-ngr] [-s speed] OPTIONS
-b buffer size in kB, 1 Int value between 1 and oo. Default: `128' -vt show throughput. -vr show read-times. -vw show write-times. -ngr non-greedy read. Don't enforce a full buffer on read before starting to write. -s throughput speed limit in kB/s, 1 Double value between 1 and oo. DESCRIPTION
Cpipe copies its standard input to its standard output while measuring the time it takes to read an input buffer and write an output buf- fer. If one or more of the -vx options is given, statistics of average throughput and the total amount of bytes copied are printed to the standard error output. Non Greedy Read Normally, cpipe does its best to totally fill its buffer (option -b) before it starts writing. In some situations however, e.g. if you talk to an interactive program via cpipe, this deadlocks the communication: said program waits for input which it will never see, because the input is stuck in cpipe's buffer. But cpipe itself will not see more input before the program does not respond. To get around this, try using -ngr. When issuing a read call, cpipe is then satisfied as soon as it gets at least one byte. Instead of filling the buffer, it stops reading and writes whatever it got to the output. Note, however, that the throughput measurements will be less exact if the number of bytes transferred in one read/write pair becomes small, because cpipe will spent relatively more time working on every byte. Limiting Throughput If a throughput limit is specified with option -s, cpipe calls usleep(3) in between copying buffers, thereby artificially extending the duration of a read/write-cycle. Since on most systems there is a certain minimum time usleep() sleeps, e.g. 0.01s, it is impossible to reach high limits with a small buffer size. In this case increasing the buffer size (option -b) might help. However, keep in mind that this limits the throughput only on the average. Every single buffer is copied as fast as possible. EXAMPLE
The command tar cCf / - usr | cpipe -vr -vw -vt > /dev/null results in an output like ... in: 19.541ms at 6.4MB/s ( 4.7MB/s avg) 2.0MB out: 0.004ms at 30.5GB/s ( 27.1GB/s avg) 2.0MB thru: 19.865ms at 6.3MB/s ( 4.6MB/s avg) 2.0MB ... The first column shows the times it takes to handle one buffer of data (128kB by default). The read-call took 19.541ms, the write-call to /dev/null took just 0.004ms and from the start of the read to the end of write, it took 19.865ms. The second column shows the result of dividing the buffer size (128kB by default) by the times in the first column. The third column contains the average over all measured values from the start of the program. Finally, the last column shows the total number of bytes transferred, which is of course the same for reading and writing. BUGS
This program uses precious processor cycles. Consequently the measured times will be different from the transfer rates possible without it. Instead of just non-greedy reading, full non-blocking I/O and use of select(2) should be used to make sure that no deadlocks occur when communicating with interactive programs. CREDITS
Peter Astrand <astrand@lysator.liu.se> recommended the speed limit. Ivo De Decker <ivo@zeus.rug.ac.be> asked for deadlock prevention, which is (hopefully) sufficiently covered by the non-greedy read. AUTHOR
Bug reports, beer and postcards go to pifpafpuf@gmx.de. New versions will show up on http://cpipe.berlios.de/. Clig-manuals 3.0.0 cpipe(1)
All times are GMT -4. The time now is 02:45 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy