Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Read/Write a fairly large amount of data to a file as fast as possible
Top Forums Programming Read/Write a fairly large amount of data to a file as fast as possible Post 302310076 by jim mcnamara on Thursday 23rd of April 2009 04:30:03 PM
Old 04-23-2009
First off, have you already proven that conventional I/O (read/write or stdio) is simply not adequate for your files? Buffering is your friend.

You might want to read Steven's 'Advanced Programming in the Unix Environment' -
the chapter (Chap 8, I think) with the table on the effect of buffering on I/O....

Rochkind's 'Advanced Unix Programming' has some examples of high-performance read/write routines using conventional syscalls, including mmap().

You should consider that pitching programmatically simpler methodology for more complex methodology is never always a given. What you gain in speed may not be worth the extra programming time and maintenance time. Is say, 100 extra hours of your time worth a 10% gain in performance? Your manager might say 'No'.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Read Write byte range/chunk of data from specific location in file

I am new to Unix so will really appreciate if someone can guide me on this. What I want to do is: Step1: Read binary file - pick first 2 bytes, convert from hex to decimal. Read the next 3 bytes as well. 2 bytes will specify the number of bytes 'n' that I want to read and write... (1 Reply)
Discussion started by: Kbenipel
1 Replies

2. AIX

amount of memory allocated to large page

We just set up a system to use large pages. I want to know if there is a command to see how much of the memory is being used for large pages. For example if we have a system with 8GB of RAm assigned and it has been set to use 4GB for large pages is there a command to show that 4GB of the *GB is... (1 Reply)
Discussion started by: daveisme
1 Replies

3. Shell Programming and Scripting

Read the apecific data from one file and write into another file

Hi, I would like to read the specific data from file and write the data in the new file. My data input is something like this.. <EXROP:R=TJ0311T; ROUTE DATA R ROUTE PARAMETERS TJ0311T DETY=UPDR TTRANS=1 FNC=3 MA=628160955000 R=TJ0311D ... (3 Replies)
Discussion started by: bha148
3 Replies

4. Shell Programming and Scripting

How to tar large amount of files?

Hello I have the following files VOICE_hhhh SUBSCR_llll DEL_kkkk Consider that there are 1000 VOICE files+1000 SUBSCR files+1000DEL files When i try to tar these files using tar -cvf backup.tar VOICE* SUBSCR* DEL* i get the error: ksh: /usr/bin/tar: arg list too long How can i... (9 Replies)
Discussion started by: chriss_58
9 Replies

5. Solaris

Read/Write Data on CD/RW Disk

Would simply like to write data (no audio) to a CD/RW disk. The disk drive states CD/RW on the front but don't know for sure if the software is configured to recognize it as a writable disk. I can read/move data from the disk to the hard drive with no issue from the disk. Any help in this... (4 Replies)
Discussion started by: jes1trish
4 Replies

6. Shell Programming and Scripting

Read user input, Encrypt the data and write to file

Hi, can some one help me how to encrypt and decrypt a file. AIM: reade user input, encrypt it and save it to file. while decryption read the encrypted file decrypt it and save the output in some variable. Example: consider we have Credentials.txt file with content username: password... (5 Replies)
Discussion started by: saichand1985
5 Replies

7. Shell Programming and Scripting

Perl : Large amount of data put into an array

This basic code works. I have a very long list, almost 10000 lines that I am building into the array. Each line has either 2 or 3 fields as shown in the code snippit. The array elements are static (for a few reasons that out of scope of this question) the list has to be "built in". It... (5 Replies)
Discussion started by: sumguy
5 Replies

8. Shell Programming and Scripting

Need a perl script to read and write the data

Hi, I have on Designdocument in that information is stored with in tabular format.I need Perlscript to read and write the datausing perl script? Regards, Ravi (0 Replies)
Discussion started by: toravi.pentaho
0 Replies

9. Shell Programming and Scripting

Need a UNIX/perl script to read and write the data

Hi, I have on Designdocument in that information is stored with in tabular format.I need Perl/unix script to read and write the data using perl script? Regards, Ravi (4 Replies)
Discussion started by: toravi.pentaho
4 Replies

10. Shell Programming and Scripting

How to make awk command faster for large amount of data?

I have nginx web server logs with all requests that were made and I'm filtering them by date and time. Each line has the following structure: 127.0.0.1 - xyz.com GET 123.ts HTTP/1.1 (200) 0.000 s 3182 CoreMedia/1.0.0.15F79 (iPhone; U; CPU OS 11_4 like Mac OS X; pt_br) These text files are... (21 Replies)
Discussion started by: brenoasrm
21 Replies

LEARN ABOUT SUNOS

rwall

rwall(1M)						  System Administration Commands						 rwall(1M)

NAME

       rwall - write to all users over a network

SYNOPSIS

       /usr/sbin/rwall hostname...

       /usr/sbin/rwall -n netgroup...

       /usr/sbin/rwall -h hostname -n netgroup

DESCRIPTION

       rwall reads a message from standard input until EOF. It then sends this message, preceded by the line:

		Broadcast Message ...

       to all users logged in on the specified host machines. With the -n option, it sends to the specified network groups.

OPTIONS

       -n netgroup     Send the broadcast message to the specified network groups.

       -h hostname     Specify the hostname, the name of the host machine.

ATTRIBUTES

       See attributes(5) for descriptions of the following attributes:

       +-----------------------------+-----------------------------+
       |      ATTRIBUTE TYPE	     |	    ATTRIBUTE VALUE	   |
       +-----------------------------+-----------------------------+
       |Availability		     |SUNWrcmdc 		   |
       +-----------------------------+-----------------------------+

SEE ALSO

       inetd(1M), listen(1M), pmadm(1M), sacadm(1M), wall(1M), attributes(5)

NOTES

       The  timeout is fairly short to allow transmission to a large group of machines (some of which may be down) in a reasonable amount of time.
       Thus the message may not get through to a heavily loaded machine.

SunOS 5.10							    6 Nov 2000								 rwall(1M)
All times are GMT -4. The time now is 04:21 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy