Sponsored Content
Top Forums Programming Read/Write a fairly large amount of data to a file as fast as possible Post 302310076 by jim mcnamara on Thursday 23rd of April 2009 04:30:03 PM
Old 04-23-2009
First off, have you already proven that conventional I/O (read/write or stdio) is simply not adequate for your files? Buffering is your friend.

You might want to read Steven's 'Advanced Programming in the Unix Environment' -
the chapter (Chap 8, I think) with the table on the effect of buffering on I/O....

Rochkind's 'Advanced Unix Programming' has some examples of high-performance read/write routines using conventional syscalls, including mmap().

You should consider that pitching programmatically simpler methodology for more complex methodology is never always a given. What you gain in speed may not be worth the extra programming time and maintenance time. Is say, 100 extra hours of your time worth a 10% gain in performance? Your manager might say 'No'.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Read Write byte range/chunk of data from specific location in file

I am new to Unix so will really appreciate if someone can guide me on this. What I want to do is: Step1: Read binary file - pick first 2 bytes, convert from hex to decimal. Read the next 3 bytes as well. 2 bytes will specify the number of bytes 'n' that I want to read and write... (1 Reply)
Discussion started by: Kbenipel
1 Replies

2. AIX

amount of memory allocated to large page

We just set up a system to use large pages. I want to know if there is a command to see how much of the memory is being used for large pages. For example if we have a system with 8GB of RAm assigned and it has been set to use 4GB for large pages is there a command to show that 4GB of the *GB is... (1 Reply)
Discussion started by: daveisme
1 Replies

3. Shell Programming and Scripting

Read the apecific data from one file and write into another file

Hi, I would like to read the specific data from file and write the data in the new file. My data input is something like this.. <EXROP:R=TJ0311T; ROUTE DATA R ROUTE PARAMETERS TJ0311T DETY=UPDR TTRANS=1 FNC=3 MA=628160955000 R=TJ0311D ... (3 Replies)
Discussion started by: bha148
3 Replies

4. Shell Programming and Scripting

How to tar large amount of files?

Hello I have the following files VOICE_hhhh SUBSCR_llll DEL_kkkk Consider that there are 1000 VOICE files+1000 SUBSCR files+1000DEL files When i try to tar these files using tar -cvf backup.tar VOICE* SUBSCR* DEL* i get the error: ksh: /usr/bin/tar: arg list too long How can i... (9 Replies)
Discussion started by: chriss_58
9 Replies

5. Solaris

Read/Write Data on CD/RW Disk

Would simply like to write data (no audio) to a CD/RW disk. The disk drive states CD/RW on the front but don't know for sure if the software is configured to recognize it as a writable disk. I can read/move data from the disk to the hard drive with no issue from the disk. Any help in this... (4 Replies)
Discussion started by: jes1trish
4 Replies

6. Shell Programming and Scripting

Read user input, Encrypt the data and write to file

Hi, can some one help me how to encrypt and decrypt a file. AIM: reade user input, encrypt it and save it to file. while decryption read the encrypted file decrypt it and save the output in some variable. Example: consider we have Credentials.txt file with content username: password... (5 Replies)
Discussion started by: saichand1985
5 Replies

7. Shell Programming and Scripting

Perl : Large amount of data put into an array

This basic code works. I have a very long list, almost 10000 lines that I am building into the array. Each line has either 2 or 3 fields as shown in the code snippit. The array elements are static (for a few reasons that out of scope of this question) the list has to be "built in". It... (5 Replies)
Discussion started by: sumguy
5 Replies

8. Shell Programming and Scripting

Need a perl script to read and write the data

Hi, I have on Designdocument in that information is stored with in tabular format.I need Perlscript to read and write the datausing perl script? Regards, Ravi (0 Replies)
Discussion started by: toravi.pentaho
0 Replies

9. Shell Programming and Scripting

Need a UNIX/perl script to read and write the data

Hi, I have on Designdocument in that information is stored with in tabular format.I need Perl/unix script to read and write the data using perl script? Regards, Ravi (4 Replies)
Discussion started by: toravi.pentaho
4 Replies

10. Shell Programming and Scripting

How to make awk command faster for large amount of data?

I have nginx web server logs with all requests that were made and I'm filtering them by date and time. Each line has the following structure: 127.0.0.1 - xyz.com GET 123.ts HTTP/1.1 (200) 0.000 s 3182 CoreMedia/1.0.0.15F79 (iPhone; U; CPU OS 11_4 like Mac OS X; pt_br) These text files are... (21 Replies)
Discussion started by: brenoasrm
21 Replies
lock_write(9r)															    lock_write(9r)

NAME
lock_write - Asserts a complex lock with write access SYNOPSIS
#include <kern/lock.h> void lock_write( lock_t lock_structptr ); ARGUMENTS
Specifies a pointer to the complex lock structure, lock. This is the lock structure associated with the resource on which you want to assert a complex lock with write access. The lock structure is an opaque data structure; that is, its associated members are referenced and manipulated by the operating system and not by the user of the complex lock mechanism. DESCRIPTION
The lock_write routine asserts a lock with exclusive write access for the resource associated with the specified lock structure pointer. This means that once a write lock is asserted, no other kernel thread can gain read or write access to the resource until it is released. To release a complex write lock successfully asserted by lock_write, call the lock_done routine. NOTES
You must call lock_init (once only) prior to calling lock_write to initialize the lock structure pointer for the resource. A resource, from the kernel module's standpoint, is data that more than one kernel thread can manipulate. You can store the resource in global vari- ables and in data structure members. RETURN VALUES
None FILES
SEE ALSO
Routines: lock_done(9r), lock_read(9r), lock_terminate(9r), lock_try_read(9r), lock_try_write(9r) Data Structures: lock(9s) lock_write(9r)
All times are GMT -4. The time now is 08:27 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy