Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Read/Write a fairly large amount of data to a file as fast as possible
Top Forums Programming Read/Write a fairly large amount of data to a file as fast as possible Post 302310076 by jim mcnamara on Thursday 23rd of April 2009 04:30:03 PM
Old 04-23-2009
First off, have you already proven that conventional I/O (read/write or stdio) is simply not adequate for your files? Buffering is your friend.

You might want to read Steven's 'Advanced Programming in the Unix Environment' -
the chapter (Chap 8, I think) with the table on the effect of buffering on I/O....

Rochkind's 'Advanced Unix Programming' has some examples of high-performance read/write routines using conventional syscalls, including mmap().

You should consider that pitching programmatically simpler methodology for more complex methodology is never always a given. What you gain in speed may not be worth the extra programming time and maintenance time. Is say, 100 extra hours of your time worth a 10% gain in performance? Your manager might say 'No'.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Read Write byte range/chunk of data from specific location in file

I am new to Unix so will really appreciate if someone can guide me on this. What I want to do is: Step1: Read binary file - pick first 2 bytes, convert from hex to decimal. Read the next 3 bytes as well. 2 bytes will specify the number of bytes 'n' that I want to read and write... (1 Reply)
Discussion started by: Kbenipel
1 Replies

2. AIX

amount of memory allocated to large page

We just set up a system to use large pages. I want to know if there is a command to see how much of the memory is being used for large pages. For example if we have a system with 8GB of RAm assigned and it has been set to use 4GB for large pages is there a command to show that 4GB of the *GB is... (1 Reply)
Discussion started by: daveisme
1 Replies

3. Shell Programming and Scripting

Read the apecific data from one file and write into another file

Hi, I would like to read the specific data from file and write the data in the new file. My data input is something like this.. <EXROP:R=TJ0311T; ROUTE DATA R ROUTE PARAMETERS TJ0311T DETY=UPDR TTRANS=1 FNC=3 MA=628160955000 R=TJ0311D ... (3 Replies)
Discussion started by: bha148
3 Replies

4. Shell Programming and Scripting

How to tar large amount of files?

Hello I have the following files VOICE_hhhh SUBSCR_llll DEL_kkkk Consider that there are 1000 VOICE files+1000 SUBSCR files+1000DEL files When i try to tar these files using tar -cvf backup.tar VOICE* SUBSCR* DEL* i get the error: ksh: /usr/bin/tar: arg list too long How can i... (9 Replies)
Discussion started by: chriss_58
9 Replies

5. Solaris

Read/Write Data on CD/RW Disk

Would simply like to write data (no audio) to a CD/RW disk. The disk drive states CD/RW on the front but don't know for sure if the software is configured to recognize it as a writable disk. I can read/move data from the disk to the hard drive with no issue from the disk. Any help in this... (4 Replies)
Discussion started by: jes1trish
4 Replies

6. Shell Programming and Scripting

Read user input, Encrypt the data and write to file

Hi, can some one help me how to encrypt and decrypt a file. AIM: reade user input, encrypt it and save it to file. while decryption read the encrypted file decrypt it and save the output in some variable. Example: consider we have Credentials.txt file with content username: password... (5 Replies)
Discussion started by: saichand1985
5 Replies

7. Shell Programming and Scripting

Perl : Large amount of data put into an array

This basic code works. I have a very long list, almost 10000 lines that I am building into the array. Each line has either 2 or 3 fields as shown in the code snippit. The array elements are static (for a few reasons that out of scope of this question) the list has to be "built in". It... (5 Replies)
Discussion started by: sumguy
5 Replies

8. Shell Programming and Scripting

Need a perl script to read and write the data

Hi, I have on Designdocument in that information is stored with in tabular format.I need Perlscript to read and write the datausing perl script? Regards, Ravi (0 Replies)
Discussion started by: toravi.pentaho
0 Replies

9. Shell Programming and Scripting

Need a UNIX/perl script to read and write the data

Hi, I have on Designdocument in that information is stored with in tabular format.I need Perl/unix script to read and write the data using perl script? Regards, Ravi (4 Replies)
Discussion started by: toravi.pentaho
4 Replies

10. Shell Programming and Scripting

How to make awk command faster for large amount of data?

I have nginx web server logs with all requests that were made and I'm filtering them by date and time. Each line has the following structure: 127.0.0.1 - xyz.com GET 123.ts HTTP/1.1 (200) 0.000 s 3182 CoreMedia/1.0.0.15F79 (iPhone; U; CPU OS 11_4 like Mac OS X; pt_br) These text files are... (21 Replies)
Discussion started by: brenoasrm
21 Replies

LEARN ABOUT DEBIAN

crmd

CRM 
DAEMON(7)						      Pacemaker Configuration						     CRM DAEMON(7)

NAME

       crmd - CRM Daemon Options

SYNOPSIS

       [dc-version=string] [cluster-infrastructure=string] [dc-deadtime=time] [cluster-recheck-interval=time] [election-timeout=time]
       [shutdown-escalation=time] [crmd-integration-timeout=time] [crmd-finalization-timeout=time] [crmd-transition-delay=time]
       [expected-quorum-votes=integer]

DESCRIPTION

       This is a fake resource that details the options that can be configured for the CRM Daemon.

SUPPORTED PARAMETERS

       dc-version = string [none]
	   Version of Pacemaker on the cluster's DC.

	   Includes the hash which identifies the exact Mercurial changeset it was built from. Used for diagnostic purposes.

       cluster-infrastructure = string [heartbeat]
	   The messaging stack on which Pacemaker is currently running.

	   Used for informational and diagnostic purposes.

       dc-deadtime = time [20s]
	   How long to wait for a response from other nodes during startup.

	   The "correct" value will depend on the speed/load of your network and the type of switches used.

       cluster-recheck-interval = time [15min]
	   Polling interval for time based changes to options, resource parameters and constraints.

	   The Cluster is primarily event driven, however the configuration can have elements that change based on time. To ensure these changes
	   take effect, we can optionally poll the cluster's status for changes. Allowed values: Zero disables polling. Positive values are an
	   interval in seconds (unless other SI units are specified. eg. 5min)

       election-timeout = time [2min]
	   *** Advanced Use Only ***.

	   If need to adjust this value, it probably indicates the presence of a bug.

       shutdown-escalation = time [20min]
	   *** Advanced Use Only ***.

	   If need to adjust this value, it probably indicates the presence of a bug.

       crmd-integration-timeout = time [3min]
	   *** Advanced Use Only ***.

	   If need to adjust this value, it probably indicates the presence of a bug.

       crmd-finalization-timeout = time [30min]
	   *** Advanced Use Only ***.

	   If you need to adjust this value, it probably indicates the presence of a bug.

       crmd-transition-delay = time [0s]
	   *** Advanced Use Only *** Enabling this option will slow down cluster recovery under all conditions

	   Delay cluster recovery for the configured interval to allow for additional/related events to occur. Useful if your configuration is
	   sensitive to the order in which ping updates arrive.

       expected-quorum-votes = integer [2]
	   The number of nodes expected to be in the cluster

	   Used to calculate quorum in openais based clusters.

AUTHOR

       Andrew Beekhof <andrew@beekhof.net>
	   Author.

Pacemaker Configuration 					    04/17/2012							     CRM DAEMON(7)
All times are GMT -4. The time now is 03:54 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy