Sponsored Content
Top Forums Programming Read/Write a fairly large amount of data to a file as fast as possible Post 302310411 by Corona688 on Friday 24th of April 2009 04:29:19 PM
Old 04-24-2009
What kind of socket do you have that your hard drive cannot keep up with? Normal read/write calls are not slow. Seeking is slow, if you're going to be seeking randomly all over the place then mmap-ing it might be better. But keep in mind that, on 32-bit machines at least, you're limited in how big an area you can map, a gig is a big chunk of a process' 4-gig address space. 64-bit's limit is much, much higher.

What you might also find useful is cache-hinting, being able to tell the kernel 'OK, I am done with this area of the file for the foreseeable future' in order to let it purge data from cache earlier than it might otherwise have, or do read-ahead differently, etc. It gives you some of the advantages of raw I/O without the problems. See fadvise and madvise.

Last edited by Corona688; 04-24-2009 at 06:11 PM..
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Read Write byte range/chunk of data from specific location in file

I am new to Unix so will really appreciate if someone can guide me on this. What I want to do is: Step1: Read binary file - pick first 2 bytes, convert from hex to decimal. Read the next 3 bytes as well. 2 bytes will specify the number of bytes 'n' that I want to read and write... (1 Reply)
Discussion started by: Kbenipel
1 Replies

2. AIX

amount of memory allocated to large page

We just set up a system to use large pages. I want to know if there is a command to see how much of the memory is being used for large pages. For example if we have a system with 8GB of RAm assigned and it has been set to use 4GB for large pages is there a command to show that 4GB of the *GB is... (1 Reply)
Discussion started by: daveisme
1 Replies

3. Shell Programming and Scripting

Read the apecific data from one file and write into another file

Hi, I would like to read the specific data from file and write the data in the new file. My data input is something like this.. <EXROP:R=TJ0311T; ROUTE DATA R ROUTE PARAMETERS TJ0311T DETY=UPDR TTRANS=1 FNC=3 MA=628160955000 R=TJ0311D ... (3 Replies)
Discussion started by: bha148
3 Replies

4. Shell Programming and Scripting

How to tar large amount of files?

Hello I have the following files VOICE_hhhh SUBSCR_llll DEL_kkkk Consider that there are 1000 VOICE files+1000 SUBSCR files+1000DEL files When i try to tar these files using tar -cvf backup.tar VOICE* SUBSCR* DEL* i get the error: ksh: /usr/bin/tar: arg list too long How can i... (9 Replies)
Discussion started by: chriss_58
9 Replies

5. Solaris

Read/Write Data on CD/RW Disk

Would simply like to write data (no audio) to a CD/RW disk. The disk drive states CD/RW on the front but don't know for sure if the software is configured to recognize it as a writable disk. I can read/move data from the disk to the hard drive with no issue from the disk. Any help in this... (4 Replies)
Discussion started by: jes1trish
4 Replies

6. Shell Programming and Scripting

Read user input, Encrypt the data and write to file

Hi, can some one help me how to encrypt and decrypt a file. AIM: reade user input, encrypt it and save it to file. while decryption read the encrypted file decrypt it and save the output in some variable. Example: consider we have Credentials.txt file with content username: password... (5 Replies)
Discussion started by: saichand1985
5 Replies

7. Shell Programming and Scripting

Perl : Large amount of data put into an array

This basic code works. I have a very long list, almost 10000 lines that I am building into the array. Each line has either 2 or 3 fields as shown in the code snippit. The array elements are static (for a few reasons that out of scope of this question) the list has to be "built in". It... (5 Replies)
Discussion started by: sumguy
5 Replies

8. Shell Programming and Scripting

Need a perl script to read and write the data

Hi, I have on Designdocument in that information is stored with in tabular format.I need Perlscript to read and write the datausing perl script? Regards, Ravi (0 Replies)
Discussion started by: toravi.pentaho
0 Replies

9. Shell Programming and Scripting

Need a UNIX/perl script to read and write the data

Hi, I have on Designdocument in that information is stored with in tabular format.I need Perl/unix script to read and write the data using perl script? Regards, Ravi (4 Replies)
Discussion started by: toravi.pentaho
4 Replies

10. Shell Programming and Scripting

How to make awk command faster for large amount of data?

I have nginx web server logs with all requests that were made and I'm filtering them by date and time. Each line has the following structure: 127.0.0.1 - xyz.com GET 123.ts HTTP/1.1 (200) 0.000 s 3182 CoreMedia/1.0.0.15F79 (iPhone; U; CPU OS 11_4 like Mac OS X; pt_br) These text files are... (21 Replies)
Discussion started by: brenoasrm
21 Replies
POSIX_MADVISE(3)					     Linux Programmer's Manual						  POSIX_MADVISE(3)

NAME
posix_madvise - give advice about patterns of memory usage SYNOPSIS
#include <sys/mman.h> int posix_madvise(void *addr, size_t len, int advice); Feature Test Macro Requirements for glibc (see feature_test_macros(7)): posix_madvise(): _POSIX_C_SOURCE >= 200112L DESCRIPTION
The posix_madvise() function allows an application to advise the system about its expected patterns of usage of memory in the address range starting at addr and continuing for len bytes. The system is free to use this advice in order to improve the performance of memory accesses (or to ignore the advice altogether), but calling posix_madvise() shall not affect the semantics of access to memory in the speci- fied range. The advice argument is one of the following: POSIX_MADV_NORMAL The application has no special advice regarding its memory usage patterns for the specified address range. This is the default behavior. POSIX_MADV_SEQUENTIAL The application expects to access the specified address range sequentially, running from lower addresses to higher addresses. Hence, pages in this region can be aggressively read ahead, and may be freed soon after they are accessed. POSIX_MADV_RANDOM The application expects to access the specified address range randomly. Thus, read ahead may be less useful than normally. POSIX_MADV_WILLNEED The application expects to access the specified address range in the near future. Thus, read ahead may be beneficial. POSIX_MADV_DONTNEED The application expects that it will not access the specified address range in the near future. RETURN VALUE
On success, posix_madvise() returns 0. On failure, it returns a positive error number. ERRORS
EINVAL addr is not a multiple of the system page size or len is negative. EINVAL advice is invalid. ENOMEM Addresses in the specified range are partially or completely outside the caller's address space. VERSIONS
Support for posix_madvise() first appeared in glibc version 2.2. CONFORMING TO
POSIX.1-2001. NOTES
POSIX.1 permits an implementation to generate an error if len is 0. On Linux, specifying len as 0 is permitted (as a successful no-op). In glibc, this function is implemented using madvise(2). However, since glibc 2.6, POSIX_MADV_DONTNEED is treated as a no-op, because the corresponding madvise(2) value, MADV_DONTNEED, has destructive semantics. SEE ALSO
madvise(2), posix_fadvise(2) COLOPHON
This page is part of release 4.15 of the Linux man-pages project. A description of the project, information about reporting bugs, and the latest version of this page, can be found at https://www.kernel.org/doc/man-pages/. Linux 2017-09-15 POSIX_MADVISE(3)
All times are GMT -4. The time now is 09:31 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy