Sponsored Content
Top Forums Programming Read/Write a fairly large amount of data to a file as fast as possible Post 302310411 by Corona688 on Friday 24th of April 2009 04:29:19 PM
Old 04-24-2009
What kind of socket do you have that your hard drive cannot keep up with? Normal read/write calls are not slow. Seeking is slow, if you're going to be seeking randomly all over the place then mmap-ing it might be better. But keep in mind that, on 32-bit machines at least, you're limited in how big an area you can map, a gig is a big chunk of a process' 4-gig address space. 64-bit's limit is much, much higher.

What you might also find useful is cache-hinting, being able to tell the kernel 'OK, I am done with this area of the file for the foreseeable future' in order to let it purge data from cache earlier than it might otherwise have, or do read-ahead differently, etc. It gives you some of the advantages of raw I/O without the problems. See fadvise and madvise.

Last edited by Corona688; 04-24-2009 at 06:11 PM..
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Read Write byte range/chunk of data from specific location in file

I am new to Unix so will really appreciate if someone can guide me on this. What I want to do is: Step1: Read binary file - pick first 2 bytes, convert from hex to decimal. Read the next 3 bytes as well. 2 bytes will specify the number of bytes 'n' that I want to read and write... (1 Reply)
Discussion started by: Kbenipel
1 Replies

2. AIX

amount of memory allocated to large page

We just set up a system to use large pages. I want to know if there is a command to see how much of the memory is being used for large pages. For example if we have a system with 8GB of RAm assigned and it has been set to use 4GB for large pages is there a command to show that 4GB of the *GB is... (1 Reply)
Discussion started by: daveisme
1 Replies

3. Shell Programming and Scripting

Read the apecific data from one file and write into another file

Hi, I would like to read the specific data from file and write the data in the new file. My data input is something like this.. <EXROP:R=TJ0311T; ROUTE DATA R ROUTE PARAMETERS TJ0311T DETY=UPDR TTRANS=1 FNC=3 MA=628160955000 R=TJ0311D ... (3 Replies)
Discussion started by: bha148
3 Replies

4. Shell Programming and Scripting

How to tar large amount of files?

Hello I have the following files VOICE_hhhh SUBSCR_llll DEL_kkkk Consider that there are 1000 VOICE files+1000 SUBSCR files+1000DEL files When i try to tar these files using tar -cvf backup.tar VOICE* SUBSCR* DEL* i get the error: ksh: /usr/bin/tar: arg list too long How can i... (9 Replies)
Discussion started by: chriss_58
9 Replies

5. Solaris

Read/Write Data on CD/RW Disk

Would simply like to write data (no audio) to a CD/RW disk. The disk drive states CD/RW on the front but don't know for sure if the software is configured to recognize it as a writable disk. I can read/move data from the disk to the hard drive with no issue from the disk. Any help in this... (4 Replies)
Discussion started by: jes1trish
4 Replies

6. Shell Programming and Scripting

Read user input, Encrypt the data and write to file

Hi, can some one help me how to encrypt and decrypt a file. AIM: reade user input, encrypt it and save it to file. while decryption read the encrypted file decrypt it and save the output in some variable. Example: consider we have Credentials.txt file with content username: password... (5 Replies)
Discussion started by: saichand1985
5 Replies

7. Shell Programming and Scripting

Perl : Large amount of data put into an array

This basic code works. I have a very long list, almost 10000 lines that I am building into the array. Each line has either 2 or 3 fields as shown in the code snippit. The array elements are static (for a few reasons that out of scope of this question) the list has to be "built in". It... (5 Replies)
Discussion started by: sumguy
5 Replies

8. Shell Programming and Scripting

Need a perl script to read and write the data

Hi, I have on Designdocument in that information is stored with in tabular format.I need Perlscript to read and write the datausing perl script? Regards, Ravi (0 Replies)
Discussion started by: toravi.pentaho
0 Replies

9. Shell Programming and Scripting

Need a UNIX/perl script to read and write the data

Hi, I have on Designdocument in that information is stored with in tabular format.I need Perl/unix script to read and write the data using perl script? Regards, Ravi (4 Replies)
Discussion started by: toravi.pentaho
4 Replies

10. Shell Programming and Scripting

How to make awk command faster for large amount of data?

I have nginx web server logs with all requests that were made and I'm filtering them by date and time. Each line has the following structure: 127.0.0.1 - xyz.com GET 123.ts HTTP/1.1 (200) 0.000 s 3182 CoreMedia/1.0.0.15F79 (iPhone; U; CPU OS 11_4 like Mac OS X; pt_br) These text files are... (21 Replies)
Discussion started by: brenoasrm
21 Replies
MADVISE(2)						     Linux Programmer's Manual							MADVISE(2)

NAME
madvise - give advice about use of memory SYNOPSIS
#include <sys/mman.h> int madvise(void *addr, size_t length, int advice); Feature Test Macro Requirements for glibc (see feature_test_macros(7)): madvise(): _BSD_SOURCE DESCRIPTION
The madvise() system call advises the kernel about how to handle paging input/output in the address range beginning at address addr and with size length bytes. It allows an application to tell the kernel how it expects to use some mapped or shared memory areas, so that the kernel can choose appropriate read-ahead and caching techniques. This call does not influence the semantics of the application (except in the case of MADV_DONTNEED), but may influence its performance. The kernel is free to ignore the advice. The advice is indicated in the advice argument which can be MADV_NORMAL No special treatment. This is the default. MADV_RANDOM Expect page references in random order. (Hence, read ahead may be less useful than normally.) MADV_SEQUENTIAL Expect page references in sequential order. (Hence, pages in the given range can be aggressively read ahead, and may be freed soon after they are accessed.) MADV_WILLNEED Expect access in the near future. (Hence, it might be a good idea to read some pages ahead.) MADV_DONTNEED Do not expect access in the near future. (For the time being, the application is finished with the given range, so the kernel can free resources associated with it.) Subsequent accesses of pages in this range will succeed, but will result either in reloading of the memory contents from the underlying mapped file (see mmap(2)) or zero-fill-on-demand pages for mappings without an underlying file. MADV_REMOVE (Since Linux 2.6.16) Free up a given range of pages and its associated backing store. Currently, only shmfs/tmpfs supports this; other file systems return with the error ENOSYS. MADV_DONTFORK (Since Linux 2.6.16) Do not make the pages in this range available to the child after a fork(2). This is useful to prevent copy-on-write semantics from changing the physical location of a page(s) if the parent writes to it after a fork(2). (Such page relocations cause problems for hardware that DMAs into the page(s).) MADV_DOFORK (Since Linux 2.6.16) Undo the effect of MADV_DONTFORK, restoring the default behavior, whereby a mapping is inherited across fork(2). MADV_HWPOISON (Since Linux 2.6.32) Poison a page and handle it like a hardware memory corruption. This operation is only available for privileged (CAP_SYS_ADMIN) pro- cesses. This operation may result in the calling process receiving a SIGBUS and the page being unmapped. This feature is intended for testing of memory error-handling code; it is only available if the kernel was configured with CONFIG_MEMORY_FAILURE. MADV_SOFT_OFFLINE (Since Linux 2.6.33) Soft offline the pages in the range specified by addr and length. The memory of each page in the specified range is preserved (i.e., when next accessed, the same content will be visible, but in a new physical page frame), and the original page is offlined (i.e., no longer used, and taken out of normal memory management). The effect of the MADV_SOFT_OFFLINE operation is invisible to (i.e., does not change the semantics of) the calling process. This feature is intended for testing of memory error-handling code; it is only available if the kernel was configured with CONFIG_MEMORY_FAILURE. MADV_MERGEABLE (since Linux 2.6.32) Enable Kernel Samepage Merging (KSM) for the pages in the range specified by addr and length. The kernel regularly scans those areas of user memory that have been marked as mergeable, looking for pages with identical content. These are replaced by a single write-protected page (which is automatically copied if a process later wants to update the content of the page). KSM only merges private anonymous pages (see mmap(2)). The KSM feature is intended for applications that generate many instances of the same data (e.g., virtualization systems such as KVM). It can consume a lot of processing power; use with care. See the kernel source file Documentation/vm/ksm.txt for more details. The MADV_MERGEABLE and MADV_UNMERGEABLE operations are only available if the kernel was configured with CONFIG_KSM. MADV_UNMERGEABLE (since Linux 2.6.32) Undo the effect of an earlier MADV_MERGEABLE operation on the specified address range; KSM unmerges whatever pages it had merged in the address range specified by addr and length. RETURN VALUE
On success madvise() returns zero. On error, it returns -1 and errno is set appropriately. ERRORS
EAGAIN A kernel resource was temporarily unavailable. EBADF The map exists, but the area maps something that isn't a file. EINVAL This error can occur for the following reasons: * The value len is negative. * addr is not page-aligned. * advice is not a valid value * The application is attempting to release locked or shared pages (with MADV_DONTNEED). * MADV_MERGEABLE or MADV_UNMERGEABLE was specified in advice, but the kernel was not configured with CONFIG_KSM. EIO (for MADV_WILLNEED) Paging in this area would exceed the process's maximum resident set size. ENOMEM (for MADV_WILLNEED) Not enough memory: paging in failed. ENOMEM Addresses in the specified range are not currently mapped, or are outside the address space of the process. CONFORMING TO
POSIX.1b. POSIX.1-2001 describes posix_madvise(3) with constants POSIX_MADV_NORMAL, etc., with a behavior close to that described here. There is a similar posix_fadvise(2) for file access. MADV_REMOVE, MADV_DONTFORK, MADV_DOFORK, MADV_HWPOISON, MADV_MERGEABLE, and MADV_UNMERGEABLE are Linux-specific. NOTES
Linux Notes The current Linux implementation (2.4.0) views this system call more as a command than as advice and hence may return an error when it can- not do what it usually would do in response to this advice. (See the ERRORS description above.) This is nonstandard behavior. The Linux implementation requires that the address addr be page-aligned, and allows length to be zero. If there are some parts of the specified address range that are not mapped, the Linux version of madvise() ignores them and applies the call to the rest (but returns ENOMEM from the system call, as it should). SEE ALSO
getrlimit(2), mincore(2), mmap(2), mprotect(2), msync(2), munmap(2) COLOPHON
This page is part of release 3.27 of the Linux man-pages project. A description of the project, and information about reporting bugs, can be found at http://www.kernel.org/doc/man-pages/. Linux 2010-06-20 MADVISE(2)
All times are GMT -4. The time now is 06:05 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy