04-23-2009
Read/Write a fairly large amount of data to a file as fast as possible
Hi,
I'm trying to figure out the best solution to the following problem, and I'm not
yet that much experienced like you. :-)
Basically I have to read a fairly large file, composed of "messages" , in order
to display all of them through an user interface (made with QT).
The messages that I write into the file, comes all at once from a socket, so
in oder to write it quickly without loosing any of them I plan to do the following:
- Create a list of preallocated pages (3-4 by default but the list grows if needed)
- Write the data that comes from the socket to the preallocated buffer
- Once a page is full, schedule a write with aio_write (AIO - Asyncronous IO).
- On callback schedule another one if any is full.
And so on..
This is the best I could come up with in the writing part, but if any of you have
a better idea, please let me know.
Now the problem comes when I have to read the file, at a later time, and display all
the messages in order to analyze them as fast as possible.
I first thought of mmap'ing the file in order to copy the data only once, from the file
to the kernel cache (if I understood correctly how mmap works internally) and then
accessing it from the application. But I'm not sure this can be done, or convenient, as
the file might be pretty big (2,3 Giga bytes, although I'm not sure about the magnitude).
Beside the kernel could unload the pages and many page faults could occurred.
So I discarded this idea.
I also thought about the opposite of what I do for writing but I'm not sure is a good idea.
The main problem is that I have to decode the messages before displaying them, as they are of different type and variable length. So reading the whole file at once and then decoding them to copy to another memory location seem time consuming to me. As it
requires 4 copies (disk -> kernel -> user space -> user space after decoding).
Anyway, now it's your turn. :-)
Any help would be appreciated.
Thanks.
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
I am new to Unix so will really appreciate if someone can guide me on this.
What I want to do is:
Step1: Read binary file - pick first 2 bytes, convert from hex to decimal. Read the next 3 bytes as well.
2 bytes will specify the number of bytes 'n' that I want to read and write... (1 Reply)
Discussion started by: Kbenipel
1 Replies
2. AIX
We just set up a system to use large pages. I want to know if there is a command to see how much of the memory is being used for large pages. For example if we have a system with 8GB of RAm assigned and it has been set to use 4GB for large pages is there a command to show that 4GB of the *GB is... (1 Reply)
Discussion started by: daveisme
1 Replies
3. Shell Programming and Scripting
Hi, I would like to read the specific data from file and write the data in the new file.
My data input is something like this..
<EXROP:R=TJ0311T;
ROUTE DATA
R ROUTE PARAMETERS
TJ0311T DETY=UPDR TTRANS=1 FNC=3
MA=628160955000
R=TJ0311D
... (3 Replies)
Discussion started by: bha148
3 Replies
4. Shell Programming and Scripting
Hello
I have the following files
VOICE_hhhh
SUBSCR_llll
DEL_kkkk
Consider that there are 1000 VOICE files+1000 SUBSCR files+1000DEL files
When i try to tar these files using
tar -cvf backup.tar VOICE* SUBSCR* DEL*
i get the error:
ksh: /usr/bin/tar: arg list too long
How can i... (9 Replies)
Discussion started by: chriss_58
9 Replies
5. Solaris
Would simply like to write data (no audio) to a CD/RW disk. The disk drive states CD/RW on the front but don't know for sure if the software is configured to recognize it as a writable disk. I can read/move data from the disk to the hard drive with no issue from the disk. Any help in this... (4 Replies)
Discussion started by: jes1trish
4 Replies
6. Shell Programming and Scripting
Hi,
can some one help me how to encrypt and decrypt a file.
AIM: reade user input, encrypt it and save it to file.
while decryption read the encrypted file decrypt it and save the output in some variable.
Example: consider we have Credentials.txt file with content username: password... (5 Replies)
Discussion started by: saichand1985
5 Replies
7. Shell Programming and Scripting
This basic code works.
I have a very long list, almost 10000 lines that I am building into the array. Each line has either 2 or 3 fields as shown in the code snippit. The array elements are static (for a few reasons that out of scope of this question) the list has to be "built in".
It... (5 Replies)
Discussion started by: sumguy
5 Replies
8. Shell Programming and Scripting
Hi,
I have on Designdocument in that information is stored with in tabular format.I need Perlscript to read and write the datausing perl script?
Regards,
Ravi (0 Replies)
Discussion started by: toravi.pentaho
0 Replies
9. Shell Programming and Scripting
Hi,
I have on Designdocument in that information is stored with in tabular format.I need Perl/unix script to read and write the data
using perl script?
Regards,
Ravi (4 Replies)
Discussion started by: toravi.pentaho
4 Replies
10. Shell Programming and Scripting
I have nginx web server logs with all requests that were made and I'm filtering them by date and time.
Each line has the following structure:
127.0.0.1 - xyz.com GET 123.ts HTTP/1.1 (200) 0.000 s 3182 CoreMedia/1.0.0.15F79 (iPhone; U; CPU OS 11_4 like Mac OS X; pt_br)
These text files are... (21 Replies)
Discussion started by: brenoasrm
21 Replies
LEARN ABOUT PHP
sendfile64
SENDFILE(2) Linux Programmer's Manual SENDFILE(2)
NAME
sendfile - transfer data between file descriptors
SYNOPSIS
#include <sys/sendfile.h>
ssize_t sendfile(int out_fd, int in_fd, off_t *offset, size_t count);
DESCRIPTION
sendfile() copies data between one file descriptor and another. Because this copying is done within the kernel, sendfile() is more effi-
cient than the combination of read(2) and write(2), which would require transferring data to and from user space.
in_fd should be a file descriptor opened for reading and out_fd should be a descriptor opened for writing.
If offset is not NULL, then it points to a variable holding the file offset from which sendfile() will start reading data from in_fd. When
sendfile() returns, this variable will be set to the offset of the byte following the last byte that was read. If offset is not NULL, then
sendfile() does not modify the file offset of in_fd; otherwise the file offset is adjusted to reflect the number of bytes read from in_fd.
If offset is NULL, then data will be read from in_fd starting at the file offset, and the file offset will be updated by the call.
count is the number of bytes to copy between the file descriptors.
The in_fd argument must correspond to a file which supports mmap(2)-like operations (i.e., it cannot be a socket).
In Linux kernels before 2.6.33, out_fd must refer to a socket. Since Linux 2.6.33 it can be any file. If it is a regular file, then send-
file() changes the file offset appropriately.
RETURN VALUE
If the transfer was successful, the number of bytes written to out_fd is returned. Note that a successful call to sendfile() may write
fewer bytes than requested; the caller should be prepared to retry the call if there were unsent bytes. See also NOTES.
On error, -1 is returned, and errno is set appropriately.
ERRORS
EAGAIN Nonblocking I/O has been selected using O_NONBLOCK and the write would block.
EBADF The input file was not opened for reading or the output file was not opened for writing.
EFAULT Bad address.
EINVAL Descriptor is not valid or locked, or an mmap(2)-like operation is not available for in_fd, or count is negative.
EINVAL out_fd has the O_APPEND flag set. This is not currently supported by sendfile().
EIO Unspecified error while reading from in_fd.
ENOMEM Insufficient memory to read from in_fd.
EOVERFLOW
count is too large, the operation would result in exceeding the maximum size of either the input file or the output file.
ESPIPE offset is not NULL but the input file is not seek(2)-able.
VERSIONS
sendfile() first appeared in Linux 2.2. The include file <sys/sendfile.h> is present since glibc 2.1.
CONFORMING TO
Not specified in POSIX.1-2001, nor in other standards.
Other UNIX systems implement sendfile() with different semantics and prototypes. It should not be used in portable programs.
NOTES
sendfile() will transfer at most 0x7ffff000 (2,147,479,552) bytes, returning the number of bytes actually transferred. (This is true on
both 32-bit and 64-bit systems.)
If you plan to use sendfile() for sending files to a TCP socket, but need to send some header data in front of the file contents, you will
find it useful to employ the TCP_CORK option, described in tcp(7), to minimize the number of packets and to tune performance.
In Linux 2.4 and earlier, out_fd could also refer to a regular file; this possibility went away in the Linux 2.6.x kernel series, but was
restored in Linux 2.6.33.
The original Linux sendfile() system call was not designed to handle large file offsets. Consequently, Linux 2.4 added sendfile64(), with
a wider type for the offset argument. The glibc sendfile() wrapper function transparently deals with the kernel differences.
Applications may wish to fall back to read(2)/write(2) in the case where sendfile() fails with EINVAL or ENOSYS.
If out_fd refers to a socket or pipe with zero-copy support, callers must ensure the transferred portions of the file referred to by in_fd
remain unmodified until the reader on the other end of out_fd has consumed the transferred data.
The Linux-specific splice(2) call supports transferring data between arbitrary file descriptors provided one (or both) of them is a pipe.
SEE ALSO
copy_file_range(2), mmap(2), open(2), socket(2), splice(2)
COLOPHON
This page is part of release 4.15 of the Linux man-pages project. A description of the project, information about reporting bugs, and the
latest version of this page, can be found at https://www.kernel.org/doc/man-pages/.
Linux 2017-09-15 SENDFILE(2)