Sponsored Content
Top Forums Programming Read/Write a fairly large amount of data to a file as fast as possible Post 302309917 by emitrax on Thursday 23rd of April 2009 08:26:20 AM
Old 04-23-2009
Read/Write a fairly large amount of data to a file as fast as possible

Hi,

I'm trying to figure out the best solution to the following problem, and I'm not
yet that much experienced like you. :-)

Basically I have to read a fairly large file, composed of "messages" , in order
to display all of them through an user interface (made with QT).

The messages that I write into the file, comes all at once from a socket, so
in oder to write it quickly without loosing any of them I plan to do the following:

- Create a list of preallocated pages (3-4 by default but the list grows if needed)
- Write the data that comes from the socket to the preallocated buffer
- Once a page is full, schedule a write with aio_write (AIO - Asyncronous IO).
- On callback schedule another one if any is full.
And so on..

This is the best I could come up with in the writing part, but if any of you have
a better idea, please let me know.

Now the problem comes when I have to read the file, at a later time, and display all
the messages in order to analyze them as fast as possible.

I first thought of mmap'ing the file in order to copy the data only once, from the file
to the kernel cache (if I understood correctly how mmap works internally) and then
accessing it from the application. But I'm not sure this can be done, or convenient, as
the file might be pretty big (2,3 Giga bytes, although I'm not sure about the magnitude).
Beside the kernel could unload the pages and many page faults could occurred.
So I discarded this idea.

I also thought about the opposite of what I do for writing but I'm not sure is a good idea.

The main problem is that I have to decode the messages before displaying them, as they are of different type and variable length. So reading the whole file at once and then decoding them to copy to another memory location seem time consuming to me. As it
requires 4 copies (disk -> kernel -> user space -> user space after decoding).

Anyway, now it's your turn. :-)
Any help would be appreciated.

Thanks.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Read Write byte range/chunk of data from specific location in file

I am new to Unix so will really appreciate if someone can guide me on this. What I want to do is: Step1: Read binary file - pick first 2 bytes, convert from hex to decimal. Read the next 3 bytes as well. 2 bytes will specify the number of bytes 'n' that I want to read and write... (1 Reply)
Discussion started by: Kbenipel
1 Replies

2. AIX

amount of memory allocated to large page

We just set up a system to use large pages. I want to know if there is a command to see how much of the memory is being used for large pages. For example if we have a system with 8GB of RAm assigned and it has been set to use 4GB for large pages is there a command to show that 4GB of the *GB is... (1 Reply)
Discussion started by: daveisme
1 Replies

3. Shell Programming and Scripting

Read the apecific data from one file and write into another file

Hi, I would like to read the specific data from file and write the data in the new file. My data input is something like this.. <EXROP:R=TJ0311T; ROUTE DATA R ROUTE PARAMETERS TJ0311T DETY=UPDR TTRANS=1 FNC=3 MA=628160955000 R=TJ0311D ... (3 Replies)
Discussion started by: bha148
3 Replies

4. Shell Programming and Scripting

How to tar large amount of files?

Hello I have the following files VOICE_hhhh SUBSCR_llll DEL_kkkk Consider that there are 1000 VOICE files+1000 SUBSCR files+1000DEL files When i try to tar these files using tar -cvf backup.tar VOICE* SUBSCR* DEL* i get the error: ksh: /usr/bin/tar: arg list too long How can i... (9 Replies)
Discussion started by: chriss_58
9 Replies

5. Solaris

Read/Write Data on CD/RW Disk

Would simply like to write data (no audio) to a CD/RW disk. The disk drive states CD/RW on the front but don't know for sure if the software is configured to recognize it as a writable disk. I can read/move data from the disk to the hard drive with no issue from the disk. Any help in this... (4 Replies)
Discussion started by: jes1trish
4 Replies

6. Shell Programming and Scripting

Read user input, Encrypt the data and write to file

Hi, can some one help me how to encrypt and decrypt a file. AIM: reade user input, encrypt it and save it to file. while decryption read the encrypted file decrypt it and save the output in some variable. Example: consider we have Credentials.txt file with content username: password... (5 Replies)
Discussion started by: saichand1985
5 Replies

7. Shell Programming and Scripting

Perl : Large amount of data put into an array

This basic code works. I have a very long list, almost 10000 lines that I am building into the array. Each line has either 2 or 3 fields as shown in the code snippit. The array elements are static (for a few reasons that out of scope of this question) the list has to be "built in". It... (5 Replies)
Discussion started by: sumguy
5 Replies

8. Shell Programming and Scripting

Need a perl script to read and write the data

Hi, I have on Designdocument in that information is stored with in tabular format.I need Perlscript to read and write the datausing perl script? Regards, Ravi (0 Replies)
Discussion started by: toravi.pentaho
0 Replies

9. Shell Programming and Scripting

Need a UNIX/perl script to read and write the data

Hi, I have on Designdocument in that information is stored with in tabular format.I need Perl/unix script to read and write the data using perl script? Regards, Ravi (4 Replies)
Discussion started by: toravi.pentaho
4 Replies

10. Shell Programming and Scripting

How to make awk command faster for large amount of data?

I have nginx web server logs with all requests that were made and I'm filtering them by date and time. Each line has the following structure: 127.0.0.1 - xyz.com GET 123.ts HTTP/1.1 (200) 0.000 s 3182 CoreMedia/1.0.0.15F79 (iPhone; U; CPU OS 11_4 like Mac OS X; pt_br) These text files are... (21 Replies)
Discussion started by: brenoasrm
21 Replies
puts(n) 						       Tcl Built-In Commands							   puts(n)

__________________________________________________________________________________________________________________________________________________

NAME
puts - Write to a channel SYNOPSIS
puts ?-nonewline? ?channelId? string _________________________________________________________________ DESCRIPTION
Writes the characters given by string to the channel given by channelId. ChannelId must be an identifier for an open channel such as a Tcl standard channel (stdout or stderr), the return value from an invocation of open or socket, or the result of a channel creation command provided by a Tcl extension. The channel must have been opened for output. If no channelId is specified then it defaults to stdout. Puts normally outputs a newline character after string, but this feature may be suppressed by specifying the -nonewline switch. Newline characters in the output are translated by puts to platform-specific end-of-line sequences according to the current value of the -translation option for the channel (for example, on PCs newlines are normally replaced with carriage-return-linefeed sequences. See the fconfigure manual entry for a discussion on ways in which fconfigure will alter output. Tcl buffers output internally, so characters written with puts may not appear immediately on the output file or device; Tcl will normally delay output until the buffer is full or the channel is closed. You can force output to appear immediately with the flush command. When the output buffer fills up, the puts command will normally block until all the buffered data has been accepted for output by the oper- ating system. If channelId is in nonblocking mode then the puts command will not block even if the operating system cannot accept the data. Instead, Tcl continues to buffer the data and writes it in the background as fast as the underlying file or device can accept it. The application must use the Tcl event loop for nonblocking output to work; otherwise Tcl never finds out that the file or device is ready for more output data. It is possible for an arbitrarily large amount of data to be buffered for a channel in nonblocking mode, which could consume a large amount of memory. To avoid wasting memory, nonblocking I/O should normally be used in an event-driven fashion with the fileevent command (do not invoke puts unless you have recently been notified via a file event that the channel is ready for more output data). EXAMPLES
Write a short message to the console (or wherever stdout is directed): puts "Hello, World!" Print a message in several parts: puts -nonewline "Hello, " puts "World!" Print a message to the standard error channel: puts stderr "Hello, World!" Append a log message to a file: set chan [open my.log a] set timestamp [clock format [clock seconds]] puts $chan "$timestamp - Hello, World!" close $chan SEE ALSO
file(n), fileevent(n), Tcl_StandardChannels(3) KEYWORDS
channel, newline, output, write Tcl 7.5 puts(n)
All times are GMT -4. The time now is 12:59 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy