Visit Our UNIX and Linux User Community


Understanding read/write and kernel interaction


 
Thread Tools Search this Thread
Top Forums Programming Understanding read/write and kernel interaction
# 1  
Old 09-16-2009
Understanding read/write and kernel interaction

Ok, so I'm trying to finalize my understanding of read/write and kernel interaction.

read():

You have a library function that has as it's first parameter (what the open file to read from is), second parameter( a pointer to a buffer (is this the location of a buffer in the user area or the controller buffer in the kernel?) and a third parameter(# of bytes to read).

So, once the controller stores words from disk into its own buffer, the DMA then transfers the data to the main memory and the CPU gains control. My question is, what now? The purpose was to read a certain amount of words. So, are those words being returned to the caller once they are in the main memory or is an address returned of where in the main memory those words are located?


write():

First param(which file to write to), second param (again, unsure. Is this pointing to a buffer in the user area already filled in with the new words to modify the file with?), third parameter (how much you replace)

Main question entails how this even works as opposed to the read. Does the controller still need to store a certain amount of words from the disk into its own internal buffer?

I understood it like this: Controller's own hardware buffer somehow stores the words to modify the file. The DMA then uses the new block within the controller buffer to modify a file within the main memory.

Any help is appreciated

Last edited by Krothos; 09-17-2009 at 12:00 AM..
# 2  
Old 09-17-2009
Quote:
Originally Posted by Krothos
read():

You have a library function
and here a system call
Quote:
that has as it's first parameter (what the open file to read from is)
a file descriptor
Quote:
second parameter( a pointer to a buffer (is this the location of a buffer in the user area or the controller buffer in the kernel?)
no kernel buffer but userland (virtual memory)
Quote:
and a third parameter(# of bytes to read).

So, once the controller stores words from disk into its own buffer
read isn't necessarily retrieving data from a disk through a controller. What is done depends on what the file descriptor points to. Might be a file, a raw device, the network, a serial port, a virtual device, ...
Quote:
, the DMA then transfers the data to the main memory and the CPU gains control. My question is, what now? The purpose was to read a certain amount of words.
These aren't words but bytes.
Quote:
So, are those words being returned to the caller once they are in the main memory or is an address returned of where in the main memory those words are located?
If by main memory you mean kernel memory, no. A process has no access to the kernel memory.
If you are asking if data is copied twice, then that depends on the kernel implementation. Generally, it is true but some OSes allow direct I/O from the driver to the user supplied buffer.
See for example Solaris directio function: directio(3C) provide advice to file system (man pages section 3: Basic Library Functions) - Sun Microsystems
Quote:

write():

First param(which file to write to), second param (again, unsure. Is this pointing to a buffer in the user area already filled in with the new words to modify the file with?),
Yes (bytes)
Quote:
third parameter (how much you replace)

Main question entails how this even works as opposed to the read. Does the controller still need to store a certain amount of words from the disk into its own internal buffer?
Same as read.
Quote:
I understood it like this: Controller's own hardware buffer somehow stores the words to modify the file. The DMA then uses the new block within the controller buffer to modify a file within the main memory.
I don't get what you mean here.
# 3  
Old 09-17-2009
Quote:
Originally Posted by jlliagre
If by main memory you mean kernel memory, no. A process has no access to the kernel memory.
So once the bytes are transferred stored over in the main memory through the DMA, how is what was read returned to the caller? So if I wanted to read 3 bytes from a file containing "Hello", how is "Hel" returned to the library function? What sees this and how/where is it returned from?

Quote:
Yes (bytes)Same as read.I don't get what you mean here.
[/quote]

What I'm talking about what is needed for the write.

What I meant was this:

Your library function is passing a reference to a buffer storing the bytes to replace a certain block in a file (for example). So, the goal is just to replace those bytes in the file with what is in the buffer (the second argument passed in).

What I am confused about is HOW exactly the modification takes place. How those bytes within the user area buffer replace the target file block of bytes.

Are the bytes within the user area buffer somehow transferred over to the buffer which resides on the device controller for let's say, a disk (the file to modify is on the disk)? And then from there, what's in the buffer of the controller somehow replaces the targeted block within the file you are trying to modify?

So you're passing in bytes to modify some block within the target file. I don't get how this takes place for a write.

That was what I was saying. I don't get the process of how write takes place in terms of a file let's say.
# 4  
Old 09-17-2009
Quote:
Originally Posted by Krothos
So once the bytes are transferred stored over in the main memory through the DMA, how is what was read returned to the caller? So if I wanted to read 3 bytes from a file containing "Hello", how is "Hel" returned to the library function? What sees this and how/where is it returned from?
That's the whole purpose of a system call. Passing data from/to the kernel.
Quote:
That was what I was saying. I don't get the process of how write takes place in terms of a file let's say.
That's the picture but the process is quite more complex than your description.
There are several layers crossed by the data between your application and the disk blocks.
As you are talking about a file, the filesystem, file cache and possibly a log are playing a role. Writes are usually delayed so you will need to wait for a flush for the data to be committed on disk. Also, some form of software or hardware raid (mirroring/striping and the likes), checksums or compression might take place, the disk itself will certainly have a cache too.
# 5  
Old 09-17-2009
Quote:
Originally Posted by jlliagre
That's the whole purpose of a system call. Passing data from/to the kernel.
Yes but the DMA then puts what was read into the main memory which then becomes stored into the buffer which was originally passed in with the read. Got it.

Quote:
That's the picture but the process is quite more complex than your description.
There are several layers crossed by the data between your application and the disk blocks.
As you are talking about a file, the filesystem, file cache and possibly a log are playing a role. Writes are usually delayed so you will need to wait for a flush for the data to be committed on disk. Also, some form of software or hardware raid (mirroring/striping and the likes), checksums or compression might take place, the disk itself will certainly have a cache too.
I understand that but I'm trying to look at it from a somewhat high level. The way I see it now, for a write, you have a pointer to buffer of what to write. You then write those bytes to the kernel buffer (one some controller buffer) from what was in the user buffer. What is in the controller buffer overwrites some bytes in the file. I know, as you said, there are many mitigating items to think about but is there a basic way you can explain how what is in the controller buffer (the bytes taken from the user buffer) overwrites bytes in a target file?

Thanks

Btw, I've read a ton of material. For whatever reason, people don't like digging further into how these things work. The only decent material which started to explain the write() in detail was an OS book which is out of print and unfortunately google books doesn't have the pages for that section.
# 6  
Old 09-17-2009
A file is an abstract object. What is written is one ore more blocks/sectors. The middle layers are responsible to make that happen properly.
# 7  
Old 09-17-2009
FWIW - directio in Solaris enables direct access that restrictions.
It is restricted to I/O data sizes by disk geometry (sector size) for example.

A memory mapped (mmap() ) file is as close as you can get to direct I/O for a file on most UNIX systems using system calls like read/write.

NOTE: a succesful write call does NOT guarantee that the data will be physically written to a file completely or correctly. See: sync fdatasync or google for 'synchronized I/O data integrity completion'

The reason for this comment is that the OP seems to assume the opposite i.e.,
successful write == successful data synchrony

Previous Thread | Next Thread
Test Your Knowledge in Computers #3
Difficulty: Easy
'Hello World' is a program only for advanced programmers.
True or False?

10 More Discussions You Might Find Interesting

1. Ubuntu

Kernel panics : trying to write / read on tiny tty driver

I'm a beginner to the Linux programming and trying my hands on some device driver examples while practising. The below code (a trimmed down version of tiny_tty.c from ldd3 book) loads perfectly using insmod and I'm able to see it in /proc/tty/drivers , /proc/modules and device nodes are getting... (1 Reply)
Discussion started by: diwsdiwa
1 Replies

2. Programming

Understanding Linux Kernel

Hi, I have started reading about Linux kernel. The books also explains about the source code here and there. I found a file named head.s in the source which i am not able to understand, it looks like some assembly language. Can anyone confirm this, so that can start looking for assemble... (3 Replies)
Discussion started by: kumaran_5555
3 Replies

3. Programming

c++, raw sockets, stopping kernel write in header?

Hi, im trying to lern about raw sockets with my debian and c++. Tried to make a icmp and tcp packet and send it with sendto. Checked on wireshark and recognized that kernel changed my headers. So searched about stopping the kernel change the header and tried it with setsockopt, like said in at this... (4 Replies)
Discussion started by: sandcastle
4 Replies

4. Homework & Coursework Questions

i have no understanding of how to write an array or use one, please help!

Use and complete the template provided. The entire template must be completed. If you don't, your post may be deleted! 1. The problem statement, all variables and given/known data: make 2 files. one to Input 10 numbers and print out the biggest number, and one to Write a script that can check... (1 Reply)
Discussion started by: wendyshephard
1 Replies

5. Shell Programming and Scripting

i have no understanding of how to write an array or use one, please help!

im in a basic unix class and our professor speaks broken engliash so i can never understand what exactly we are doing in class and i have no prior experience with unix. we were given an assignment to make 2 files. one to Input 10 numbers and print out the biggest number, and one to Write a script... (1 Reply)
Discussion started by: wendyshephard
1 Replies

6. Solaris

Which file is read by kernel to set its default system kernel parameters values?

Hi gurus Could anybody tell me which file is read by kernel to set its default system kernal parameters values in solaris. Here I am not taking about /etc/system file which is used to load kernal modules or to change any default system kernal parameter value Is it /dev/kmem file or something... (1 Reply)
Discussion started by: girish.batra
1 Replies

7. IP Networking

read/write,write/write lock with smbclient fails

Hi, We have smb client running on two of the linux boxes and smb server on another linux system. During a backup operation which uses smb, read of a file was allowed while write to the same file was going on.Also simultaneous writes to the same file were allowed.Following are the settings in the... (1 Reply)
Discussion started by: swatidas11
1 Replies

8. UNIX for Dummies Questions & Answers

Understanding read/write and kernel interaction

Ok, so I'm trying to finalize my understanding of read/write and kernel interaction. read(): You have a library function that has as it's first parameter (what the open file to read from is), second parameter( a pointer to a buffer (is this the location of a buffer in the user area or the... (1 Reply)
Discussion started by: Krothos
1 Replies

9. Shell Programming and Scripting

Find all files with group read OR group write OR user write permission

I need to find all the files that have group Read or Write permission or files that have user write permission. This is what I have so far: find . -exec ls -l {} \; | awk '/-...rw..w./ {print $1 " " $3 " " $4 " " $9}' It shows me all files where group read = true, group write = true... (5 Replies)
Discussion started by: shunter63
5 Replies

10. UNIX for Dummies Questions & Answers

How to read freebsd kernel source?

I got the freebsd kernel source from the first install CD(in directory:\7.0-RELEASE\src\),isn't right? if so,how can i read it? Is there any tools? (4 Replies)
Discussion started by: zhouq3132
4 Replies

Featured Tech Videos