DD command using block device as input


 
Thread Tools Search this Thread
Special Forums Hardware DD command using block device as input
# 1  
Old 07-03-2012
DD command using block device as input

Hi,

I am trying to measure the speed of reading a given block size using the dd command. However depending on which input I use: a regular file (on the same device) or /dev/sdb1, I get some really different results.

Code:
$sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'; dd if=pirate of=/dev/null bs=512 count=1
1+0 records in
1+0 records out
512 bytes (512 B) copied, 0.0190755 s, 26.8 kB/s

$sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches'; sudo dd if=/dev/sdb1 of=/dev/null bs=512 count=1
1+0 records in
1+0 records out
512 bytes (512 B) copied, 1.9649e-05 s, 26.1 MB/s

Note:
- Freshly formated ext4
ATA device, with non-removable media
Transport: Serial, SATA Rev 3.0

- I did the same experiment with varying blocksizes, and up to 128K the phenomena is the same.

- The file pirate is 100G and filefrag reports: 57 extents found

Questions:
- Is this normal?
- I am wondering if the speed of repositioning the disk head to the beginning of a device is fast comparing to any file offset movement. because reading 512B should be elementary ...

Thanks a lot

Last edited by methyl; 07-03-2012 at 07:36 PM.. Reason: please use code tags
# 2  
Old 07-03-2012
Just because it's a raw disk device doesn't mean you're actually reading the disk raw. Disk cache and read-ahead gets used for it just like everything else.

Try hdparm for some more direct tests.
# 3  
Old 07-03-2012
Why is read-ahead a problem since above is only a 512bytes read, all it takes is normally positioning the drive head .. Then if the cache is the reason, it should affect the file as well, but I am getting the same result for all reps?

More importantly, my experiment is really about time/vs blocksize, and I am trying to make it more accurate :
- Shall I only use regular files?
- position the disk head somewhere else after each test?
- is it possible to flush all caches programmatically ?

Sorry for all those questions .. Smilie
# 4  
Old 07-03-2012
Quote:
Originally Posted by nytty
Why is read-ahead a problem since above is only a 512bytes read
Because, I repeat, you are not getting raw access here.

You are not getting raw access.

You are not telling the drive 'move to sector x, read'.

You are telling the operating system 'give me data from position x'.

The operating system goes 'Hmmm, someone asked for that a little while ago', pulls it from cache, and gives it to you without touching the disk.

Dropping caches all the time is a bad idea. If you really want to get disk access speeds, use hdparm.

Quote:
Then if the cache is the reason, it should affect the file as well, but I am getting the same result for all reps?
Your results aren't accurate. You're running huge programs to do tiny things and most of what you're measuring is going to be error and bias of some sort. If you want to do real benchmarks, hdparm.

There isn't a significant difference between file and raw disk unless your file is very badly fragmented.
# 5  
Old 07-03-2012
Alright, I get your point. I though that using /dev/sdb1 moves to position 0 Smilie
AFAIK hdparm only gives you the speed of the disk and not the time of reading a given amount of data.

So far with my tests I find that reading 16kb takes about the same time as reading 512b. I need this information to set the optimal page size of my system (oracle and mysql suggest these sizes)

Can you elaborate on why dropping the caches is a bad idea?
How about this sudo procedure:
- drop_caches
- flush disk-cache with hdparm
- make sure dd is in memory
- position the read head randomly on disk
- with DD: read X amount of bytes from the beginning of my file (this is timed)
# 6  
Old 07-03-2012
Quote:
Originally Posted by nytty
Alright, I get your point. I though that using /dev/sdb1 moves to position 0 Smilie
AFAIK hdparm only gives you the speed of the disk and not the time of reading a given amount of data.
Do a little math. x megabytes per second is 1/x seconds per megabyte.

Quote:
So far with my tests I find that reading 16kb takes about the same time as reading 512b.
Disks do read-ahead for you. Disks transfer to the host in larger bundles than 512 bytes anyway. Disks even do their own caching which the OS has no control over, which is going to throw off all your results supremely.

Too bad there isn't a tool which can tell you more about what your disk's doing, test uncached reads, or even configure hardware read-ahead to your preference... something like hdparm...
Quote:
I need this information to set the optimal page size of my system (oracle and mysql suggest these sizes)
Tell me exactly what they're asking you. I suspect you've gotten it a bit mixed up.
Quote:
Can you elaborate on why dropping the caches is a bad idea?
Because it's not realistic. Your system needs cache to work. Disk speeds are going to be awful without it.
Quote:
How about this sudo procedure:
- drop_caches
- flush disk-cache with hdparm
- make sure dd is in memory
- position the read head randomly on disk
- with DD: read X amount of bytes from the beginning of my file (this is timed)
Same problem as the exact same thing you did before. YOu're running huge programs to do tiny things and your results are going to be meaningless.

Last edited by Corona688; 07-03-2012 at 04:51 PM..
# 7  
Old 07-03-2012
At the application level page size is an important concept, it's the unit of interaction with the disk. Imagine a database system requesting a record A, it'd be completely inefficient to only read this record (couple of bytes), instead we collocate a bunch of records in a page and we bet on spatial/temporal locality ...
It's a tradeof, if you set the page size to too big, you risk over reading stuff you dont need. Too small, and you risk to get multiple requests for otherwise contiguous records. A bit what the os does with 4096b pages.
I am trying to study that, using a buffered read of different sizes and calculate the throughtput ... I know only basics about disks, but i am certain that the metric i am studying is not uniform, thus doing the math is just meaningless.
I should cope with all parameters(cache,vm,readahead..ect) and have something like:
bs=512b 0.01sec 50kb/s
bs=16kb 0.0004sec 164mb/s
bs=2M 0.01sec 200mb/s
Then i will decide that 16kb is the best time vs throughtput trade off and this will be my page size.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. OS X (Apple)

Change Name of Bluetooth Device from Command Line in macOS

Mac Version 10.15.2 (macOS Catalina) Does anyone know how to change the name of a connected bluetooth device from the command line on macOS? I am having trouble with various bluetooth devices which I cannot get the "rename" option in the GUI to "save" properly and so I cannot rename a few... (0 Replies)
Discussion started by: Neo
0 Replies

2. UNIX for Advanced & Expert Users

Command to see the logical volume path, device mapper path and its corresponding dm device path

Currently I am using this laborious command lvdisplay | awk '/LV Path/ {p=$3} /LV Name/ {n=$3} /VG Name/ {v=$3} /Block device/ {d=$3; sub(".*:", "/dev/dm-", d); printf "%s\t%s\t%s\n", p, "/dev/mapper/"v"-"n, d}' Would like to know if there is any shorter method to get this mapping of... (2 Replies)
Discussion started by: royalibrahim
2 Replies

3. Shell Programming and Scripting

tcgetattr: Inappropriate ioctl for device after ssh command

Hello everyone I am finishing a script allowing me to purge logs on multiple servers, i have one last pb with the ssh command.........it is throwing me the following error : tcgetattr: Inappropriate ioctl for device (full screen in attached file 1, full script in attached file 2) It... (15 Replies)
Discussion started by: jimmy75_13
15 Replies

4. Linux

Block device fs vs normal directory

I use CentOS I talked with this guy who seems to think loop mounting a fs ext3 image on a directory, as opposed to just using the underlying filesystem, will work better as far as IO conflicts. I have no idea why this would be better? I haven't been able to contact him. Basically we have a... (1 Reply)
Discussion started by: stevenswj
1 Replies

5. Shell Programming and Scripting

Block device fs vs normal directory

I talked with this guy who seems to think loop mounting a fs ext3 image on a directory, as opposed to just using the underlying filesystem, will work better as far as IO conflicts. I have no idea why this would be better? I haven't been able to contact him. Basically we have a daemon that... (0 Replies)
Discussion started by: stevenswj
0 Replies

6. Red Hat

How to Block the USB device

Hi friends, I wanna know how to block USB devices in my RedHat flavor Operating system. regards, Prakash (3 Replies)
Discussion started by: prakashkumar41
3 Replies

7. Linux

shrinking root partition and using free space to create a block device

We are intending to protect a set of user specified files using LVM mirroring where the protected space on which the user files are stored is mirrored on an LV on a different disk. Our problem is that for a user with a custom layout has installed linux with 2 partitons for swap and / and there is... (0 Replies)
Discussion started by: kickdgrass
0 Replies

8. UNIX for Dummies Questions & Answers

problem with output of find command being input to basename command...

Hi, I am triying to make sure that there exists only one file with the pattern abc* in path /path/. This directory is having many huge files. If there is only one file then I have to take its complete name only to use furter in my script. I am planning to do like this: if ; then... (2 Replies)
Discussion started by: new_learner
2 Replies

9. Solaris

block device problem

Hi ...... I have file system problem when i try to mount get the following message : mount: /dev/rdsk/c1t4d0s7 not a block device .... I run fsck it's ok ,,,, after run newfs -N /dev/dsk/c1t4d0s7 and fsck -o b=535952 /dev/dsk/c1t4d0s7 .............it's ok ........ At the and when i try... (2 Replies)
Discussion started by: tt155
2 Replies

10. UNIX for Dummies Questions & Answers

what is the command for listing the device specs in solaris

Im looking to get the statistics on a machine memory, cpu speed drive size etc thanks in advance (3 Replies)
Discussion started by: simplimarvelous
3 Replies
Login or Register to Ask a Question