Sponsored Content
Top Forums Programming Tool to simulate non-sequential disk I/O (simulate db file sequential read) in C POSIX Post 302531294 by vrghost on Thursday 16th of June 2011 10:25:46 AM
Old 06-16-2011
Tool to simulate non-sequential disk I/O (simulate db file sequential read) in C POSIX

Writing a Tool to simulate non-sequential disk I/O (simulate db file sequential read) in C POSIX

I have over the years come across the same issue a couple of times, and it normally is that the read speed on SAN is absolutely atrocious when doing non-sequential I/O to the disks. Problem being of course that most databases will be doing non-sequential I/O to disks, databases most common read process is db file sequential reads, which would not cause a sequential read of the actual blocks on the device.

My second issue is that it is normally tricky to segregate the different processes enough to be able to clearly test or even show the exact issue with the non-sequential reads and writes, so I end up in very lengthy discussions about possibilities of changing computing theory rather than actually changing the SAN to be able to handle these type of requests, and while that allows for very creative use of similes it is not very efficient use of my time, and I really have little need for more overtime.

So my thought was, how would one go about writing a utility that takes a file that opens a large file and read random blocks of data throughout the file, that way simulating the same effect in a controlled environment.

The general layout I was thinking is

Input for program

name [file to read] [block size] [number of reads]

set block size
set number of reads

get file size

open file

for I << number of reads
set random block address
read random block address from file

close


My problem is, how would I go about reading a random block address from a file.
And is there any way to get the time in milliseconds the operation took

And the POSIX bit, basically the systems I need to use this code on are locked down pretty heavily, and installing a new compiler is a couple of months worth of work, so I want a tool that will be able to be compiled on almost any old compiler.
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

inserting uniq sequential numbers at the start of the file

Hi Unix gurus, I have a file. I need to insert sequential number at the starting of the file. Fields are delimited by "|". I know the starting number. Example: File is as follows |123|4test|test |121|2test|test |x12|1test|test |vd123|5test|test starting number is : 120 ... (7 Replies)
Discussion started by: jingi1234
7 Replies

2. Programming

Reading special characters while converting sequential file to line sequential

We have to convert a sequential file to a 80 char line sequential file (HP UX platform).The sequential file contains special characters. which after conversion of the file to line sequential are getting coverted into "new line" or "tab" and file is getting distorted. Is there any way to read these... (2 Replies)
Discussion started by: Rajeshsu
2 Replies

3. Shell Programming and Scripting

Finding missing sequential file names

So, I've got a ton of files that I want to go through (ie something like 300,000), and they're all labeled sequentially. However I'm not 100% positive that they are all there. Is there any way of running through a sequence of numbers, checking if the file is in the folder, if not appending it... (2 Replies)
Discussion started by: Julolidine
2 Replies

4. Shell Programming and Scripting

how to scan a sequential file to fetch some of the records?

Hi I am working on a script which needs to scan a sequential file and fetch the row where 2nd column = 'HUB' Can any one help me with this... Thanks (1 Reply)
Discussion started by: manmeet
1 Replies

5. Shell Programming and Scripting

How to sca a sequential file and fetch some substring data from it

Hi, I have a task where i need to scan second column of seuential file and fetch first 3 digits of that column For e.g. FOLLOWING IS THE SAMPLE FOR MY SEQUENTIAL FILE AU_ID ACCT_NUM CRNCY_CDE THHSBC001 30045678 THB THHSBC001 10154267 THB THHSBC001 ... (2 Replies)
Discussion started by: manmeet
2 Replies

6. Shell Programming and Scripting

Sequential comparison (one row with file and so on)

Dear linux experts, I'd like to ask for your support, I've read some posts in this forum about files comparison but haven't found what I'm looking for. I need to create a sequential script to compare row-by-row one file with 34 similar files but without success so far. This is what I get: ... (2 Replies)
Discussion started by: Gery
2 Replies

7. Shell Programming and Scripting

sequential to line sequential

Hi I have a file sequential way i.e. written in contineous mode and the Record Seperator is AM from which the record is seperated .Now to process I have to make line sequential,and more over record length is not same it varies as per the input address, AM1234563 John Murray 24 Old streeet old... (5 Replies)
Discussion started by: vakharia Mahesh
5 Replies

8. UNIX for Dummies Questions & Answers

Inserting a sequential number into a field on a flat file

I have a csv flatfile with a few million rows. I need to replace a field (field number is 85) in the file with a sequential number. As an example, let's assume there are only 4 fields in the file: A,A,,32 A,A,,27 A,B,,43 C,C,,354 If I wanted to amend the 3rd field in this way my... (2 Replies)
Discussion started by: BristolSmithy
2 Replies

9. Shell Programming and Scripting

Read directories sequential based on timestamp

Hi, I have a directory structure like below Directoryname create time d1 12:00 d2 12:05 d3 12:08 I want to read the directories based on timestamp.That is oldest directory must be read first and kick off certain process. ... (7 Replies)
Discussion started by: chetan.c
7 Replies

10. Shell Programming and Scripting

Sequential Reading from two file in a loop

Hello All, I have two files with me file1.txt and file2.txt file1.txt has: 333 222 111 file2.txt has ccc bbb aaa ccc is related to 333 only, bbb is related to 222 only and aaa is related to 111 only. I have to get the values from each of the file and pass them in the URL... (3 Replies)
Discussion started by: ankur328
3 Replies
SA(4)							   BSD Kernel Interfaces Manual 						     SA(4)

NAME
sa -- SCSI Sequential Access device driver SYNOPSIS
device sa DESCRIPTION
The sa driver provides support for all SCSI devices of the sequential access class that are attached to the system through a supported SCSI Host Adapter. The sequential access class includes tape and other linear access devices. A SCSI Host adapter must also be separately configured into the system before a SCSI sequential access device can be configured. MOUNT SESSIONS
The sa driver is based around the concept of a ``mount session'', which is defined as the period between the time that a tape is mounted, and the time when it is unmounted. Any parameters set during a mount session remain in effect for the remainder of the session or until replaced. The tape can be unmounted, bringing the session to a close in several ways. These include: 1. Closing a `rewind device', referred to as sub-mode 00 below. An example is /dev/sa0. 2. Using the MTOFFL ioctl(2) command, reachable through the 'offline' command of mt(1). It should be noted that tape devices are exclusive open devices, except in the case where a control mode device is opened. In the latter case, exclusive access is only sought when needed (e.g., to set parameters). SUB-MODES Bits 0 and 1 of the minor number are interpreted as 'sub-modes'. The sub-modes differ in the action taken when the device is closed: 00 A close will rewind the device; if the tape has been written, then a file mark will be written before the rewind is requested. The device is unmounted. 01 A close will leave the tape mounted. If the tape was written to, a file mark will be written. No other head positioning takes place. Any further reads or writes will occur directly after the last read, or the written file mark. 10 A close will rewind the device. If the tape has been written, then a file mark will be written before the rewind is requested. On completion of the rewind an unload command will be issued. The device is unmounted. BLOCKING MODES
SCSI tapes may run in either 'variable' or 'fixed' block-size modes. Most QIC-type devices run in fixed block-size mode, where most nine- track tapes and many new cartridge formats allow variable block-size. The difference between the two is as follows: Variable block-size: Each write made to the device results in a single logical record written to the tape. One can never read or write part of a record from tape (though you may request a larger block and read a smaller record); nor can one read multiple blocks. Data from a sin- gle write is therefore read by a single read. The block size used may be any value supported by the device, the SCSI adapter and the system (usually between 1 byte and 64 Kbytes, sometimes more). When reading a variable record/block from the tape, the head is logically considered to be immediately after the last item read, and before the next item after that. If the next item is a file mark, but it was never read, then the next process to read will immediately hit the file mark and receive an end-of-file notification. Fixed block-size: Data written by the user is passed to the tape as a succession of fixed size blocks. It may be contiguous in memory, but it is considered to be a series of independent blocks. One may never write an amount of data that is not an exact multiple of the blocksize. One may read and write the same data as a different set of records. In other words, blocks that were written together may be read sepa- rately, and vice-versa. If one requests more blocks than remain in the file, the drive will encounter the file mark. As there is some data to return (unless there were no records before the file mark), the read will succeed, returning that data. The next read will return immediately with a value of 0. (As above, if the file mark is never read, it remains for the next process to read if in no-rewind mode.) BLOCK SIZES
By default, the driver will NOT accept reads or writes to a tape device that are larger than may be written to or read from the mounted tape using a single write or read request. Because of this, the application author may have confidence that his wishes are respected in terms of the block size written to tape. For example, if the user tries to write a 256KB block to the tape, but the controller can handle no more than 128KB, the write will fail. The previous FreeBSD behavior, prior to FreeBSD 10.0, was to break up large reads or writes into smaller blocks when going to the tape. The problem with that behavior, though, is that it hides the actual on-tape block size from the application writer, at least in variable block mode. If the user would like his large reads and writes broken up into separate pieces, he may set the following loader tunables. Note that these tunables WILL GO AWAY in FreeBSD 11.0. They are provided for transition purposes only. kern.cam.sa.allow_io_split This variable, when set to 1, will configure all sa devices to split large buffers into smaller pieces when needed. kern.cam.sa.%d.allow_io_split This variable, when set to 1, will configure the given sa unit to split large buffers into multiple pieces. This will override the global setting, if it exists. There are several sysctl(8) variables available to view block handling parameters: kern.cam.sa.%d.allow_io_split This variable allows the user to see, but not modify, the current I/O split setting. The user is not permitted to modify this setting so that there is no chance of behavior changing for the application while a tape is mounted. kern.cam.sa.%d.maxio This variable shows the maximum I/O size in bytes that is allowed by the combination of kernel tuning parameters (MAXPHYS, DFLTPHYS) and the capabilities of the controller that is attached to the tape drive. Applications may look at this value for a guide on how large an I/O may be permitted, but should keep in mind that the actual maximum may be restricted further by the tape drive via the SCSI READ BLOCK LIMITS command. kern.cam.sa.%d.cpi_maxio This variable shows the maximum I/O size supported by the controller, in bytes, that is reported via the CAM Path Inquiry CCB (XPT_PATH_INQ). If this is 0, that means that the controller has not reported a maximum I/O size. FILE MARK HANDLING
The handling of file marks on write is automatic. If the user has written to the tape, and has not done a read since the last write, then a file mark will be written to the tape when the device is closed. If a rewind is requested after a write, then the driver assumes that the last file on the tape has been written, and ensures that there are two file marks written to the tape. The exception to this is that there seems to be a standard (which we follow, but do not understand why) that certain types of tape do not actually write two file marks to tape, but when read, report a `phantom' file mark when the last file is read. These devices include the QIC family of devices. (It might be that this set of devices is the same set as that of fixed block devices. This has not been determined yet, and they are treated as separate behaviors by the driver at this time.) IOCTLS
The sa driver supports all of the ioctls of mtio(4). FILES
/dev/[n][e]sa[0-9] general form: /dev/sa0 Rewind on close /dev/nsa0 No rewind on close /dev/esa0 Eject on close (if capable) /dev/sa0.ctl Control mode device (to examine state while another program is accessing the device, e.g.). DIAGNOSTICS
None. SEE ALSO
mt(1), cam(4) AUTHORS
The sa driver was written for the CAM SCSI subsystem by Justin T. Gibbs and Kenneth Merry. Many ideas were gleaned from the st device driver written and ported from Mach 2.5 by Julian Elischer. The current owner of record is Matthew Jacob who has suffered too many years of breaking tape drivers. BUGS
This driver lacks many of the hacks required to deal with older devices. Many older SCSI-1 devices may not work properly with this driver yet. Additionally, certain tapes (QIC tapes mostly) that were written under FreeBSD 2.X are not automatically read correctly with this driver: you may need to explicitly set variable block mode or set to the blocksize that works best for your device in order to read tapes written under FreeBSD 2.X. Fine grained density and compression mode support that is bound to specific device names needs to be added. Support for fast indexing by use of partitions is missing. BSD
August 23, 2013 BSD
All times are GMT -4. The time now is 09:24 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy