06-17-2009
File alignment and performances... (difficult)
Hello !
I will use my best english possible to explain my objective. I'm french so pardon for the lack of precision...
So, what i would like to do in shell script (but you will possibly answer ''not possible in script'' have to use low level langage or something like that) is described below. All the blue is already done and working.
- from a list of files of the same size, sorted by their name. Their name are adjacent numbers so they are like : 0001, 0002, 0003 ect...
- copy them to one big file by concatenating each of them, one by one to one big file (doing that with dd and big block size because of fast I/O system and big file size)
- cut the bigfile into pieces to recreate original files...
Before explaining why the "split" command do not match for me let's precise the context of my objective...
I'm trying to raise disk I/O performance on some group of files by putting them near to each others physically on the hard drive. Those files are big files (about 10MB each) that have to be read in a sequence order (like 1 then 2 then 3 ...) and hard drive head movement when file1 is far from file2 cost a LOT of performance.
As it is not possible to change the physical address of a file on a storage device, objective is to ''bluff'' the OS filesystem : copying a lot of files into one big (thus filesystem will try to write one big file in adjacent sectors) file.
I don't want to grow this post too big but if you want more details i will give some with pleasure.
So, i don't want split command because it's copying from one source file to multi destination. As i said before, generating new files will allow filesystem to spread them all over the drive, and i loose performance again...
Would some other command could help ? Is it possible to cut one big file into piece by only generating new entries in inode table to be as fast as possible ?
Is there some other solution than script thinkable ?
Thanks a lot for your help and your ideas !
Have a good day !
-----Post Update-----
perhaps should i have posted this to filesystem & disk section ?
10 More Discussions You Might Find Interesting
1. UNIX for Advanced & Expert Users
Sir,
I have a file containing say 1000 lines that contain 100 paragraphs of 10 lines each separated by blank lines.I have to match a pattern or a string "hdfhasdjkasdhs" and print the complete paragraphs containing these strings.I can do this with the help of line editor ex,but how can I use... (1 Reply)
Discussion started by: Piyush
1 Replies
2. AIX
Helo:
We updated form AIX 4.3.3 to AIX 5.1-7 and after this we spent more than double time in read from external disks. Aparently the ssa cards microcode is at last level and all the ptf and apars are instaled.
Out backups expent more than double time, but curiously in read only, if we write in... (0 Replies)
Discussion started by: Javier Gutierre
0 Replies
3. Solaris
:rolleyes: Hi,
How to take the time diffence between start and finish time from a log file?
It is like
..... started at Jun 20 23:20
.
.
..... finished at Jun 21 01:40
Tryed so many ways but failed to ger exact way. :confused:
Your help will be honoured.
Ta........Lokesha (1 Reply)
Discussion started by: Lokesha
1 Replies
4. Shell Programming and Scripting
Hi,
I'm a beginners, this is one of my first script, it's easy, but I don't know how to write this script:
The script receive in input 4 parameters:
1) user_name
2) r and/or w and/or x ( rwx, rw, x, ....)
3) u and/or g and/or o ( u, uo, ugo, ...)
4) the path name
The script print a... (2 Replies)
Discussion started by: DNAx86
2 Replies
5. UNIX for Dummies Questions & Answers
en...how to grep some words from some file,
the goal is that, we donot want the words are exactly 9 charactor,
and want to grep from some words that are longer than 9 and it contain a substring that with 9 different charactors (2 Replies)
Discussion started by: shrimpy
2 Replies
6. Linux
Hi
I would like to thank you all for this excellent forum.
Today i tried to compare two files and i get some problem with it.
I have two files and i want to get all the data that match the first file like this
File1 (pattern file)
___________________________
9007
9126
9918
9127 ... (6 Replies)
Discussion started by: tafil
6 Replies
7. Solaris
Hello every body,
Maybe someone could help me.
I have a SUN Server with 6 disks, each of 150 Gb.
I have mounted the first two disk in mirror (RAID1) for the system files.
I have mounted 3 disks in RAID5 for users file systems.
I kept the last one as spare and I have mounted it standalone... (6 Replies)
Discussion started by: aribault
6 Replies
8. Shell Programming and Scripting
I don't know if this is a big issue or not, but I'm having difficulties. I apoligize for the upcoming essay :o.
I'm writing a script, similar to a paint program that edits images, but in the form of ANSI block characters. The program so far is working. I managed to save the image into a file,... (14 Replies)
Discussion started by: tinman47
14 Replies
9. Programming
Hello,
I was reading Heuritics text and came across an algorithm below. Finding hard to analyze it can any one help me out below...
How to analyze if I take say no. of types are 5 and each type has say 20 coins.
thanks.
Let {c1, c2...cn=1} be a set of distinct coin types where ci is... (1 Reply)
Discussion started by: sureshcisco
1 Replies
10. Shell Programming and Scripting
Hi,
I have one file with one column and several hundred entries
File1:
NA1
NA2
NA3And now I need to run a command within a mapping aligner tool to insert these sample names into a sequence alignment file (SAM) such that they look like this
@RG ID:Library1 SM:NA1 PL:Illumina ... (7 Replies)
Discussion started by: nans
7 Replies
LEARN ABOUT CENTOS
tunefs
tunefs(1M) System Administration Commands tunefs(1M)
NAME
tunefs - tune an existing UFS file system
SYNOPSIS
tunefs [-a maxcontig] [-d rotdelay] [-e maxbpg] [-m minfree] [-o space | time] special | filesystem
DESCRIPTION
tunefs is designed to change the dynamic parameters of a file system that affect the layout policies. When using tunefs with filesystem,
filesystem must be in /etc/vfstab. The parameters that can be changed are indicated by the options given below.
OPTIONS
The following options are supported:
-a maxcontig
The maximum number of logical blocks, belonging to one file, that is allocated contiguously. The default is calculated as follows:
maxcontig = disk drive maximum transfer size / disk block size
If the disk drive's maximum transfer size cannot be determined, the default value for maxcontig is calculated from kernel parameters as
follows:
If maxphys is less than ufs_maxmaxphys, which is 1 Mbyte, then maxcontig is set to maxphys. Otherwise, maxcontig is set to ufs_maxmax-
phys.
You can set maxcontig to any positive integer value.
The actual value will be the lesser of what has been specified and what the hardware supports.
-d rotdelay
This parameter is obsolete as of the Solaris 10 release. The value is always set to 0, regardless of the input value.
-e maxbpg
Indicates the maximum number of contiguous logical blocks any single file can allocate from a cylinder group before it is forced to
begin allocating blocks from another cylinder group. Typically this value is set to approximately one quarter of the total contiguous
logical blocks in a cylinder group. The intent is to prevent any single file from using up all the blocks in a single cylinder group,
thus degrading access times for all files subsequently allocated in that cylinder group.
The effect of this limit is to cause big files to do long seeks more frequently than if they were allowed to allocate all the blocks in
a cylinder group before seeking elsewhere. For file systems with exclusively large files, this parameter should be set higher.
-m minfree
Specifies the minimum free space threshold, or the percentage of space held back from normal users. This value can be set to 0. How-
ever, up to a factor of three in throughput will be lost over the performance obtained at a 10% threshold. Note: If the value is raised
above the current usage level, users will be unable to allocate files until enough files have been deleted to get under the higher
threshold.
-o space|time
The file system can either be instructed to try to minimize the time spent allocating blocks, or to try to minimize the space fragmen-
tation on the disk. The default is time.
Generally, you should optimize for time unless the file system is over 90% full.
USAGE
See largefile(5) for the description of the behavior of tunefs when encountering files greater than or equal to 2 Gbyte ( 2**31 bytes).
ATTRIBUTES
See attributes(5) for descriptions of the following attributes:
+-----------------------------+-----------------------------+
| ATTRIBUTE TYPE | ATTRIBUTE VALUE |
+-----------------------------+-----------------------------+
|Availability |SUNWcsu |
+-----------------------------+-----------------------------+
SEE ALSO
mkfs_ufs(1M), newfs(1M), attributes(5), largefile(5)
SunOS 5.10 5 Dec 2003 tunefs(1M)