Sponsored Content
Top Forums Programming performance of writing into a file Post 93815 by jim mcnamara on Thursday 22nd of December 2005 09:51:22 AM
Old 12-22-2005
First the open-write-close thing is fine if it happens once in a while. If it is going on constantly it is a big performance drain. There are other options - check into aio:
Code:
man aio

To answer your question: it depends on two things: your hardware and how fragmented the file is.

If the file was very recently closed then the data may still be in the on-board cache in the disk controller or SAN box cache. In which case there is no appreciable performance difference.

If the file/disk is fragmented then you have another set of problems. The first write may require an extra head seek and then some rotational latency because it is likely the filesystem will have to add another separate physical sector (cluster of sectors) to the file. Or that the write will span disjoint sectors (depending on the filesystem and what you are writing)

It also depends on how much data you are writing. write() will defer direct I/O til later under a lof of circumstances, except a fflush() or sync(). close() does not necessarily flush the file to disk - see Marc Rochkind's 'Advanced Unix Programming' for more information on deferred write. Think of write as more of a promise than a fact in some circumstances.

Last edited by jim mcnamara; 12-22-2005 at 10:58 AM..
 

10 More Discussions You Might Find Interesting

1. News, Links, Events and Announcements

Announcing collectl - new performance linux performance monitor

About 4 years ago I wrote this tool inspired by Rob Urban's collect tool for DEC's Tru64 Unix. What makes this tool as different as collect was in its day is its ability to run at a low overhead and collect tons of stuff. I've expanded the general concept and even include data not available in... (0 Replies)
Discussion started by: MarkSeger
0 Replies

2. UNIX for Dummies Questions & Answers

Problem writing file path to txt file

if test -z "$1" then echo "you must give a filename or filepath" else path=`dirname $1` f_name =`basename $1` if path="." then path=`pwd` fi fi cat $f_name $path >> index.txt The only problem I am encountering with this is writing $path to index.txt Keeps going gaga: cat:... (1 Reply)
Discussion started by: Vintage_hegoog
1 Replies

3. Shell Programming and Scripting

Performance issue in UNIX while generating .dat file from large text file

Hello Gurus, We are facing some performance issue in UNIX. If someone had faced such kind of issue in past please provide your suggestions on this . Problem Definition: /Few of load processes of our Finance Application are facing issue in UNIX when they uses a shell script having below... (19 Replies)
Discussion started by: KRAMA
19 Replies

4. Shell Programming and Scripting

Searching for Log / Bad file and Reading and writing to a flat file

Need to develop a unix shell script for the below requirement and I need your assistance: 1) search for file.log and file.bad file in a directory and read them 2) pull out "Load_Start_Time", "Data_File_Name", "Error_Type" from log file 4) concatinate each row from bad file as... (3 Replies)
Discussion started by: mlpathir
3 Replies

5. Shell Programming and Scripting

Comparing rows in same file and writing the result in new file

Help needed... Can you tell me how to compare the last two couple entries in a file and print their result in new file..:confused: I have one file Check1.txt \abc1 12345 \abc2 12327 \abc1 12345 \abc2 12330 I want to compare the entries in Check1 and write to... (1 Reply)
Discussion started by: kichu
1 Replies

6. Shell Programming and Scripting

Writing file name and date from LS command into a file to be imported into mysql

I am looking to do a ls on a folder and have the output of the ls be structured so that is is modificaiton date, file name with the date in a format that is compatible with mysql. I am trying to build a table that stores the last modification date of certain files so I can display it on some web... (4 Replies)
Discussion started by: personalt
4 Replies

7. Shell Programming and Scripting

reading a file extracting information writing to a file

Hi I am trying to extract information out of a file but keep getting grep cant open errors the code is below: #bash #extract orders with blank address details # # obtain the current date # set today to the current date ccyymmdd format today=`date +%c%m%d | cut -c24-31` echo... (8 Replies)
Discussion started by: Bruble
8 Replies

8. UNIX for Dummies Questions & Answers

Need help in not fetching a file while file writing operation is not completed

Hi All, We have a Unix program in oracle when we run the program this connects to specified ftp and will get the file into local server. We are facing a problem like when file writing operations is not completed, this program is getting the incomplete file. Could anyone please help me... (2 Replies)
Discussion started by: world.apps
2 Replies

9. UNIX for Dummies Questions & Answers

Writing a script that will take the first line from each file and store it in an output file

Hi, I have 1000 files names data1.txt through data1000.txt inside a folder. I want to write a script that will take each first line from the files and write them as output into a new file. How do I go about doing that? Thanks! (2 Replies)
Discussion started by: evelibertine
2 Replies

10. Shell Programming and Scripting

awk - writing matching pattern to a new file and deleting it from the current file

Hello , I have comma delimited file with over 20 fileds that i need to do some validations on. I have to check if certain fields are null and then write the line containing the null field into a new file and then delete the line from the current file. Can someone tell me how i could go... (2 Replies)
Discussion started by: goddevil
2 Replies
DKCTL(8)						    BSD System Manager's Manual 						  DKCTL(8)

NAME
dkctl -- program to manipulate disks SYNOPSIS
dkctl device dkctl device command [arg [...]] DESCRIPTION
dkctl allows a user or system administrator to manipulate and configure disks in various ways. It is used by specifying a disk to manipu- late, the command to perform, and any arguments the command may require. device is the disk (wdN, sdN, ...) containing the wedges unless noted otherwise. If dkctl is called without any command, it displays strategy, cache, and all of the wedges of the specified device. COMMANDS
The following commands are supported: getcache Get and display the cache enables for the specified device. setcache none | r | w | rw [save] Set the cache enables for the specified device. The enables are as follows: none Disable all caches on the disk. r Enable the read cache, and disable all other caches on the disk. w Enable the write cache, and disable all other caches on the disk. rw Enable both the read and write caches on the disk. save If specified, and the cache enables are savable, saves the cache enables in the disk's non-volatile parame- ter storage. synccache [force] Causes the cache on the disk to be synchronized, flushing all dirty write cache blocks to the media. If force is speci- fied, the cache synchronization command will be issued even if the kernel does not believe that there are any dirty cache blocks in the disk's cache. keeplabel [yes | no] Specify to keep or drop the in-core disklabel on the last close of the disk device. (Keep if yes is specified, drop if no is specified.) badsector flush | list | retry Used for managing the kernel's bad sector list for wd(4) devices. The software bad sector list is only maintained if the option ``WD_SOFTBADSECT'' was specified on kernel configuration. flush Clears the in kernel list of bad sectors. list Prints out the list of bad sector ranges recorded by the kernel. retry Flushes the in kernel list and then retries all of the previously recorded bad sectors, causing the list to self update. This option can only be used with character devices. addwedge name startblk blkcnt ptype Define a ``wedge'' on the specified disk starting at block number startblk and spanning blkcnt blocks. You need to cre- ate the partition first with fdisk(8) or gpt(8), dkctl will just name it. The wedge will have the volume name name and the partition type ptype. Valid choices for ptype would be unused, swap, ffs, lfs, ext2fs, cd9660, ados, hfs, msdos, filecore, raidframe, ccd, appleufs, ntfs, and cgd. The device name of the virtual block device assigned to the wedge will be displayed after the wedge has been successfully created. See dk(4) for more information about disk wedges. delwedge dk Delete the wedge specified by its device name dk from the specified disk. getwedgeinfo Display information about the specified disk wedge. device in this case is the wedge name. listwedges List all of the wedges configured on the specified disk. strategy [name] Get and set the disk I/O scheduler (buffer queue strategy) on the drive. If you do not provide a name argument, the cur- rently selected strategy will be shown. To set the bufq strategy, the name argument must be specified. name must be the name of one of the built-in kernel disk I/O schedulers. To get the list of supported schedulers, use the following com- mand: $ sysctl kern.bufq.strategies Note: The addwedge and delwedge commands only modify the in-kernel representation of disks; for modifying information on the disks them- selves, refer to fdisk(8) or gpt(8). SEE ALSO
ioctl(2), dk(4), sd(4), wd(4), disklabel(5), atactl(8), fdisk(8), gpt(8), scsictl(8) HISTORY
The dkctl command first appeared in NetBSD 1.6. AUTHORS
The dkctl command was written by Jason R. Thorpe of Wasabi Systems, Inc. BSD
January 5, 2011 BSD
All times are GMT -4. The time now is 11:30 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy