Login or Register to Ask a Question and Join Our Community


UNIX for Advanced & Expert Users

Performance issue!

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums UNIX for Advanced & Expert Users Performance issue!
# 1  
Old 03-16-2009
Question Performance issue!
In my C program i am using very large file(approx 400MB) to read parts of it frequently. But due to large file the performance of the program goes down very badly. It shows very high I/O usage and I/O wait time.

My question is, What are the ways to optimize or tune I/O on linux or how i can get better performance when i have to read a large flat file?

I am using
Red Hat Enterprise Linux ES release 4 (Nahant Update 4)
2.6.9-42.EL, x86_64 GNU/Linux
# 2  
Old 03-17-2009
Did you mmap() the file?
# 3  
Old 03-19-2009
You can upgrade your hard disk to increase the read write capacity.
# 4  
Old 03-19-2009
If it's a local disk you could experiment with hdparm and your filesystems tuning utility (like tune2fs/reisertune/...) to tune it.
# 5  
Old 03-20-2009
No i haven't used mmap() the file. Should i use this?

Is there any specific hard disk type comes which is effective in read/write speed?

I have local disk. Which options u will recommend tuning? I will try few which i understand.
# 6  
Old 03-20-2009
For IDE disks, dma settings are quite important to performance.

Code:
$ hdparm -d /dev/hda

/dev/hda:
 using_dma     =  1 (on)

$ hdparm -I /dev/hda

/dev/hda:

ATA device, with non-removable media
        Model Number:       Maxtor 6L300R0
        Serial Number:      L60Z0NBG
        Firmware Revision:  BAJ41G20
Standards:
        Used: ATA/ATAPI-7 T13 1532D revision 0
        Supported: 7 6 5 4
Configuration:
        Logical         max     current
        cylinders       16383   65535
        heads           16      1
        sectors/track   63      63
        --
        CHS current addressable sectors:    4128705
        LBA    user addressable sectors:  268435455
        LBA48  user addressable sectors:  586114704
        device size with M = 1024*1024:      286188 MBytes
        device size with M = 1000*1000:      300090 MBytes (300 GB)
Capabilities:
        LBA, IORDY(can be disabled)
        Standby timer values: spec'd by Standard, no device specific minimum
        R/W multiple sector transfer: Max = 16  Current = 16
        Advanced power management level: unknown setting (0x0000)
        Recommended acoustic management value: 192, current value: 0
        DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 *udma5 udma6
             Cycle time: min=120ns recommended=120ns
        PIO: pio0 pio1 pio2 pio3 pio4
             Cycle time: no flow control=120ns  IORDY flow control=120ns
Commands/features:
        Enabled Supported:
           *    SMART feature set
                Security Mode feature set
           *    Power Management feature set
           *    Write cache
           *    Look-ahead
           *    Host Protected Area feature set
           *    WRITE_VERIFY command
           *    WRITE_BUFFER command
           *    READ_BUFFER command
           *    NOP cmd
           *    DOWNLOAD_MICROCODE
                Advanced Power Management feature set
                SET_MAX security extension
                Automatic Acoustic Management feature set
           *    48-bit Address feature set
           *    Device Configuration Overlay feature set
           *    Mandatory FLUSH_CACHE
           *    FLUSH_CACHE_EXT
           *    SMART error logging
           *    SMART self-test
                Media Card Pass-Through
           *    General Purpose Logging feature set
           *    WRITE_{DMA|MULTIPLE}_FUA_EXT
           *    URG for READ_STREAM[_DMA]_EXT
           *    URG for WRITE_STREAM[_DMA]_EXT
           *    SMART Command Transport (SCT) feature set
           *    SCT Data Tables (AC5)
Security:
        Master password revision code = 65534
                supported
        not     enabled
        not     locked
                frozen
        not     expired: security count
        not     supported: enhanced erase
HW reset results:
        CBLID- above Vih
        Device num = 0 determined by the jumper
Checksum: correct

Those two commands show whether DMA is enabled on an IDE drive, and if it is, what kind. hdparm -d1 /dev/hda will attempt to enable DMA transfer on an IDE drive if it isn't already.

hdparm can also do benchmarks:
Code:
$ hdparm -tT /dev/hda

/dev/hda:
 Timing cached reads:   292 MB in  2.00 seconds = 145.96 MB/sec
 Timing buffered disk reads:  140 MB in  3.04 seconds =  46.08 MB/sec
$

# 7  
Old 03-23-2009
My output is
1.
/dev/hda:
using_dma = 1 (on)

2.
/dev/hda:

ATAPI CD-ROM, with removable media
Model Number: CD-ROM 52x.Max
Serial Number: 1999/04/28
Firmware Revision: 1.12
Standards:
Used: ATAPI for CD-ROMs, SFF-8020i, r2.5
Supported: CD-ROM ATAPI-1
Configuration:
DRQ response: 50us.
Packet size: 12 bytes
Capabilities:
LBA, IORDY(can be disabled)
DMA: sdma0 sdma1 sdma2 mdma0 mdma1 *mdma2
Cycle time: min=120ns recommended=120ns
PIO: pio0 pio1 pio2 pio3 pio4
Cycle time: no flow control=120ns IORDY flow control=120ns
3.
/dev/hda:
read() failed: Input/output error
Timing buffered disk reads: read() failed: Input/output error

Do you see anything i can tune?
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Red Hat

Performance issue in Linux

IN solaris, for network high-availability we are using IPMP concept, can u tell me in REDHAT LINUX what we are using... also pls share good step to read & understand the that concept... Also performance issue in linux what are step & cmd can u tell me??? (2 Replies)
Discussion started by: tiger09
2 Replies

2. AIX

Performance issue

Hi We have an AIX5.3 server with application which is written in C. We are facing server (lpar) hangs intermediately. If we open new telnet window prompts for user and takes hell of a time to authenticate, not only that if we run ps -aef then also it takes lot of time. surprisingly there is no... (2 Replies)
Discussion started by: powerAIX
2 Replies

3. AIX

Performance issue

Hi, We have 2 lpars on p6 blade. One of the lpar is having 3 core cpu with 5gb memory running sybase as database. An EOD process takes 25 min. to complete. Now we have an lpar on P7 server with entitled cpu capacity of 2 with 16 Gb memory and sybase as database. The EOD process which takes... (17 Replies)
Discussion started by: vjm
17 Replies

4. UNIX for Dummies Questions & Answers

Performance issue

hi I am having a performance issue with the following requirement i have to create a permutation and combination on a set of three files such that each record in each file is picked and the output is redirected in a specific format but it is taking around 70 odd hours to prepare a combination... (7 Replies)
Discussion started by: mad_man12
7 Replies

5. Shell Programming and Scripting

Performance issue or something else?

Hi All, I have the following script which I use in Nagios to check the health of the applications, the problem with it is that the curl part ($TOTAL) does not return anything after running for 2-3 hrs, even though from command line the script runs fine but not from Nagios. There are 17... (1 Reply)
Discussion started by: jacki
1 Replies

6. Solaris

Performance issue

Hi Gurus, I am beginner in solaris and want to know what are the things we need to check for performance monitoring on our solairs OS. for DISK,CPU and MEMORY. Also how we do ipforwarding in slaris Many thanks for your help Pradeep P (4 Replies)
Discussion started by: ppandey21
4 Replies

7. Shell Programming and Scripting

performance issue

I want to read a file. is it good to use File I/O or shell script?? which one is the best option? (1 Reply)
Discussion started by: vishwaraj
1 Replies

8. UNIX for Advanced & Expert Users

performance issue

Hi, on a linux server I have the following : vmstat 2 10 procs memory swap io system cpu r b w swpd free buff cache si so bi bo in cs us sy id 0 4 0 675236 39836 206060 1617660 3 3 3 6 8 7 1 1 ... (1 Reply)
Discussion started by: big123456
1 Replies

9. AIX

performance issue

We have a AIX v5.3 on a p5 system with a poor performing Ingres database. We added one CPU to the system to see if this would help. Now there are two CPU's. with sar and topas -P I see good results: CPU usage around 30% with topas I only see good results in the process output screen, the... (1 Reply)
Discussion started by: rein
1 Replies

10. UNIX for Advanced & Expert Users

Performance issue

Hello all, I just stuck up in an uncertain situation related to network performance... I am trying to access one of my remote client unix machine from a distant location.. The client machine is Ultra-5_10 , with SunOS 5.5.1 The ndd result ( hme1 )shows that the machine is hooked to a... (5 Replies)
Discussion started by: shibz
5 Replies
Login or Register to Ask a Question