File - reading - Performance improvement Post: 302197896

Sponsored Content

Top Forums Programming File - reading - Performance improvement Post 302197896 by dhanamurthy on Wednesday 21st of May 2008 11:31:15 PM

05-22-2008

Registered User

File - reading - Performance improvement

Hi All
I am reading a huge file of size 2GB atleast. I am reading each line and cutting certain columns and writing it to another file.

Here is the logic.

Code:

int main()
{
     
	  string u_line;
	  string Char_List;
	  string u_file;
	  int line_pos;
	  string temp_form_u_file;
	  ofstream temp_u_file;
	  u_file=getenv("u_file");
	  temp_form_u_file=getenv("DATA_DIR");
	  ifstream U_File;
	  temp_u_file.open(temp_form_u_file.c_str(),ios::app);
      
	  
	  if (temp_u_file.fail()) {
      cout << "Unable to open file "<<temp_form_u_file<<" for writing" << endl;
      exit(1);
      }
       
	    
      U_File.open(u_file.c_str());
      if (U_File.fail())
      {
         cout<<"File "<<u_file<<" unable to open for reading\n";
         cout<<"dart_report job failed\n";
	     exit(3);
      } 
    
      while (! U_File.eof() )
      {

		 line_pos=72;
		 u_line.erase();
		 getline (U_File,u_line);

		 if ( ! u_line.empty())  {
         while (line_pos< u_line.length())
	     {
           
	       if (u_line.substr(line_pos,2)!= "  ")
	       {
  
				Char_List=u_line.substr(line_pos,41);
				Char_List.append(u_line.substr(16,4));
				Char_List.append("\n");
				temp_u_file<< Char_List;
          
           } 
                line_pos=line_pos+41;

	    }

      }
    }  
}

When i run this program it takes 2.5 to 3 hours to read the 2 GB file. I am trying to reduce the time taken to reading. Is there any way i can reduce the processing time of the program.

Kindly let me know. If i can use Shell Script it is also okay. But i feel 'C' will be faster than Shell Scripting.

Please give me your suggestions.

Regards
Dhana

Last edited by Yogesh Sawant; 05-22-2008 at 02:35 AM.. Reason: added code tags

dhanamurthy

View Public Profile for dhanamurthy

Find all posts by dhanamurthy

7 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Script ready but might need some improvement.

Hi All, I have written a script which does some editing in the files, based on user input.This might not be the most elegant way of doing it and there would be many improvements needed. Please go through it and let me know how it could be improved. Suggestions are welcome!! Thanks!...

2. Shell Programming and Scripting

Any improvement possible in this script

Hi! Thank you for the help yesterday This is the finished product There is one more thing I would like to do to it but I’m not to certain On how to proceed I would like to log all output to a log in order to Be able to roll back This script is meant to be used in repairing a...

3. Infrastructure Monitoring

Possible performance improvement (Bash and flat file)

Hello, I am pretty new to shell scripts and I recently wrote one that seems to do what it should but I am exploring the possibility of improving its performance and would appreciate some help. Here is what it does - Its meant to monitor a bunch of systems (reads in IPs one at a time from a flat...

4. UNIX for Advanced & Expert Users

linux os improvement

can anyone help to share the knowledge on linux os improvement? 1) os account - use window AD authentication, such as ldap, but how to set /etc/passwd, where to put user home? 2) user account activity - how to log os user activity share the idea and what tools can do that...thx

5. Shell Programming and Scripting

I need the improvement for my script

Hi All, Here is my script #! /bin/sh var1=some email id var2=some email id grep -i "FAILED FILE FORMAT VALIDATION" /opt >tmp2 diff tmp1 tmp2 | grep ">" >tmp3 if then cat tmp3 | mailx -s " Error Monitoring" $var2 else echo "Pattern NOt Found" | mailx -s " Error Monitoring" $var1...

6. UNIX for Dummies Questions & Answers

Improvement in shell script

Hi This is my Following code: #!/bin/sh echo "TOTAL_NO_OF_MAILS" read TOTAL_NO_OF_MAILS echo "TOTAL_NO_OF_TICKETS " read TOTAL_NO_OF_TICKETS echo "TICKETS_IN_QUEUE" read TICKETS_IN_QUEUE rm -rf `pwd`/Focus echo "Hi Team\nSTATS IN CLRS MAIL BOX\n\n==============================" >> Focus...

7. Shell Programming and Scripting

Performance improvement in grep

Below script is used to search numeric data from around 400 files in a folder. I have 300 such folders. Need help in performance improvement in the script. Below Script searches 20 such folders ( 300 files in each folder) simultaneously. This increases cpu utilization upto 90% What changes...

LEARN ABOUT OSF1

dvdfs

dvdfs(4)						     Kernel Interfaces Manual							  dvdfs(4)

NAME

       dvdfs - The Digital Versatile Disk File System  [This feature is not supported in Tru64 UNIX Version 5.0]

STANDARDS

       Interfaces documented on this reference page conform to industry standards as follows:

       ISO/ITEC 13346:1995

	      Optical Storage Technology Association (OSTA) Universal Disk Format (UDF) Specification, "OSTA UDF Compliant Domain", Version 2.00.

       ISO 9660:1988

	      Information Processing-Volume and file structure of CD-ROM for information interchange.

       Refer to the standards(5) reference page for more information about industry standards and associated tags.

DESCRIPTION

       The Digital Versatile Disk File System enables the reading of disks formatted in the Universal Disk Format (UDF).

       User  data sectors in a DVD-ROM can contain any type of data in any format.  However, for Tru64 UNIX support through the dvdfs file system,
       the OSTA UDF file format standard is mandatory.	The file system limit for DVDFS is int cfs_cdfs_mount_limit = 512.

       Additionally, DVD-ROM standards require that the logical sector size and the logical block (the user data block) size be 2048 bytes.

       DVD support in the UDF specification includes these levels: DVD-ROM (Read-only):

	      Supports reading of UDF-formated files from DVD-ROM disks.  DVD-R (Write-once):

	      Supports the reading of UDF-formated files from a DVD-ROM disk and supports writing, one time, UDF-formated  files  onto	a  DVD-ROM
	      disk.  DVD-RAM (Rewritable):

	      Supports reading and writing of UDF-formated files from and to a DVD-ROM disk.

       At this time, the Tru64 UNIX dvdfs file system supports only the DVD-ROM (Read-only) format.

       See  mount(8) for information about mounting and unmounting a dvdfs file system, fstab(4) for information about including dvdfs file system
       in a system's /etc/fstab file, and disklabel(8) for information about labeling DVD-ROM disks.

RELATED INFORMATION

       Commands: disklabel(8), mount(8)

       Files: cdfs(4) fstab(4) delim off

																	  dvdfs(4)