File - reading - Performance improvement Post: 302197896

Sponsored Content

Top Forums Programming File - reading - Performance improvement Post 302197896 by dhanamurthy on Wednesday 21st of May 2008 11:31:15 PM

05-22-2008

Registered User

File - reading - Performance improvement

Hi All
I am reading a huge file of size 2GB atleast. I am reading each line and cutting certain columns and writing it to another file.

Here is the logic.

Code:

int main()
{
     
	  string u_line;
	  string Char_List;
	  string u_file;
	  int line_pos;
	  string temp_form_u_file;
	  ofstream temp_u_file;
	  u_file=getenv("u_file");
	  temp_form_u_file=getenv("DATA_DIR");
	  ifstream U_File;
	  temp_u_file.open(temp_form_u_file.c_str(),ios::app);
      
	  
	  if (temp_u_file.fail()) {
      cout << "Unable to open file "<<temp_form_u_file<<" for writing" << endl;
      exit(1);
      }
       
	    
      U_File.open(u_file.c_str());
      if (U_File.fail())
      {
         cout<<"File "<<u_file<<" unable to open for reading\n";
         cout<<"dart_report job failed\n";
	     exit(3);
      } 
    
      while (! U_File.eof() )
      {

		 line_pos=72;
		 u_line.erase();
		 getline (U_File,u_line);

		 if ( ! u_line.empty())  {
         while (line_pos< u_line.length())
	     {
           
	       if (u_line.substr(line_pos,2)!= "  ")
	       {
  
				Char_List=u_line.substr(line_pos,41);
				Char_List.append(u_line.substr(16,4));
				Char_List.append("\n");
				temp_u_file<< Char_List;
          
           } 
                line_pos=line_pos+41;

	    }

      }
    }  
}

When i run this program it takes 2.5 to 3 hours to read the 2 GB file. I am trying to reduce the time taken to reading. Is there any way i can reduce the processing time of the program.

Kindly let me know. If i can use Shell Script it is also okay. But i feel 'C' will be faster than Shell Scripting.

Please give me your suggestions.

Regards
Dhana

Last edited by Yogesh Sawant; 05-22-2008 at 02:35 AM.. Reason: added code tags

dhanamurthy

View Public Profile for dhanamurthy

Find all posts by dhanamurthy

7 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Script ready but might need some improvement.

Hi All, I have written a script which does some editing in the files, based on user input.This might not be the most elegant way of doing it and there would be many improvements needed. Please go through it and let me know how it could be improved. Suggestions are welcome!! Thanks!...

2. Shell Programming and Scripting

Any improvement possible in this script

Hi! Thank you for the help yesterday This is the finished product There is one more thing I would like to do to it but I’m not to certain On how to proceed I would like to log all output to a log in order to Be able to roll back This script is meant to be used in repairing a...

3. Infrastructure Monitoring

Possible performance improvement (Bash and flat file)

Hello, I am pretty new to shell scripts and I recently wrote one that seems to do what it should but I am exploring the possibility of improving its performance and would appreciate some help. Here is what it does - Its meant to monitor a bunch of systems (reads in IPs one at a time from a flat...

4. UNIX for Advanced & Expert Users

linux os improvement

can anyone help to share the knowledge on linux os improvement? 1) os account - use window AD authentication, such as ldap, but how to set /etc/passwd, where to put user home? 2) user account activity - how to log os user activity share the idea and what tools can do that...thx

5. Shell Programming and Scripting

I need the improvement for my script

Hi All, Here is my script #! /bin/sh var1=some email id var2=some email id grep -i "FAILED FILE FORMAT VALIDATION" /opt >tmp2 diff tmp1 tmp2 | grep ">" >tmp3 if then cat tmp3 | mailx -s " Error Monitoring" $var2 else echo "Pattern NOt Found" | mailx -s " Error Monitoring" $var1...

6. UNIX for Dummies Questions & Answers

Improvement in shell script

Hi This is my Following code: #!/bin/sh echo "TOTAL_NO_OF_MAILS" read TOTAL_NO_OF_MAILS echo "TOTAL_NO_OF_TICKETS " read TOTAL_NO_OF_TICKETS echo "TICKETS_IN_QUEUE" read TICKETS_IN_QUEUE rm -rf `pwd`/Focus echo "Hi Team\nSTATS IN CLRS MAIL BOX\n\n==============================" >> Focus...

7. Shell Programming and Scripting

Performance improvement in grep

Below script is used to search numeric data from around 400 files in a folder. I have 300 such folders. Need help in performance improvement in the script. Below Script searches 20 such folders ( 300 files in each folder) simultaneously. This increases cpu utilization upto 90% What changes...

LEARN ABOUT LINUX

secure_getenv

GETENV(3)						     Linux Programmer's Manual							 GETENV(3)

NAME

       getenv - get an environment variable

SYNOPSIS

       #include <stdlib.h>

       char *getenv(const char *name);

DESCRIPTION

       The getenv() function searches the environment list to find the environment variable name, and returns a pointer to the corresponding value
       string.

RETURN VALUE

       The getenv() function returns a pointer to the value in the environment, or NULL if there is no match.

CONFORMING TO

       SVr4, POSIX.1-2001, 4.3BSD, C89, C99.

NOTES

       The strings in the environment list are of the form name=value.

       As typically implemented, getenv() returns a pointer to a string within the environment list.  The caller must take care not to modify this
       string, since that would change the environment of the process.

       The  implementation  of	getenv() is not required to be reentrant.  The string pointed to by the return value of getenv() may be statically
       allocated, and can be modified by a subsequent call to getenv(), putenv(3), setenv(3), or unsetenv(3).

SEE ALSO

       clearenv(3), putenv(3), setenv(3), unsetenv(3), environ(7)

COLOPHON

       This page is part of release 3.27 of the Linux man-pages project.  A description of the project, and information about reporting bugs,  can
       be found at http://www.kernel.org/doc/man-pages/.

GNU
								    2008-03-17								 GETENV(3)