Sponsored Content
Top Forums Programming File - reading - Performance improvement Post 302197896 by dhanamurthy on Wednesday 21st of May 2008 11:31:15 PM
Old 05-22-2008
File - reading - Performance improvement

Hi All
I am reading a huge file of size 2GB atleast. I am reading each line and cutting certain columns and writing it to another file.

Here is the logic.

Code:
int main()
{
     
	  string u_line;
	  string Char_List;
	  string u_file;
	  int line_pos;
	  string temp_form_u_file;
	  ofstream temp_u_file;
	  u_file=getenv("u_file");
	  temp_form_u_file=getenv("DATA_DIR");
	  ifstream U_File;
	  temp_u_file.open(temp_form_u_file.c_str(),ios::app);
      
	  
	  if (temp_u_file.fail()) {
      cout << "Unable to open file "<<temp_form_u_file<<" for writing" << endl;
      exit(1);
      }
       
	    
      U_File.open(u_file.c_str());
      if (U_File.fail())
      {
         cout<<"File "<<u_file<<" unable to open for reading\n";
         cout<<"dart_report job failed\n";
	     exit(3);
      } 
    
      while (! U_File.eof() )
      {

		 line_pos=72;
		 u_line.erase();
		 getline (U_File,u_line);

		 if ( ! u_line.empty())  {
         while (line_pos< u_line.length())
	     {
           
	       if (u_line.substr(line_pos,2)!= "  ")
	       {
  
				Char_List=u_line.substr(line_pos,41);
				Char_List.append(u_line.substr(16,4));
				Char_List.append("\n");
				temp_u_file<< Char_List;
          
           } 
                line_pos=line_pos+41;

	    }

      }
    }  
}

When i run this program it takes 2.5 to 3 hours to read the 2 GB file. I am trying to reduce the time taken to reading. Is there any way i can reduce the processing time of the program.

Kindly let me know. If i can use Shell Script it is also okay. But i feel 'C' will be faster than Shell Scripting.

Please give me your suggestions.

Regards
Dhana

Last edited by Yogesh Sawant; 05-22-2008 at 02:35 AM.. Reason: added code tags
 

7 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Script ready but might need some improvement.

Hi All, I have written a script which does some editing in the files, based on user input.This might not be the most elegant way of doing it and there would be many improvements needed. Please go through it and let me know how it could be improved. Suggestions are welcome!! Thanks!... (2 Replies)
Discussion started by: nua7
2 Replies

2. Shell Programming and Scripting

Any improvement possible in this script

Hi! Thank you for the help yesterday This is the finished product There is one more thing I would like to do to it but I’m not to certain On how to proceed I would like to log all output to a log in order to Be able to roll back This script is meant to be used in repairing a... (4 Replies)
Discussion started by: Ex-Capsa
4 Replies

3. Infrastructure Monitoring

Possible performance improvement (Bash and flat file)

Hello, I am pretty new to shell scripts and I recently wrote one that seems to do what it should but I am exploring the possibility of improving its performance and would appreciate some help. Here is what it does - Its meant to monitor a bunch of systems (reads in IPs one at a time from a flat... (9 Replies)
Discussion started by: prafulnama
9 Replies

4. UNIX for Advanced & Expert Users

linux os improvement

can anyone help to share the knowledge on linux os improvement? 1) os account - use window AD authentication, such as ldap, but how to set /etc/passwd, where to put user home? 2) user account activity - how to log os user activity share the idea and what tools can do that...thx (5 Replies)
Discussion started by: goodbid
5 Replies

5. Shell Programming and Scripting

I need the improvement for my script

Hi All, Here is my script #! /bin/sh var1=some email id var2=some email id grep -i "FAILED FILE FORMAT VALIDATION" /opt >tmp2 diff tmp1 tmp2 | grep ">" >tmp3 if then cat tmp3 | mailx -s " Error Monitoring" $var2 else echo "Pattern NOt Found" | mailx -s " Error Monitoring" $var1... (1 Reply)
Discussion started by: Gopalak
1 Replies

6. UNIX for Dummies Questions & Answers

Improvement in shell script

Hi This is my Following code: #!/bin/sh echo "TOTAL_NO_OF_MAILS" read TOTAL_NO_OF_MAILS echo "TOTAL_NO_OF_TICKETS " read TOTAL_NO_OF_TICKETS echo "TICKETS_IN_QUEUE" read TICKETS_IN_QUEUE rm -rf `pwd`/Focus echo "Hi Team\nSTATS IN CLRS MAIL BOX\n\n==============================" >> Focus... (11 Replies)
Discussion started by: wasim999
11 Replies

7. Shell Programming and Scripting

Performance improvement in grep

Below script is used to search numeric data from around 400 files in a folder. I have 300 such folders. Need help in performance improvement in the script. Below Script searches 20 such folders ( 300 files in each folder) simultaneously. This increases cpu utilization upto 90% What changes... (3 Replies)
Discussion started by: vegasluxor
3 Replies
Text::Wrap(3pm) 					 Perl Programmers Reference Guide					   Text::Wrap(3pm)

NAME
Text::Wrap - line wrapping to form simple paragraphs SYNOPSIS
Example 1 use Text::Wrap $initial_tab = " "; # Tab before first line $subsequent_tab = ""; # All other lines flush left print wrap($initial_tab, $subsequent_tab, @text); print fill($initial_tab, $subsequent_tab, @text); @lines = wrap($initial_tab, $subsequent_tab, @text); @paragraphs = fill($initial_tab, $subsequent_tab, @text); Example 2 use Text::Wrap qw(wrap $columns $huge); $columns = 132; # Wrap at 132 characters $huge = 'die'; $huge = 'wrap'; $huge = 'overflow'; Example 3 use Text::Wrap $Text::Wrap::columns = 72; print wrap('', '', @text); DESCRIPTION
"Text::Wrap::wrap()" is a very simple paragraph formatter. It formats a single paragraph at a time by breaking lines at word boundries. Indentation is controlled for the first line ($initial_tab) and all subsequent lines ($subsequent_tab) independently. Please note: $ini- tial_tab and $subsequent_tab are the literal strings that will be used: it is unlikley you would want to pass in a number. Text::Wrap::fill() is a simple multi-paragraph formatter. It formats each paragraph separately and then joins them together when it's done. It will destory any whitespace in the original text. It breaks text into paragraphs by looking for whitespace after a newline. In other respects it acts like wrap(). OVERRIDES
"Text::Wrap::wrap()" has a number of variables that control its behavior. Because other modules might be using "Text::Wrap::wrap()" it is suggested that you leave these variables alone! If you can't do that, then use "local($Text::Wrap::VARIABLE) = YOURVALUE" when you change the values so that the original value is restored. This "local()" trick will not work if you import the variable into your own namespace. Lines are wrapped at $Text::Wrap::columns columns. $Text::Wrap::columns should be set to the full width of your output device. In fact, every resulting line will have length of no more than "$columns - 1". It is possible to control which characters terminate words by modifying $Text::Wrap::break. Set this to a string such as '[s:]' (to break before spaces or colons) or a pre-compiled regexp such as "qr/[s']/" (to break before spaces or apostrophes). The default is simply 's'; that is, words are terminated by spaces. (This means, among other things, that trailing punctuation such as full stops or commas stay with the word they are "attached" to.) Beginner note: In example 2, above $columns is imported into the local namespace, and set locally. In example 3, $Text::Wrap::columns is set in its own namespace without importing it. "Text::Wrap::wrap()" starts its work by expanding all the tabs in its input into spaces. The last thing it does it to turn spaces back into tabs. If you do not want tabs in your results, set $Text::Wrap::unexapand to a false value. Likewise if you do not want to use 8-character tabstops, set $Text::Wrap::tabstop to the number of characters you do want for your tabstops. If you want to separate your lines with something other than " " then set $Text::Wrap::seporator to your preference. When words that are longer than $columns are encountered, they are broken up. "wrap()" adds a " " at column $columns. This behavior can be overridden by setting $huge to 'die' or to 'overflow'. When set to 'die', large words will cause "die()" to be called. When set to 'overflow', large words will be left intact. Historical notes: 'die' used to be the default value of $huge. Now, 'wrap' is the default value. EXAMPLE
print wrap(" ","","This is a bit of text that forms a normal book-style paragraph"); AUTHOR
David Muir Sharnoff <muir@idiom.com> with help from Tim Pierce and many many others. perl v5.8.0 2002-06-01 Text::Wrap(3pm)
All times are GMT -4. The time now is 03:41 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy