Hi All
I am reading a huge file of size 2GB atleast. I am reading each line and cutting certain columns and writing it to another file.
Here is the logic.
When i run this program it takes 2.5 to 3 hours to read the 2 GB file. I am trying to reduce the time taken to reading. Is there any way i can reduce the processing time of the program.
Kindly let me know. If i can use Shell Script it is also okay. But i feel 'C' will be faster than Shell Scripting.
Please give me your suggestions.
Regards
Dhana
Last edited by Yogesh Sawant; 05-22-2008 at 02:35 AM..
Reason: added code tags
I believe shell script should be faster. With C/C++, there is a lot of copying of data to/from kernel, which makes C/C++ programs slow. To make C/C++ programs faster, you may use multithreading also.
Hi,
I would suggest to use fread that is read data in bulk say thousands at a time and then manipulate it.You will surely get the performance improvement.
I think, read and write are the most low level system calls. All the other function like fread and fwrite again uses some low level function to do their work.
I think, using read for reading a chunk of data can improve the performance since their is not much overhead involved.
HI
read(fd, buffer, n_to_read)
I am trying to use the above call, but i will not be able to read the entire line as i will not now the length of the line before hand.
This part is little tricky to handle.
If you have any idea please let me know.
Hello!
What you can try out is: have a huge circular buffer for example say around 6144 (6KB) , you can experiment with the size!!
What i mean by circular is have two pointers, start_ptr and processed_ptr.
1) Have start_ptr and processed_ptr as global
2) You must take care of rollover of processed_ptr for every read
Below script is used to search numeric data from around 400 files in a folder. I have 300 such folders. Need help in performance improvement in the script.
Below Script searches 20 such folders ( 300 files in each folder) simultaneously. This increases cpu utilization upto 90% What changes... (3 Replies)
Hi This is my Following code:
#!/bin/sh
echo "TOTAL_NO_OF_MAILS"
read TOTAL_NO_OF_MAILS
echo "TOTAL_NO_OF_TICKETS "
read TOTAL_NO_OF_TICKETS
echo "TICKETS_IN_QUEUE"
read TICKETS_IN_QUEUE
rm -rf `pwd`/Focus
echo "Hi Team\nSTATS IN CLRS MAIL BOX\n\n==============================" >> Focus... (11 Replies)
Hi All,
Here is my script
#! /bin/sh
var1=some email id
var2=some email id
grep -i "FAILED FILE FORMAT VALIDATION" /opt >tmp2
diff tmp1 tmp2 | grep ">" >tmp3
if
then
cat tmp3 | mailx -s " Error Monitoring" $var2
else
echo "Pattern NOt Found" | mailx -s " Error Monitoring" $var1... (1 Reply)
can anyone help to share the knowledge on linux os improvement?
1) os account
- use window AD authentication, such as ldap, but how to set /etc/passwd, where to put user home?
2) user account activity
- how to log os user activity
share the idea and what tools can do that...thx (5 Replies)
Hello,
I am pretty new to shell scripts and I recently wrote one that seems to do what it should but I am exploring the possibility of improving its performance and would appreciate some help. Here is what it does - Its meant to monitor a bunch of systems (reads in IPs one at a time from a flat... (9 Replies)
Hi!
Thank you for the help yesterday
This is the finished product
There is one more thing I would like to do to it but I’m not to certain
On how to proceed I would like to log all output to a log in order to
Be able to roll back
This script is meant to be used in repairing a... (4 Replies)
Hi All,
I have written a script which does some editing in the files, based on user input.This might not be the most elegant way of doing it and there would be many improvements needed.
Please go through it and let me know how it could be improved.
Suggestions are welcome!!
Thanks!... (2 Replies)