optimize for file reading


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting optimize for file reading
# 1  
Old 04-07-2009
Tools optimize for file reading

HI,
I am new to unix . I don't know it is proper place to put my doubt or not .In my requirement , I am reading a file data and validating each fields and one record having 11 fields and each field having specific length.so when i read the file and validation the 11 fields. It taking 3-4 min per 1000 records. i need to optimize the time . Below are the code snap shot . Please help me regarding this.

Thanks in Advance .


code : while read line
do
FIELD=`echo $line| awk -F, '{print NF }'`
FIRST=`echo $line | awk -F, '{print $1 }' | wc -c`
SECOND=`echo $line | awk -F, '{print $2 }' | wc -c`
THIRD=`echo $line | awk -F, '{print $3 }' | wc -c`
FOURTH=`echo $line | awk -F, '{print $4 }' | wc -c`
FIFTH=`echo $line | awk -F, '{print $5 }' | wc -c`
SIXTH=`echo $line | awk -F, '{print $6 }' | wc -c`
SEVENTH=`echo $line | awk -F, '{print $7 }' | wc -c`
EIGHT=`echo $line | awk -F, '{print $8 }' | wc -c`
NINTH=`echo $line | awk -F, '{print $9 }' | wc -c`
TENTH=`echo $line | awk -F, '{print $10 }' | wc -c`
LASTFILED=`echo $line | awk -F, '{print $11 }'`
if [ ${FIELD} -ne 11 ]
then
echo " ERROR:- Invalid number of seperator at Record no : ${COUNT} " >> ${LOG_FILE}
fi
done < (Input file name)


input data:
xl,35077,CTC (13),13,BBSR,,,,,CDM
xA,85013,CTC (13),13,BBSR1,,,,,CDO
# 2  
Old 04-07-2009
see if you wanna check the no of fields and the lenghth of other fileds
use length(string) function of awk for example
Code:
awk -F"," 'NF==1&&length($1)>something&&length($2)<something||length($3)... so on

if you give your exact req i am sure it can be done with single awk command i mean a oneliner

Last edited by vidyadhar85; 04-07-2009 at 03:41 AM..
# 3  
Old 04-08-2009
Tools

Thanks vidyadhar85 for your reply,
Actually my requirement is , It will check the each fields length of records and if it correct , then it will check the 3 field values ,If the third field values is blank , then it replace with 7th field value...and then create a output file with field 1,2,3 and 4th.

Also can you suggest me the shell script is better than the perl script or not , Because this requirement for optimization ..i need to check the time taken for execution .


Please Help me regarding this.

Last edited by julirani; 04-08-2009 at 03:27 AM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Help Optimize the Script Further

Hi All, I have written a new script to check for DB space and size of dump log file before it can be imported into a Oracle DB. I'm relatively new to shell scripting. Please help me optimize this script further. (0 Replies)
Discussion started by: narayanv
0 Replies

2. Shell Programming and Scripting

Looking to optimize code

Hi guys, I feel a bit comfortable now doing bash scripting but I am worried that the way I do it is not optimized and I can do much better as to how I code. e.g. I have a whole line in a file from which I want to extract some values. Right now what I am doing is : STATE=`cat... (5 Replies)
Discussion started by: Junaid Subhani
5 Replies

3. Shell Programming and Scripting

Optimize my mv script

Hello, I'm wondering if there is a quicker way of doing this. Here is my mv script. d=/conversion/program/out cd $d ls $d > /home/tempuser/$$tmp while read line ; do a=`echo $line|cut -c1-5|sed "s/_//g"` b=`echo $line|cut -c16-21` if ;then mkdir... (13 Replies)
Discussion started by: whegra
13 Replies

4. Shell Programming and Scripting

Optimize awk command

WARNING=${1} CRITICAL=${2} echo ${OUTPUT} | gawk -F'' ' { V = $2 R = $0 } END { for ( i = 1; i <= n; i++) { if((V > 0) && (V < V)) print R, ((V - V) / V) * 100 else if ((V > V) && (V > 0)) ... (6 Replies)
Discussion started by: SkySmart
6 Replies

5. Shell Programming and Scripting

Optimize the nested IF

Hi, I have to assign a value for a varaiable based on a Input. I have written the below code: if then nf=65 elif then nf=46 elif then nf=164 elif then nf=545 elif then nf=56 elif then (3 Replies)
Discussion started by: machomaddy
3 Replies

6. Shell Programming and Scripting

pl help me to Optimize the given code

Pl help to me to write the below code in a simple way ... i suupose to use this code 3 to 4 places in my makefile(gnu) .. **************************************** @for i in $(LIST_A); do \ for j in $(LIST_B); do\ if ;then\ echo "Need to sign"\ echo "List A = $$i , List B =$$j"\ ... (2 Replies)
Discussion started by: pk_arun
2 Replies

7. Shell Programming and Scripting

Optimize shell code

#!/usr/bin/perl use strict; use warnings; use Date::Manip; my $date_converted = UnixDate(ParseDate("3 days ago"),"%e/%h/%Y"); open FILE,">$ARGV"; while(<DATA>){ my @tab_delimited_array = split(/\t/,$_); $tab_delimited_array =~ s/^\ =~ s/^\-//; my $converted_date =... (2 Replies)
Discussion started by: sandy1028
2 Replies

8. Shell Programming and Scripting

can we optimize this command

can we optimize this command ? sed 's#AAAA##g' /study/i.txt | sed '1,2d' | tr -d '\n\' > /study/i1.txt; as here i am using two files ...its overhead..can we optimise to use only 1 file sed 's#AAAA##g' /study/i.txt | sed '1,2d' | tr -d '\n\' > /study/i.txt; keeping them same but it... (9 Replies)
Discussion started by: crackthehit007
9 Replies

9. AIX

How to optimize our tape backups ?

Hi, I am currently looking at how we can optimize and speed up our backups here. I am just a beginner operator and our system admin hardly knows anything (long term interim). There is this particular TAR backup of DB backups that for a 10.5Gb amount of files, it takes 5 hours to do the backup on... (6 Replies)
Discussion started by: Browser_ice
6 Replies

10. Shell Programming and Scripting

optimize the script

Hi, I have this following script below. Its searching a log file for 2 string and if found then write the strings to success.txt and If not found write strings to failed.txt . if one found and not other...then write found to success.txt and not found to failed.txt. I want to optimize this... (3 Replies)
Discussion started by: amitrajvarma
3 Replies
Login or Register to Ask a Question