04-07-2009
optimize for file reading
HI,
I am new to unix . I don't know it is proper place to put my doubt or not .In my requirement , I am reading a file data and validating each fields and one record having 11 fields and each field having specific length.so when i read the file and validation the 11 fields. It taking 3-4 min per 1000 records. i need to optimize the time . Below are the code snap shot . Please help me regarding this.
Thanks in Advance .
code : while read line
do
FIELD=`echo $line| awk -F, '{print NF }'`
FIRST=`echo $line | awk -F, '{print $1 }' | wc -c`
SECOND=`echo $line | awk -F, '{print $2 }' | wc -c`
THIRD=`echo $line | awk -F, '{print $3 }' | wc -c`
FOURTH=`echo $line | awk -F, '{print $4 }' | wc -c`
FIFTH=`echo $line | awk -F, '{print $5 }' | wc -c`
SIXTH=`echo $line | awk -F, '{print $6 }' | wc -c`
SEVENTH=`echo $line | awk -F, '{print $7 }' | wc -c`
EIGHT=`echo $line | awk -F, '{print $8 }' | wc -c`
NINTH=`echo $line | awk -F, '{print $9 }' | wc -c`
TENTH=`echo $line | awk -F, '{print $10 }' | wc -c`
LASTFILED=`echo $line | awk -F, '{print $11 }'`
if [ ${FIELD} -ne 11 ]
then
echo " ERROR:- Invalid number of seperator at Record no : ${COUNT} " >> ${LOG_FILE}
fi
done < (Input file name)
input data:
xl,35077,CTC (13),13,BBSR,,,,,CDM
xA,85013,CTC (13),13,BBSR1,,,,,CDO
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Hi,
I have this following script below. Its searching a log file for 2 string and if found then write the strings to success.txt and If not found write strings to failed.txt . if one found and not other...then write found to success.txt and not found to failed.txt.
I want to optimize this... (3 Replies)
Discussion started by: amitrajvarma
3 Replies
2. AIX
Hi, I am currently looking at how we can optimize and speed up our backups here. I am just a beginner operator and our system admin hardly knows anything (long term interim).
There is this particular TAR backup of DB backups that for a 10.5Gb amount of files, it takes 5 hours to do the backup on... (6 Replies)
Discussion started by: Browser_ice
6 Replies
3. Shell Programming and Scripting
can we optimize this command ?
sed 's#AAAA##g' /study/i.txt | sed '1,2d' | tr -d '\n\' > /study/i1.txt;
as here i am using two files ...its overhead..can we optimise to use only 1 file
sed 's#AAAA##g' /study/i.txt | sed '1,2d' | tr -d '\n\' > /study/i.txt;
keeping them same but it... (9 Replies)
Discussion started by: crackthehit007
9 Replies
4. Shell Programming and Scripting
#!/usr/bin/perl
use strict;
use warnings;
use Date::Manip;
my $date_converted = UnixDate(ParseDate("3 days ago"),"%e/%h/%Y");
open FILE,">$ARGV";
while(<DATA>){
my @tab_delimited_array = split(/\t/,$_);
$tab_delimited_array =~ s/^\ =~ s/^\-//;
my $converted_date =... (2 Replies)
Discussion started by: sandy1028
2 Replies
5. Shell Programming and Scripting
Pl help to me to write the below code in a simple way ...
i suupose to use this code 3 to 4 places in my makefile(gnu) ..
****************************************
@for i in $(LIST_A); do \
for j in $(LIST_B); do\
if ;then\
echo "Need to sign"\
echo "List A = $$i , List B =$$j"\
... (2 Replies)
Discussion started by: pk_arun
2 Replies
6. Shell Programming and Scripting
Hi,
I have to assign a value for a varaiable based on a Input. I have written the below code:
if
then
nf=65
elif
then
nf=46
elif
then
nf=164
elif
then
nf=545
elif
then
nf=56
elif
then (3 Replies)
Discussion started by: machomaddy
3 Replies
7. Shell Programming and Scripting
WARNING=${1}
CRITICAL=${2}
echo ${OUTPUT} | gawk -F'' ' {
V = $2
R = $0
} END {
for ( i = 1; i <= n; i++) {
if((V > 0) && (V < V))
print R, ((V - V) / V) * 100
else if ((V > V) && (V > 0))
... (6 Replies)
Discussion started by: SkySmart
6 Replies
8. Shell Programming and Scripting
Hello,
I'm wondering if there is a quicker way of doing this.
Here is my mv script.
d=/conversion/program/out
cd $d
ls $d > /home/tempuser/$$tmp
while read line ; do
a=`echo $line|cut -c1-5|sed "s/_//g"`
b=`echo $line|cut -c16-21`
if ;then mkdir... (13 Replies)
Discussion started by: whegra
13 Replies
9. Shell Programming and Scripting
Hi guys,
I feel a bit comfortable now doing bash scripting but I am worried that the way I do it is not optimized and I can do much better as to how I code.
e.g.
I have a whole line in a file from which I want to extract some values.
Right now what I am doing is :
STATE=`cat... (5 Replies)
Discussion started by: Junaid Subhani
5 Replies
10. Shell Programming and Scripting
Hi All,
I have written a new script to check for DB space and size of dump log file before it can be imported into a Oracle DB.
I'm relatively new to shell scripting.
Please help me optimize this script further. (0 Replies)
Discussion started by: narayanv
0 Replies
IGAWK(1) Utility Commands IGAWK(1)
NAME
igawk - gawk with include files
SYNOPSIS
igawk [ all gawk options ] -f program-file [ -- ] file ...
igawk [ all gawk options ] [ -- ] program-text file ...
DESCRIPTION
Igawk is a simple shell script that adds the ability to have ``include files'' to gawk(1).
AWK programs for igawk are the same as for gawk, except that, in addition, you may have lines like
@include getopt.awk
in your program to include the file getopt.awk from either the current directory or one of the other directories in the search path.
OPTIONS
See gawk(1) for a full description of the AWK language and the options that gawk supports.
EXAMPLES
cat << EOF > test.awk
@include getopt.awk
BEGIN {
while (getopt(ARGC, ARGV, "am:q") != -1)
...
}
EOF
igawk -f test.awk
SEE ALSO
gawk(1)
Effective AWK Programming, Edition 1.0, published by the Free Software Foundation, 1995.
AUTHOR
Arnold Robbins (arnold@skeeve.com).
Free Software Foundation Nov 3 1999 IGAWK(1)