Reading a file line by line and processing for each line


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Reading a file line by line and processing for each line
# 1  
Old 02-24-2009
Reading a file line by line and processing for each line

Hi,

I am a beginner in shell scripting. I have written the following script, which is supposed to process the while loop for each line in the sid_home.txt file. But I'm getting the 'end of file' unexpected for the last line. The file sid_home.txt gets generated as expected, but the script doesn't execute the instructions in the while loop:

#!/bin/ksh
cd /home/dba/utils/scripts/monitor
sed '/^#.*$/d' /var/opt/oracle/oratab | sed '/^[Cc]lient.*$/d' | sed '/^CRS.*$/d' | awk -F: '{print$1, $2;}' > sid_home.txt
{ while read LINE;do
ORACLE_SID=`awk '{print $1;}' $LINE`; export ORACLE_SID
ORACLE_HOME=`awk '{print $2;}' $LINE`; export ORACLE_HOME
DATE_TIME=`date +%m%d%y%a_%H%M`; export DATE_TIME
TSPALERT_LOG=tspalert_${ORACLE_SID}_${DATE_TIME}.log;export TSPALERTLOG
ORACLE_BASE=${ORACLE_HOME}; export ORACLE_BASE
ORACLE_BIN=${ORACLE_HOME}/bin; export ORACLE_BIN
LD_LIBRARY_PATH=/usr/openv/netbackup/bin:/usr/dt/lib:/usr/openwin/lib:/usr/local/lib:${ORACLE_HOME}/lib32
LD_LIBRARY_PATH_64=${ORACLE_HOME}/lib
ORACLE_DOC=${ORACLE_HOME}/orainst/doc
${ORACLE_HOME}/bin/sqlplus -s /nolog << EOF
set heading off
spool $TSPALERT_LOG;
select name from v$database;
select to_char(sysdate,'DD-MM-YYYY HH24:MI:SS') from dual;
select tablespace_name,sum(bytes)/1048576 MBUsed, sum(maxbytes)/1048576 TotalMB, sum(bytes)/sum(maxbytes)*100 PercentUsed
from dba_data_files
group by tablespace_name
having sum(bytes)/sum(maxbytes)*100 >= 80;
spool off
EXIT;
EOF
if [ -f ${TSPALERT_LOG} ]; then
echo "**ERROR**: TABLESPACE NEARLY FULL OR NOT ENOUGH ROOM FOR THE NEXT EXTENT IN ${ORACLE_SID}. Check log ${TSPALERT_LOG} " > /dev/null
mv ${TSPALERT_LOG} ${TSPALERT_LOG}.old > /dev/null
fi
done } < sid_home.txt
# 2  
Old 02-27-2009
Beginner? This is one of the better ksh codes that I have seen. And believe me - I have seen plenty od it.
Some hints that might help you:
- Try enclosing variables with quotes: variable="${var}" (the way you are using could work fine but under some circumstances you can get into problems - an example is when ${var} is equal to 'some text with $trange characters').
- I'm not sure if there is a need to enclose that function within brackets - if I remember correct you should be able to "print" the file into the loop without brackets
- Send this sid_home.txt file and make sure that the problem is not related to end of line characters (unix/max/windows format) - if you send it then I could take a look onto it
- Use tabulation - this code you have sent is not very readable
- You might try the alternative syntax as well (might be worth giving a try): cat "${file}" | while $(read LINE);do ... done
- Execute your code with "set -x;set -v" and check the output. You might try some debuggers as well.
# 3  
Old 02-27-2009
Quote:
Originally Posted by adderek
Beginner? This is one of the better ksh codes that I have seen. And believe me - I have seen plenty od it.

You must be joking! It's extremely inefficient and poorly written code.
Quote:
Some hints that might help you:
- Try enclosing variables with quotes: variable="${var}" (the way you are using could work fine but under some circumstances you can get into problems - an example is when ${var} is equal to 'some text with $trange characters').

It is never necessary to quote the assignment of one variable to another; it doesn't matter what the variable contains. It only needs to be quoted when there is a literal space in the assignment:

Code:
x='the quick brown fox' ## Quotes necessary for literal spaces
var=$x  ## No quotes necessary

Quote:
- I'm not sure if there is a need to enclose that function within brackets - if I remember correct you should be able to "print" the file into the loop without brackets

There's no need, but it doesn't hurt.
Quote:
- Send this sid_home.txt file and make sure that the problem is not related to end of line characters (unix/max/windows format) - if you send it then I could take a look onto it

Why? All that's needed is to look at the file with od -c and look for \r characters.

However, it would help to post a few lines of the file so we can see what the input to the loop contains.

It should contain a list of filenames, since $LINE is being used as a filename argument to the (probably unnecessary) awk commands.
Quote:
- Use tabulation - this code you have sent is not very readable

I think you mean indentation. That's always good, and more importantly, code posted to this forum should always be enclosed in [code] tags.
Quote:
- You might try the alternative syntax as well (might be worth giving a try): cat "${file}" |

Why would you use cat? It cannot make any difference.
Quote:
while $(read LINE);do ... done

What do you think while $(read LINE) does?

The assignment to LINE only lasts for the duration of the subshell inside $( ... ); it will not be available to the rest of the script.

In other words, it does nothing.
Quote:
- Execute your code with "set -x;set -v" and check the output. You might try some debuggers as well.
# 4  
Old 02-27-2009
Reply

Quote:
Originally Posted by cfajohnson

You must be joking! It's extremely inefficient and poorly written code.
I agree that it might be better. You can always do things better - however the price is often too much to optimize. But it is far better than most of the code that I have seen (like megabytes of ksh code that used global variables shared between thousands of files creating/closing a new connection to oracle every 10 seconds). In fact in my company there is a tool that fails whenever you enter non alphanumeric character in it - and the tool is used in hundreds of divisions (one can even implant own commands to execute with ROOT privileges.... standard code created by "professionals"). I guess that this code is far better than such examples - thus it does not seen like a result of "beginner's work".
Quote:
Originally Posted by cfajohnson

It is never necessary to quote the assignment of one variable to another; it doesn't matter what the variable contains. It only needs to be quoted when there is a literal space in the assignment:
Code:
x='the quick brown fox' ## Quotes necessary for literal spaces
var=$x  ## No quotes necessary

Code:
LD_LIBRARY_PATH_64=${ORACLE_HOME}/lib

I see that this line should be quoted. Couple of others as well.
Would work with modern shells but would fail with many others.

Quote:
Originally Posted by cfajohnson

I think you mean indentation. That's always good, and more importantly, code posted to this forum should always be enclosed in [code] tags.
Correct.

Quote:
Originally Posted by cfajohnson

Why would you use cat? It cannot make any difference.
You should always try alternative solutions if you cannot find the bug and the alternative solution is simple/quick to implement. After your alternative code works you might trace the source by comparing the two versions.

Quote:
Originally Posted by cfajohnson

What do you think while $(read LINE) does?

The assignment to LINE only lasts for the duration of the subshell inside $( ... ); it will not be available to the rest of the script.

In other words, it does nothing.
I haven't worked in ksh for over a year. However I tend to use $() instead of ``. Sorry for inconvenience if the $(read LINE) would not return LINE to the parent.
In that case the line written by me should be:
Quote:
- You might try the alternative syntax as well (might be worth giving a try): cat "${file}" | while read LINE;do ... done
# 5  
Old 02-27-2009
Quote:
Originally Posted by adderek
Code:
LD_LIBRARY_PATH_64=${ORACLE_HOME}/lib

I see that this line should be quoted. Couple of others as well.
Would work with modern shells but would fail with many others.

That line does not need quotes in any Bourne-type shell.
Quote:
I haven't worked in ksh for over a year. However I tend to use $() instead of ``.

Backticks would be just as wrong as $( ... ), and for exactly the same reason.
# 6  
Old 02-28-2009
Well.... then I have been using the wrong bourne-type shells. Was I?

I guess that instead of helping sagarparadkar this discussion has now changed the topic.
Sagarparadkar, could you provide some example data file to us? Could you verify that the End Of Line characters are correct fr your script and input file?
# 7  
Old 03-02-2009
Hi Derek and Johnson,

Thanks a lot for your inputs.

I tried playing around my code, and finally came up with the following version of code, which I found working. I am sure this version also can be optimized a lot further. The script gets the ORACLE_SID and ORACLE_HOME from oratab file to sid_home.txt file, both separated by a space character. I could have directly read the oratab file, but I didn't want to have trial and error of my script directly with that file.

cd /home/dba/utils/scripts/monitor
sed '/^#.*$/d' /var/opt/oracle/oratab | sed '/^[Cc]lient.*$/d' | sed '/^CRS.*$/d' | awk -F: '{print$1, $2;}' > sid_home.txt

while read LINE
do
ORACLE_SID=`echo ${LINE}|cut -d" " -f1`; export ORACLE_SID
ORACLE_HOME=`echo ${LINE}|cut -d" " -f2`; export ORACLE_HOME
DATE_TIME=`date +%m%d%y%a_%H%M`; export DATE_TIME
TSPALERT_LOG=tspalert_${ORACLE_SID}_${DATE_TIME}.log;export TSPALERTLOG
ORACLE_BASE=${ORACLE_HOME}; export ORACLE_BASE
ORACLE_BIN=${ORACLE_HOME}/bin; export ORACLE_BIN
LD_LIBRARY_PATH=/usr/openv/netbackup/bin:/usr/dt/lib:/usr/openwin/lib:/usr/local/lib:${ORACLE_HOME}/lib32
LD_LIBRARY_PATH_64=${ORACLE_HOME}/lib
ORACLE_DOC=${ORACLE_HOME}/orainst/doc
${ORACLE_HOME}/bin/sqlplus -s /nolog <<!eof 2>&1
connect / as sysdba;
set heading off
spool ${TSPALERT_LOG};
SELECT NAME FROM V\$DATABASE;
SELECT TO_CHAR(SYSDATE,'DD-MM-YYYY HH24:MI:SS') FROM DUAL;
SELECT TABLESPACE_NAME,SUM(BYTES)/1048576 MBUSED, SUM(MAXBYTES)/1048576 TOTALMB, SUM(BYTES)/SUM(MAXBYTES)*100 PERCENTUSED
FROM DBA_DATA_FILES GROUP BY TABLESPACE_NAME
HAVING SUM(BYTES)/SUM(MAXBYTES)*100 >= 80;
spool off
EXIT
!eof
done < sid_home.txt
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Reading a file line by line and print required lines based on pattern

Hi All, i want to write a shell script read below file line by line and want to exclude the lines which contains empty value for MOUNTPOINT field. i am using centos 7 Operating system. want to read below file. # cat /tmp/d5 NAME="/dev/sda" TYPE="disk" SIZE="60G" OWNER="root"... (4 Replies)
Discussion started by: balu1234
4 Replies

2. Shell Programming and Scripting

Reading line by line from live log file using while loop and considering only those lines start from

Hi, I want to read a live log file line by line and considering those line which start from time stamp; Below code I am using, which read line but throws an exception when comparing line that does not contain error code tail -F /logs/COMMON-ERROR.log | while read myline; do... (2 Replies)
Discussion started by: ketanraut
2 Replies

3. Shell Programming and Scripting

Comparison of fields then increment a counter reading line by line in a file

Hi, i have a scenario were i should compare a few fields from each line then increment a variable based on that. Example file 989878|8999|Y|0|Y|N|V 989878|8999|Y|0|N|N|V 989878|8999|Y|2344|Y|N|V i have 3 conditions to check and increment a variable on every line condition 1 if ( $3... (4 Replies)
Discussion started by: selvankj
4 Replies

4. Shell Programming and Scripting

Reading text file, comparing a value in a line, and placing only part of the line in a variable?

I need some help. I would like to read in a text file. Take a variable such as ROW-D-01, compare it to what's in one line in the text file such as PROD/VM/ROW-D-01 and only input PROD/VM into a variable without the /ROW-D-01. Is this possible? any help is appreciated. (2 Replies)
Discussion started by: xChristopher
2 Replies

5. Shell Programming and Scripting

How to start reading from the nth line till the last line of a file.

Hi, For my reuirement, I have to read a file from the 2nd line till the last line<EOF>. Say, I have a file as test.txt, which as a header record in the first line followed by records in rest of the lines. for i in `cat test.txt` { echo $i } While doing the above loop, I have read... (5 Replies)
Discussion started by: machomaddy
5 Replies

6. UNIX for Dummies Questions & Answers

Parsing file, reading each line to variable, evaluating date/time stamp of each line

So, the beginning of my script will cat & grep a file with the output directed to a new file. The data I have in this file needs to be parsed, read and evaluated. Basically, I need to identify the latest date/time stamp and then calculate whether or not it is within 15 minutes of the current... (1 Reply)
Discussion started by: hynesward
1 Replies

7. Shell Programming and Scripting

[Solved] Problem in reading a file line by line till it reaches a white line

So, I want to read line-by-line a text file with unknown number of files.... So: a=1 b=1 while ; do b=`sed -n '$ap' test` a=`expr $a + 1` $here do something with b etc done the problem is that sed does not seem to recognise the $a, even when trying sed -n ' $a p' So, I cannot read... (3 Replies)
Discussion started by: hakermania
3 Replies

8. Shell Programming and Scripting

reading a file inside awk and processing line by line

Hi Sorry to multipost. I am opening the new thread because the earlier threads head was misleading to my current doubt. and i am stuck. list=`cat /u/Test/programs`; psg "ServTest" | awk -v listawk=$list '{ cmd_name=($5 ~ /^/)? $9:$8 for(pgmname in listawk) ... (6 Replies)
Discussion started by: Anteus
6 Replies

9. Shell Programming and Scripting

KSH: Reading a file line by line into multiple arrays

Hi - I have a file that contains data in this format:- #comment value1 value2 value3 #comment value4 value5 value6 value7 #comment value8 value9 I need to read value1, value2 and value3 into one array, value4 value5 value6 and value7 into another array and value8 and value9 into a 3rd... (2 Replies)
Discussion started by: sniper57
2 Replies
Login or Register to Ask a Question

Featured Tech Videos