Shell Programming and Scripting

Hi all,

I am using Sed to convert a log file into a csv, so far so good. Kudos to this forum for helping me thus far!

My current problem. There are some lines in the log file that I do not want. How can I delete lines where the line legth is less than say 100?

Here are some sample lines from my log file:

Code:

www.somesite.com.au,111.11.24.21,385,12/Jul/2010:10:05:52 +1000,"GET  https://www.somesite.com.au/moo/home/ HTTP/1.1",200,137446
</event>
<event,rev="1.2">
www.somesite.com.au,111.11.24.21,385,12/Jul/2010:10:05:52  +1000,"GET https://www.somesite.com.au/moo/home/  HTTP/1.1",200,137446

so in this example, it would be safe to say that I want to delete any line that is less than 100 characters long

Thanks in advance to the community (I hope I can contribute someday! )

sed is not keen at doing arithmetic evaluation.
You should need first to evaluate length though an *sh script, then pass line to sed, but then you loose sed's benefit.
using awk from the beginning, may be more obvious

Nah. It can be easily done with sed. The following will delete all lines that are less than 100 characters long:

Code:

sed '/.\{100\}/!d'

Regards,
Alister

Here you go.

Code:

#!/bin/ksh
set -x
logfile=$dir/logfilename.txt
thr=100
cat $logfile | while read a
do
nochar = `echo $a | wc -c`
sleep 1
if (( $nochar < $thr )) then
echo echo "$a" >>new.logfile.log
else "no. of chars less than 10";
fi
done

Let me know if this helps.

Thanks,
Sameer.

Thanks so much all, will test them out and see which will be the most appropriate for my scenario. =]

by awk:

Code:

awk 'length>100 ' urfile

That read command will discard leading and trailing whitespace and mishandle any occurrences of a backslash, leading to erroneous line length counts. Also, by using ${#a}, you can determine length without using wc.

Code:

while IFS= read -r a; do
    [ ${#a} -ge 100 ] && echo "$a"
done < "$logfile" >> "$logfile".new

Regards,
Alister