Count lines and words of a stream output with tail


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Count lines and words of a stream output with tail
# 8  
Old 10-18-2013
Quote:
Originally Posted by CarloM
Some rough (and untested!) code:
Code:
tempfile=$(mktemp -t XXXXXXXX.$$) || exit 1

( tail -f file.txt |  awk '/get|point/ {gsub (/point.*/,"");print NF}' | tee "${tempfile}" ) &
tailpid=$!

while true
do
        sleep 1
        found=$(wc -l "${tempfile}"|awk '{print $1}')
        [ $found -gt 3 ] && break
done

rm "${tempfile}"
kill $tailpid

Quote:
Originally Posted by CarloM
One way to do it would be to run the tail/awk pipe in a background subshell, redirect that output to a temp file, and have the parent script terminate the subshell when the file is 3 lines long.
That is exactly what your code does, but killing the subshell leaves each process in the tail|awk|tee pipeline alive and orphaned.

On a more general note, I can't think of any reason to ever use the following pipeline to count the lines of a single file:
Code:
wc -l file |awk '{print $1}'

wc will not print a pathname if you redirect stdin, wc -l < file. Alternatively, you can use AWK alone and print NR from an END block.

Quote:
Originally Posted by Kibou
Here is the final version of the script. It works like a charm!
After running your script, you may want to consult with ps and, if necessary, rework your solution.

Regards,
Alister
This User Gave Thanks to alister For This Post:
# 9  
Old 10-18-2013
Quote:
killing the subshell leaves each process in the tail|awk|tee pipeline alive and orphaned.
Good point. (I did say it was untested! Smilie)

It's not hard to correct though. Something like:
Code:
childpids=$(ps -ef | awk -vPPID=$tailpid -vORS=' ' '$3==PPID {print $2}')
kill $childpids
 kill $tailpid

# 10  
Old 10-18-2013
Quote:
Originally Posted by CarloM
It's not hard to correct though. Something like:
Code:
childpids=$(ps -ef | awk -vPPID=$tailpid -vORS=' ' '$3==PPID {print $2}')
kill $childpids
 kill $tailpid

It would be simpler to just place the monitored and monitoring code in the same process group and use kill 0 to terminate the entire group.
Code:
set -m
exec 3>&2 2>/dev/null

sh 2>&3 <<'EOF'
    tail ... | awk ... | tee ... &

    while :; do
        monitoring-commands
    done

    cleanup-commands
    kill 0
EOF

The highlighted redirections can be omitted if the job control messages aren't a bother.

However, in my opinion, a fifo-based solution is a superior approach. Since reading on a fifo blocks, there's no need for a polling while-loop and explicit sleeps.
Code:
mkfifo data-pipe

tail -f file > data-pipe &
awk 'awk-script' data-pipe
kill $!

When the awk script exits (after three matches, presumably), tail is killed.

If portability isn't a concern, (ksh or bash) coprocesses can be used instead of mkfifo.

Regards,
Alister

Last edited by alister; 10-19-2013 at 08:31 AM.. Reason: Deleted unused variable
These 2 Users Gave Thanks to alister For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Count words/lines between two tags using awk

Is there an efficient awk that can count the number of lines that occur in between two tags. For instance, consider the following text: <s> Hi PP - my VBD - name DT - is NN - . SENT . </s> <s> Her PP - name VBD - is DT - the NN - same WRT - . SENT - </s> I am interested to know... (4 Replies)
Discussion started by: owwow14
4 Replies

2. Shell Programming and Scripting

How count the number of two words associated with the two words occurring in the file?

Hi , I need to count the number of errors associated with the two words occurring in the file. It's about counting the occurrences of the word "error" for where is the word "index.js". As such the command should look like. Please kindly help. I was trying: grep "error" log.txt | wc -l (1 Reply)
Discussion started by: jmarx
1 Replies

3. UNIX for Dummies Questions & Answers

Count the lines with the same values in a column and write the output to a file

Hey everyone! I have a tab delimited data set which I want to create an output contained the calculation of number of those lines with a certain value in 2nd and 3rd column. my input file is like this: ID1 1 10M AAATTTCCGG ID2 5 4M ACGT ID3 5 8M ACCTTGGA ID4 5 ... (7 Replies)
Discussion started by: @man
7 Replies

4. Shell Programming and Scripting

Deleting lines from a stream after matching a pattern

Hi, I have a requirement to to an ldapsearch and remove the shadow attributes in the output file. What I do is ldapsearch() | operation to remove shadow > FILE The ldapsearch gives output like this(with same line formation): objectClass: FSConfig objectClass: extensibleObject fsCAIP:... (10 Replies)
Discussion started by: lorzinian
10 Replies

5. Shell Programming and Scripting

Scripting help to identify words count in lines

Hi everybody, i have this biological situation to fix: > Id.1 ACGTACANNNNNNNNNNNACGTGCNNNNNNNACTGTGGT >Id.2 ACGGGT >Id.3 ACGTNNNNNNNNNNNNACTGGGGG >Id.4 ACGTGCGNNNNNNNNGGTCANNNNNNNNCGTGCAAANNNNN ........ .... These are nucleotidic sequences with some "NNNN..." always of the same... (4 Replies)
Discussion started by: Giorgio C
4 Replies

6. Shell Programming and Scripting

Shell script to find out words, replace them and count words

hello, i 'd like your help about a bash script which: 1. finds inside the html file (it is attached with my post) the code number of the Latest Stable Kernel, 2.finds the link which leads to the download location of the Latest Stable Kernel version, (the right link should lead to the file... (3 Replies)
Discussion started by: alex83
3 Replies

7. Shell Programming and Scripting

Count the no of lines between two words

Please help in the following problem: Input is: Pritam 123 456 Patil myname youname Pritam myproject thisproject iclic Patil remaining text some more text I need the command which will display the no of lines between two words in the whole file. e.g. Display all the no of lines... (5 Replies)
Discussion started by: zsudarshan
5 Replies

8. Shell Programming and Scripting

awk help needed in trying to count lines,words and characters

Hello, i am trying to write a script file in awk which yields me the number of lines,characters and words, i checked it many many times but i am not able to find any mistake in it. Please tell me where i went wrong. BEGIN{ print "Filename Lines Words Chars\n" } { filename=filename + 1... (2 Replies)
Discussion started by: salman4u
2 Replies

9. UNIX for Dummies Questions & Answers

how can I use the stream output in other program

Hello I wander if im doing : ls -l and its giving me lets say 3 results : -rw-r--r-- 1 blah other 1789 May 19 2003 foo.c -rw-r--r-- 1 blah other 1014 May 19 2003 foo.h -rw-r--r-- 1 blah other 270 May 19 2003 foo1.c now I would like to use the first... (1 Reply)
Discussion started by: umen
1 Replies

10. Programming

stream output??

Hi all, I've a structure let's say typdef struct a { int a; int b; char* string} b; I need to make the function smth like readdir(), returning everytime it's called pointer to the next structure. Let's say functions would read the next file in the directory and all the permitions put it in... (2 Replies)
Discussion started by: solvman
2 Replies
Login or Register to Ask a Question