How do I read several files and track "wc -l" Every 1min X 4 Iterations


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting How do I read several files and track "wc -l" Every 1min X 4 Iterations
# 1  
Old 06-30-2010
How do I read several files and track "wc -l" Every 1min X 4 Iterations

I need help with these requirement:

I have a FILE1, single column with list of filenames (about 30-150). I need to read each line from FILE1, do a word-count-lines of the filenames and echo a message if word-count-lines is same for 1minute X 4 iterations

Example and idea below:

FILE1:
Code:
logfile1
logfile2
logfile3
logfile....n

Code:
var1=`cat logfile1 | wc -l`  Iteration1 1st minute
var2=`cat logfile1 | wc -l`  Iteration2 2nd minute
var3=`cat logfile1 | wc -l`  Iteration3 3rd minute
var4=`cat logfile1 | wc -l`  Iteration4 4th minute
var1a=`cat logfile2 | wc -l`  Iteration1 1st minute
var2a=`cat logfile2 | wc -l`  Iteration2 2nd minute
var3a=`cat logfile2 | wc -l`  Iteration3 3rd minute
var4a=`cat logfile2 | wc -l`  Iteration4 4th minute

Please don't hesitate to ask for more clarity.
Thanks for your help in advance!

Moderator's Comments:
Mod Comment Please use code tags

Last edited by Franklin52; 06-30-2010 at 04:17 PM..
# 2  
Old 07-01-2010
Code:
for line in FILE1
do
cat line | wc -l > tmp
echo "the number of lines in ${line} are ${tmp} >> output_file
done

this will store the result of all your lines in output_file with the number of lines in front of it.
but i could not understand what do you mean by
Code:
message if word-count-lines is same for 1minute X 4 iterations

let me know what do you mean by this. or show me a sample output
# 3  
Old 07-19-2010
How Do I read all files in a changing directory and report on each file?

dazdseg,

Sorry for late follow-up, being out sick...

I have a directory that changes name every day to date name (format ccyy-mm-dd) there are up to 200 files (count varies by day) in the directory with name type xxxx.log.
I need to read each file and echo/email/report on each file if new records have not being written in 4 minutes, I'm suppose to check each file every minute but only report if condition stay same for 4 minutes.

Hope that is clearer.

Thanks,
Olu
# 4  
Old 07-20-2010
Can you just check the time the files were last modified and sound the alarm if they haven't changed in 4 minutes? That should be simpler.

I wanted to see if I could do this with wc -l and basic commands. I'm not saying this is a good idea, but I was successful. The upside is that it's fairly simple, downside is that it creates an extra file for each .log file where it stores a record of the number of lines in the .log file. It could be done with one file for the records, but it's too late tonight to try it. Also, this will not give accurate results for the first three minutes, but should work after that.
Code:
#! /bin/sh 
for file in *.log
do
wc -l >> "${file}.record" < "$file"
num=$(tail -4 "${file}".record | uniq | wc -l)
[ $num = 1 ] && echo "Alert! ${file} Unchanged for 4 minutes!"
done

This can be ran on a 1 minute cron or an infinite loop with a "sleep 1m" at the end.

This could be added to rotate the .record files to keep only the last 4 lines:
Code:
for i in *.record
do
newrecord=$(tail -4 $i)
echo "$newrecord" > $i
done

Both .log files are identical and I ran this 4 times to populate the .record files before this test. On the 4th run without a change, it rang the alarm:
Code:
# echo "blah blah" | tee -a file2.log file.log
blah blah
# ./test
# ./test
# ./test
# ./test
Alert! file.log Unchanged for 4 minutes!
Alert! file2.log Unchanged for 4 minutes!
#

# 5  
Old 07-20-2010
The "last modified time" is not accurate of record being written into the file. Is it and how would you construct the code?

The basic commands code looks good but accuracy of the first 3minutes is important to eliminate false alert. The log files may be an issue because I will be processing about 200 log files.

Thanks for you help but I will look for other ideas if you have any.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Delete all log files older than 10 day and whose first string of the first line is "MSH" or "<?xml"

Dear Ladies & Gents, I have a requirement to delete all the log files in /var/log/test directory that are older than 10 days and their first line begin with "MSH" or "<?xml" or "FHS". I've put together the following BASH script, but it's erroring out: for filename in $(find /var/log/test... (2 Replies)
Discussion started by: Hiroshi
2 Replies

2. UNIX for Dummies Questions & Answers

Using "mailx" command to read "to" and "cc" email addreses from input file

How to use "mailx" command to do e-mail reading the input file containing email address, where column 1 has name and column 2 containing “To” e-mail address and column 3 contains “cc” e-mail address to include with same email. Sample input file, email.txt Below is an sample code where... (2 Replies)
Discussion started by: asjaiswal
2 Replies

3. Shell Programming and Scripting

Read from "list1" and list matches in "list2"

I want to print any matching IP addresse in List1 with List 2; List 1 List of IP addresses; 161.85.58.210 250.57.15.129 217.23.162.249 74.76.129.101 30.221.177.237 3.147.200.59 170.58.142.64 127.65.109.33 150.167.242.146 223.3.20.186 25.181.180.99 2.55.199.32 (3 Replies)
Discussion started by: lewk
3 Replies

4. Shell Programming and Scripting

Purpose of "read" and "$END$" in ksh ?

Hi, Could anyone please shed some light on the following script lines and what is it doing as it was written by an ex-administrator? cat $AMS/version|read a b verno d DBVer=$(/usr/bin/printf "%7s" $verno) I checked that the cat $AMS/version command returns following output: ... (10 Replies)
Discussion started by: dbadmin100
10 Replies

5. Shell Programming and Scripting

awk command to replace ";" with "|" and ""|" at diferent places in line of file

Hi, I have line in input file as below: 3G_CENTRAL;INDONESIA_(M)_TELKOMSEL;SPECIAL_WORLD_GRP_7_FA_2_TELKOMSEL My expected output for line in the file must be : "1-Radon1-cMOC_deg"|"LDIndex"|"3G_CENTRAL|INDONESIA_(M)_TELKOMSEL"|LAST|"SPECIAL_WORLD_GRP_7_FA_2_TELKOMSEL" Can someone... (7 Replies)
Discussion started by: shis100
7 Replies

6. Shell Programming and Scripting

"Join" or "Merge" more than 2 files into single output based on common key (column)

Hi All, I have working (Perl) code to combine 2 input files into a single output file using the join function that works to a point, but has the following limitations: 1. I am restrained to 2 input files only. 2. Only the "matched" fields are written out to the "matched" output file and... (1 Reply)
Discussion started by: Katabatic
1 Replies

7. Shell Programming and Scripting

read parts of binary files by "ranges"

i read the "cat" manpages, but i could not find to tell it like "read file XY.BIN from byte 1000 to byte 5000" can somebody please point me into the right direction? cat would be the ideal tool for my purpose, the way it behaves, but i miss this ranges option. thanks for any input. (2 Replies)
Discussion started by: scarfake
2 Replies

8. Shell Programming and Scripting

read -p "prompt text" foo say "read: bad option(s)" in Bourne-Shell

Hallo, i need a Prompting read in my script: read -p "Enter your command: " command But i always get this Error: -p: is not an identifier When I run these in c-shell i get this error /usr/bin/read: read: bad option(s) How can I use a Prompt in the read command? (9 Replies)
Discussion started by: wiseguy
9 Replies

9. Shell Programming and Scripting

Awk - to test multiple files "read" permission ?

Hi Masters, Iam new to this Forum and this is my first post. My question is: I've some datafiles belongs the type (A, B, C) in the location 'export/home/lokiman ' dataA1.txt dataB28.txt dataC35.txt 1) I've to check the read permission for each file, if it not there then I've to... (1 Reply)
Discussion started by: lokiman
1 Replies

10. Shell Programming and Scripting

how to request a "read" or "delivered" receipt for mails

Dears, I've written a script which allows me to send mails in different formats with different attaches. Now I still want to add a feature to this script. My users would like to be able to receive a "read" or "delivered" receipt for their mails. The script send mails on behalve of an specific... (1 Reply)
Discussion started by: plelie2
1 Replies
Login or Register to Ask a Question