Flag Programs that have coding problems


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Flag Programs that have coding problems
# 1  
Old 05-22-2016
Flag Programs that have coding problems

I create a shell script program (AIX OS) that flags any logs that certain keywords such as memsize, sasfoundation, and real time. My program isn't working maybe because my syntax logistic isn't correct.

Here are examples of the logs:
file1.log.02897 file2.log.02896 filez.log.02899 file4.log.02898 file5.log.028999 file6.log.02898 filex.log.028999 filem.log.028999
file11.log.028999 filexx.log.028999 file56.log.028999 filexz.log.028999 fileq.log.028999

first three attachments are examples of data in each the log files:

last attachment progflag.csv is an example of how the result needs to be displayed in the file.

Here is my code:
Code:
#!/bin/bash

#tar current logs and move the tar package to /tmp/log . Untar a logs

sudo cd /home/user/myfiles ; tar cf - */*/*.log.* | ( cd /tmp/log ; tar xf - )

#change all files under  log directory to write and execute
chmod 775 files =/tmp/log/*.*

#set target threshold to 3600 sec.   (1hour)
echo threshold = 3600

#read in multiple .log files from directory /tmp/log

for file in /tmp/log/*.log.*; do
    while read -r LINE; do
        echo "$LINE"

# search for keywords line by line
  flag_line=$(echo $LINE | awk '{print $NF}')

# if MEMSIZE has a value, extract value and then Output column name MEMSIZE and values to progflag.csv
  if echo $flag_line "MEMSIZE" | cut -d: -f2 then 
  echo MEMSIZE=$flag_line 

# extract numeric value from row titled REAL TIME

 if echo $flag_line "REAL TIME:" | cut -d: -f3 then 
  echo SEC=$flag_line
  
# if sec is greater than threshold then output column name SECOND and values to progflag.csv
  IF $sec  > $thresold then
   echo SECOND=sec

# if path SASFoundation exist than extract SASFoundation and add SASFoundation to alias SASEXE.  Output column name SASEXE and 
    value to  progflag.csv

  if echo $flag_line '/SASFoundation' grep SASFoundation
   echo SASEXE=$flag_line
   
   #output each title of filename with extracted data to progflag.csv
  
  Echo filename=$LINE
    Echo $MEMSIZE $SECOND $SASEXE $LINE
 done < "$file"  >> progflag.csv

#send the attachment via email
 if progflag.csv then
(cat ./progflag.csv)|mailx -s "subject text" -a "Programs flagged" receiver@domain.com

#after process is completed, remove all files from directory tmp/log
 sudo rm -rf $files $progflag.csv

# 2  
Old 05-22-2016
You could do most of the heavy lifting using awk:

Code:
cd /tmp/log
awk -F '[=:;.]' '
  function pr() {if(NR>1) printf "%s\t%s\t%s\t%s\n", K[1],K[2],K[3],K[0]}
  BEGIN {
      printf "MEMSIZE\tSECOND\tPATH\tFilename\n"
      for(i=split("memsize ,Real Time ,path",A,",");i;i--) L[A[i]]=i
  }
  FNR==1 {
      pr()
       K[0]=FILENAME
      K[1]=K[2]=K[3]=x
  }
  $1 in L {v=$2;gsub(/^[\\/ ]*/,"",v);gsub(/ *$/,"",v);K[L[$1]]=v}
  END{pr()}
' *.log.* > progflag.csv

[ -s progflag.csv ] && mailx -s "subject text -a "Programs flagged" receiver@domain.com < progflag.csv

This User Gave Thanks to Chubler_XL For This Post:
# 3  
Old 05-23-2016
I'm trying to figure out what some of your code is trying to do. Can you please answer the following questions:

1. Will the user running this script have read access to files matching the pattern /home/user/myfiles/*/*/*.log.*? If yes, why are you wasting time making copies of these files? If not, why are you copying these files into a directory and changing their access permissions such that every user with access to your system will be able to read this (presumably private) data?

2. Why does anyone need write or execute permission to the log files you are copying while this script is running?

3. What exactly are you trying to do with the command?:
Code:
chmod 775 files =/tmp/log/*.*

What it does do is change the permissions of a file named files and of all files in the directory located in the current working directory named =/tmp/log whose names in that directory contain the string .log. so that any process running with the user ID of the file's owner or with a group ID matching the file's group will have read, write, and execute permissions and all other processes will be able to read and execute them.
This User Gave Thanks to Don Cragun For This Post:
# 4  
Old 05-24-2016
your question:
1. Will the user running this script have read access to files matching the pattern /home/user/myfiles/*/*/*.log.* ?

no. I will schedule the shell script program to run using ascheduler.

Code:
sudo cd /home/user/myfiles ; tar cf - */*/*.log | ( cd /tmp/log ; tar xf - )

This script copies log files from one directory to another. The log files are generated with the current dates on the end of the .log extension for example file1.log.05242016


I ran your code which is awesome. I had one issue. I checked the brackets in the script and the code appears to be good. This is the error message that I am getting

Code:
awk: 0602-542 there is an extra ] character.
The source line is 12.
The error context is


v=$2;gsub(/^[\\/ ]*/,"",v);gsub(/ *$/,"",v);K[L[$1]]=v}
awk: 0602-521 There is a regular expresssion error.
[] imbalance

0 progflag.csv


Last edited by Corona688; 05-24-2016 at 06:03 PM..
# 5  
Old 05-25-2016
Quote:
Originally Posted by dellanicholson
Code:
awk: 0602-542 there is an extra ] character.
The source line is 12.
The error context is


v=$2;gsub(/^[\\/ ]*/,"",v);gsub(/ *$/,"",v);K[L[$1]]=v}
awk: 0602-521 There is a regular expresssion error.
[] imbalance

0 progflag.csv

Try
Code:
 v=$2;gsub("^[/ ]*","",v);gsub(/ *$/,"",v);K[L[$1]]=v

This User Gave Thanks to Chubler_XL For This Post:
# 6  
Old 05-27-2016
Thanks, for this script
Code:
v=$2;gsub("^[/ ]*","",v);gsub(/ *$/,"",v);K[L[$1]]=v

it works

But

When the code ran and populated the progflag.csv file,
columns MEMSIZE, SECOND and PATH were blank. The column for filename
was populated with the names of each log files from the directory (tmp/log)

a. What I want the program to do is to only read in all files that have an extension
.log.

a. there are other files in the log directory that doesn't have .log. extension

1 a. Also , What I want the program to output in the proflag.csv for example
if the following information in file12.log doesn't exist, such as MEMSIZE ,
a directory path will the text /SASFoundation and the variable for Real Time value
(Real Time 0.1) is less than 1.0, there is no output for
file12.log in progflag.csv .

b. Another example, if in file9.log, the following variables and values exist for
example "MEMSIZE=200" and variable Real Time (Real Time 4.0) row value is
More than 1.0, output the values and the column names to the proflag.csv.

c. last example, if in file25.log, there is a directory path will text /SASFoundation
and the variable Real Time (Real Time 0.2) row value is less than 1.0 output the
values and its column name to the proflag.csv

Here is an example(1a-c) of the result: progflag.csv

MEMSIZE SECOND PATH filename:
200 4.0 file1.log
SASFoundation file25.log

4. last, step only send an email if progflag.csv file rows are populated.



2. So, how do you add an if then condition statement to awk in your code
program?

for example

awk
if MEMSIZE then Or if SECOND ( SECOND > 1.0)
is greater than 1.0 (SECOND is the alias for Real Time. In this case Real Time 4.0)
or SASFoundation (Path is the alias for SASFoundation ) then output to
progflag.csv

Last edited by RudiC; 05-27-2016 at 01:43 PM.. Reason: Added code tags.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

sed print flag

I have an input file that looks something like this: .... key1: ABC .... key2: DEF .... key1: GGG .... key2: HHH .... The row of dots represents any number of lines that don't contain the strings "key1:" or "key2:" The strings key1: and key2: will always appear alternately as in the... (8 Replies)
Discussion started by: pmennen
8 Replies

2. UNIX for Dummies Questions & Answers

Difference between inbuilt suid programs and user defined root suid programs under bash shell?

Hey guys, Suppose i run passwd via bash shell. It is a suid program, which temporarily runs as root(owner) and modifies the user entries. However, when i write a C file and give 4755 permission and root ownership to the 'a.out' file , it doesn't run as root in bash shell. I verified this by... (2 Replies)
Discussion started by: syncmaster
2 Replies

3. UNIX for Dummies Questions & Answers

Get the id with specify flag

.... means multi line ddd,bug fgdrg dfdfsdfdfsd fsdfdfdfd fdfdsfdsfsd ....... flag2 ...... aaa,bug sfsfsfsfs dfdfsdfdfsd fsdfdfdfd fdfdsfdsfsd ...... flag1 ...... ddd,bug fgdrg dfdfsdfdfsd (9 Replies)
Discussion started by: yanglei_fage
9 Replies

4. UNIX for Dummies Questions & Answers

'h' flag in du

Hey, all! Why is the "human readable" flag changing the behavior of du? And while I'm at it, can you make du only look at files, not directories. I often find myself wanting to find the largest file(s) in a dir or vol. Using 'find' itself, it seems you have to at least be able to guess the size of... (2 Replies)
Discussion started by: sudon't
2 Replies

5. AIX

Flag issue

Hi all, We dont have access to aix source code and i have a doubt. The flag SC_NO_RESERVE, is it got to do anything with the failover? If the flag is set the paths are going to failed state. If flag is not set everything comes up fine after failover. Thanks in advance for helping ... (2 Replies)
Discussion started by: gnakul
2 Replies

6. Shell Programming and Scripting

Add the flag

#!/bin/bash while : do ./abc.sh PB sleep 60 ./abc.sh RA sleep 60 ./abc.sh GS sleep 68400 done Instead of making the script sleep for sometime, it doesn't work all the time as time may shift over a period. How to make a script wake up every 30 seconds and check the current time, if... (2 Replies)
Discussion started by: sandy1028
2 Replies

7. UNIX for Dummies Questions & Answers

Are programs like sys_open( ) ,sys_read( ) et al examples of system level programs ?

Are the programs written on schedulers ,thread library , process management, memory management, et al called systems programs ? How are they different from the programs that implement functions like open() , printf() , scanf() , read() .. they have a prefix sys_open, sys_close, sys_read etc , right... (1 Reply)
Discussion started by: vishwamitra
1 Replies

8. AIX

du flag -x

Hi, I would like to know if there's any option to use with the du command so that I can list only the files/directories on the current filesystem... I usually use du -gs *But I'd like to see only the directories in the filesystem I am on, and not the mount point directory of other fss... ... (6 Replies)
Discussion started by: Casey
6 Replies

9. Shell Programming and Scripting

Problems with find's -newer Flag

I am writing a script that looks in a reports directory, copies a specified script to a working folder, copies some data files into the working folder, runs the report, zips the new files, then uploads them. Right now to determine what files to zip (as I don't know how many report files there... (6 Replies)
Discussion started by: droppedonjapan
6 Replies

10. Shell Programming and Scripting

the flag -c in sh command

I need your help please. In a production system, i've seen many running process as follow: sh -c ./pathname/shellname what exactly the flag option -c is used for? ive tried to look at the man page, but it doesnt say much. ill appreciate yor help. Thanks (4 Replies)
Discussion started by: alexcol
4 Replies
Login or Register to Ask a Question