Read file lines and pass line values as arguments.


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Read file lines and pass line values as arguments.
# 1  
Old 11-16-2017
Read file lines and pass line values as arguments.

Dears,

Need help to implement below requirement


A file (detail.txt)contain :
Code:
1st column: Stream					
2nd column: PathAddress			    
3rd column: Counterlimit
4th column: TransactionDateColumn	
5th column: DateType				
6th column: SleepValue				
7th column: Status

Need to write a script where it read each line of the file.
Check the status column (0 Not active, 1 active)
then pass as following column values as an argument.
stream
pathaddress
counterlimit on below script.

Code:
#!/bin/sh
DATE=`date +"%Y-%m-%d %H:%M:%S"`
LATENCY ()
{
stream=
path=
counter=0
while [ $counter -le $counterlimit ]
do
	for  files in `ls $path | grep -v .temp | head -3`
	do
		if [[ -f $files ]] && [[ -s $files ]];
		then 
			if ## 4th column of the detail.txt having value 1
			then 
				TT_FIRST=`awk -F ',' 'NR==1{print $1}' $files`
				TT_LAST=`awk -F ',' 'END{print $1}' $files`
				FILENAME=`ls $files | cut -d '/' -f6`
				TIMESTAMP=$( date -r $files +'%s')
				FILE_CR_TIME=$( date +'%Y-%m-%d %H:%M:%S' -r $files)
				TRANS_TIME_FIRST=$(date -d @$(printf '%.0f\n' $TT_FIRST) +'%s')
				TRANS_TIME_LAST=$(date -d @$(printf '%.0f\n' $TT_LAST) +'%s')
				RECORD_TIME_FIRST=$(date -d @$(printf '%.0f\n' $TT_FIRST) +"%Y-%m-%d %H:%M:%S")
				RECORD_TIME_LAST=$(date -d @$(printf '%.0f\n' $TT_LAST) +"%Y-%m-%d %H:%M:%S")
				TIME_LAG_FIRST=$(expr $TIMESTAMP - $TRANS_TIME_FIRST)
				TIME_LAG_LAST=$(expr $TIMESTAMP - $TRANS_TIME_LAST)
				DELAY_PROCESS=$((`expr $(date -u +"%s") - $TIMESTAMP`))
				TRANS_DIFFTIME=$(expr $TRANS_TIME_LAST - $TRANS_TIME_FIRST)
				counter=$counterlimit
			else
			if   ## 4th column of the detail.txt having value 4
			then
				TTHEX_FIRST=`awk -F ',' 'NR==1{print $4}' $files`
				TTHEX_LAST=`awk -F ',' 'END{print $4}' $files`
				TIMESTAMP=$( date -r $files +'%s')
				FILENAME=`ls $files | cut -d '/' -f5`
				FILE_CR_TIME=$( date +'%Y-%m-%d %H:%M:%S' -r $files)
				TRANS_TIME_FIRST=$(date -d @$(expr `printf "%d" 0x$TTHEX_FIRST` / 1000) +'%s')
				TRANS_TIME_LAST=$(date -d @$(expr `printf "%d" 0x$TTHEX_LAST` / 1000) +'%s')
				RECORD_TIME_FIRST=$(date -d  @$(expr `printf "%d" 0x$TTHEX_FIRST` / 1000) +"%Y-%m-%d %H:%M:%S")
				RECORD_TIME_LAST=$(date -d  @$(expr `printf "%d" 0x$TTHEX_FIRST` / 1000) +"%Y-%m-%d %H:%M:%S")
				TIME_LAG_FIRST=$(expr $TIMESTAMP - $TRANS_TIME_FIRST)
				TIME_LAG_LAST=$(expr $TIMESTAMP - $TRANS_TIME_LAST)
				DELAY_PROCESS=$((`expr $(date -u +"%s") - $TIMESTAMP`))
				TRANS_DIFFTIME=$(expr $TRANS_TIME_LAST - $TRANS_TIME_FIRST)
				counter=100
			else
				echo "file doesn't exist" >/dev/null
			fi
			fi
		if [ "${FILENAME}" != "" ];
		then
			echo "${DATE} ${stream} ${FILENAME} ${FILE_CR_TIME} ${RECORD_TIME_FIRST} ${TIME_LAG_FIRST} ${RECORD_TIME_LAST} ${TIME_LAG_LAST} ${DELAY_PROCESS} ${TRANS_DIFFTIME}" | hdfs dfs -appendToFile - /AUDIT/FINAL.csv
		else
			echo "Filename is empty" >/dev/null 
		fi 
	done
((counter++))
sleep $sleepValue
done
}


detail.txt contains:
Code:
S1	/streamslz/LZ/S1	100	1		1	10	1
IUCS	/streamslz/LZ/AnritsuIUCS	100	4	2	5 1
IUPS	/streamslz/LZ/IUPS		100	1	1	5	1



Regards,
sadique

Last edited by jim mcnamara; 11-16-2017 at 09:33 AM.. Reason: quote tags changed to CODE tags
# 2  
Old 11-23-2017
I'm not sure what all you need, but hopefully this examples give some ideas for your needs.

1st I made some example file. I hope that if you use some input files, you give those also. Much faster to understand what you are looking.

My script include /bin/somesh.
Somesh = bash or ksh. ksh source

Code:
#!/bin/somesh
###########################################################
# make some example files
# example file
cat <<EOF > detail.txt
S1      /streamslz/LZ/S1        100     1               1       10      0
IUCS    /streamslz/LZ/AnritsuIUCS       100     4       2       5 0
IUPS    /streamslz/LZ/IUPS              100     1       1       5       0
IUPS    /tmp/example            2       1       1       5       1
EOF

mkdir -p /tmp/example 2>/dev/null
cat <<EOF > /tmp/example/1.txt
2017-11-20 12:01:00,2,3,4,5,6,7,8,
2017-11-20 13:01:00,3,3,3,3,3,33,
2017-11-20 14:01:00,98,97,96,95,94
EOF
cat <<EOF > /tmp/example/2.txt
2017-11-21 10:01:00,22,3,4,5,6,7,8,
2017-11-21 11:02:00,3,3,3,3,3,33,
2017-11-21 12:03:00,298,297,296,295,294
EOF

In this example I have used main to handle detail.txt and then call some function after testing status.
I use shell builtin properties to use time and use find to print filelist with timestamp.

Code:
#!/bin/somesh
###########################################################
# timestamp
DATE=$( printf "%(%Y-%m-%d %H:%M:%S)T" now )

###########################################################
dosome1()
{
        :
}

###########################################################
dosome2()
{
        # copy args to the variables
        stream="$1"
        path="$2"
        counterlimit="$3"
        datecol="$4"
        sleepinterval="$5"

        counter=0

        CDIR=$PWD
        while (( counter <= counterlimit ))
        do
                cnt=0

               [ ! -d "$path" ] && echo "no $path " >&2 && continue
                cd "$path" || continue # some problem to change dir

                find . -maxdepth 1 -name "*" -type f -printf "%p %TY-%Tm-%Td %TH:%TM:%TS\n" | sort | \
                while read files FILE_CR_TIME
                do
                        [ ! -r "$files" ] && echo "no read access $files" >&2 && continue  # no read access
                        ((cnt+=1))

                        ((cnt>3)) && break # only 3 files / dir

                        FILENAME="${files##*/}"
                        TIMESTAMP=$(printf "%(%#)T" "$FILE_CR_TIME")
                        #echo "FN:$FILENAME epoc:$TIMESTAMP modtime:$FILE_CR_TIME"


                        IFS="," read F1_FLD1 F1_FLD2 F1_FLD3 F1_FLD4 F1_ENDFLDS < $files
                        IFS="," read FL_FLD1 FL_FLD2 FL_FLD3 FL_FLD4 FL_ENDFLDS <<-EOF
                                 $(tail -1 $files)
                                EOF

                        #echo "$FILENAME:$FILENAME TT_FIRST:$F1_FLD1 TT_LAST:$FL_FLD1"
                        case "$datecol" in
                                1)
                                        TT_FIRST=$F1_FLD1
                                        TT_LAST=$FL_FLD1
                                        TRANS_TIME_FIRST=$(printf "%(%#)T" "$TT_FIRST")
                                        TRANS_TIME_LAST=$(printf "%(%#)T" "$TT_LAST")
                                        RECORD_TIME_FIRST=$(printf "%(%Y-%m-%d %H:%M:%S)T" "$TT_FIRST")
                                        RECORD_TIME_LAST=$(printf "%(%Y-%m-%d %H:%M:%S)T" "$TT_LAST")
                                        ((TIME_LAG_FIRST= TIMESTAMP - TRANS_TIME_FIRST))
                                        ((TIME_LAG_LAST= TIMESTAMP - TRANS_TIME_LAST))
                                        #d=$( printf "%(%#)T" now )
                                        ((DELAY_PROCESS = $( printf "%(%#)T" now )  - TIMESTAMP))
                                        ((TRANS_DIFFTIME=TRANS_TIME_LAST - TRANS_TIME_FIRST))
                                        counter=$counterlimit
                                        ;;
                                4)
                                        TTHEX_FIRST=$F1_FLD4
                                        TTHEX_LAST=$FL_FLD4
                                        # ....
                                        counter=100
                                        ;;
                        esac
                        echo "$cnt $DATE $stream $FILENAME $FILE_CR_TIME $RECORD_TIME_FIRST $TIME_LAG_FIRST $RECORD_TIME_LAST $TIME_LAG_LAST $DELAY_PROCESS $TRANS_DIFFTIME"
                done
                ((counter+=1))
                cd $CDIR
                sleep $sleepinterval
        done
}



###########################################################
# MAIN
###########################################################

while read stream pathaddress counterlimit transactiondatecolumn datetype sleepvalue status extraflds
do
        case "$status" in
                0) # not active
                        #echo dosome1 "$stream" "$pathaddress" "$counterlimit" "$transactiondatecolumn" "$datetype"
                        dosome1 "$stream" "$pathaddress" "$counterlimit"  "$transactiondatecolumn" "$datetype" 10
                        ;;
                1) # active
                        #echo dosome2 "$stream" "$pathaddress" "$counterlimit" "$transactiondatecolumn" "$datetype"
                        dosome2 "$stream" "$pathaddress" "$counterlimit"  "$transactiondatecolumn" "$datetype" 10
                        ;;
        esac
done < detail.txt


Last edited by kshji; 11-23-2017 at 02:08 PM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

How to pass command line arguments to awk program?

#!/bin/awk -f BEGIN { FS=":"; } { if ( $7 == "" ) { print $1 ": no password!"; } } I want to execute this program for a particular user to check for his password from the file /etc/passwd (as the input file) and the user details to be given... (1 Reply)
Discussion started by: sri.phani
1 Replies

2. Shell Programming and Scripting

Need a program that read a file line by line and prints out lines 1, 2 & 3 after an empty line...

Hello, I need a program that read a file line by line and prints out lines 1, 2 & 3 after an empty line... An example of entries in the file would be: SRVXPAPI001 ERRO JUN24 07:28:34 1775 REASON= 0000, PROCID= #E506 #1065: TPCIPPR, INDEX= 003F ... (8 Replies)
Discussion started by: Ferocci
8 Replies

3. Shell Programming and Scripting

Awk: Comparing arguments with in line values of file and printing the result

I need to develop a script where I will take two date arguments as parameter date1 and date2 which will in format YYYYMM. Below is the input file say sample.txt. sample.txt will have certain blocks starting with P1. Each block will have a value 118,1:TIMESTAMP. I need to compare the... (7 Replies)
Discussion started by: garvit184
7 Replies

4. Programming

How to pass the command line arguments to the shell script in c language?

hi, I am new in the shell script, and c programming with linux. I am looking to pass the arguments in c program that should be executed by the shell script. e.g. #include <stdio.h> #include <stdlib.h> int main(int argc, char *argv) { int i; for (i=1;i<argc; i++) { ... (2 Replies)
Discussion started by: sharlin
2 Replies

5. Shell Programming and Scripting

Help with script to read lines from file and count values

Hi, I need some help with a script I'm trying to write. I have a log file containing references to a number of different webservices. I wish to write a script that will list the webservices with a count as to how many times they appear in the log. An example of the log file content: ... (2 Replies)
Discussion started by: gman2010
2 Replies

6. Shell Programming and Scripting

How can i pass 2 command line arguments with a range between them

Hi, I am writting a script. i am not sure what i am trying to do is possible or not. thats why asking the best of the best. the script i want to write will recieve as input parameters 2 different options. as in MODE 1 -- start_date / end_date (2 dates has 2 go at a time MODE 2... (2 Replies)
Discussion started by: dazdseg
2 Replies

7. Shell Programming and Scripting

bash: read file line by line (lines have '\0') - not full line has read???

I am using the while-loop to read a file. The file has lines with null-terminated strings (words, actually.) What I have by that reading - just a first word up to '\0'! I need to have whole string up to 'new line' - (LF, 10#10, 16#A) What I am doing wrong? #make file 'grb' with... (6 Replies)
Discussion started by: alex_5161
6 Replies

8. Shell Programming and Scripting

Pass command line arguments to awk

I am trying to pass max as a sommand line argument when I call awk. Made the modification in the BEGIN but it is not working I'm getting an error as below: awk: txsrx.awk:82: (FILENAME=jcd.tx FNR=4161) fatal: cannot open file `40' for reading (No such file or directory) Somehow it... (2 Replies)
Discussion started by: kristinu
2 Replies

9. Shell Programming and Scripting

how do I pass or read values from file ?

Hi one & All , My Need is to Create 64 Partition and create File System in Linux. I have the Script ... for((a=0;a<=63;a++)) do fdisk /dev/cciss/c0d$a done for((a=0;a<=63;a++)) do mkfs.ext2 /dec/cciss/'c0d'$a'p1' done the moment I run the Script I get the Prompt ... Command... (1 Reply)
Discussion started by: nix-kid
1 Replies

10. Shell Programming and Scripting

read a file as input and pass each line to another script

Hi, I am trying to write a ftp script which will read a file for filenames and ftp those files to another server. Here's my ftp script, but it's scanning the current directory for file names. My question is how can I pass multiple files (these files will have the name of data files that need to... (0 Replies)
Discussion started by: sajjad02
0 Replies
Login or Register to Ask a Question