Shell Programming and Scripting

I agree with both of you regarding your comments and i will post a more detailed response shortly.

---------- Post updated at 01:35 PM ---------- Previous update was at 01:34 PM ----------

Now i have to post another response to allow me to send URL's in my detailed response :-)

---------- Post updated at 01:38 PM ---------- Previous update was at 01:35 PM ----------

ok more detailed coming next this forum software must have a bug as i have edited the offending URL add in my detailed post and it still will not let me post. Lets try again.

---------- Post updated at 01:40 PM ---------- Previous update was at 01:38 PM ----------

Ok here is what i have gone with now;

Code:

#!/bin/bash
clear
srvice=("plexmediaserver" "plexconnect" "plexpy" "nzbdrone" "couchpotato" "headphones" "utserver" "qbittorrent-nox" "HTPC-Manager" "sabnzbdplus" "webmin");

for i in "${srvice[@]}";  do
   systemctl is-active $i > /dev/null 2>&1
   state=$(systemctl is-active $i >/dev/null && echo Alive || echo Dead)
   #state="${state/failed/inactive}"
   printf "%-18s : %-20s\n" "${i^}" "${state^}"
done

---------- Post updated at 09:15 PM ---------- Previous update was at 01:40 PM ----------

Interestingly I have found that much of the script isn't needed with conky, however, it has been a valuable learning experience.

---------- Post updated at 09:16 PM ---------- Previous update was at 09:15 PM ----------

Hi bakunin, thank you for your comments and suggestions. I agree with your comments. I tend to notarize my scripts so that years later I do understand what I was doing at the time when I put the script together and also as technology evolves I can update it and make it better or more streamlined or reduce the number of processes as Stomp demonstrated.

Being reasonably new to the NIX platform and coding within it and I don't often need to do a lot of coding, I thought I would attempt to put good scripts together from the outset. I also have to mentioned of all the scripting languages I have used over the years BASH is fantastic and fast becoming my favorite - I do also like JavaScript.

Thanks for the comments and examples using sed - i didn't know you could do that - very valuable and i will be using that in future also.

Regarding printf - whilst putting this script together, i have been doing a lot of reading and looking at different examples around the internet and have found this command to be very powerful and prefer to keep scripts as super simple as possible and certainly where possible would prefer to use printf for formatting rather than adding another process of command into the logic. I was having trouble getting the syntax correct using printf (obviously my lack of knowledge and understanding) hence the reason I first attempted with sed.

Sorry, that is not a bug but a feature: as a means to fight spammers who occasionally swamped the forum with links for "goods of questionable reputation" (fake passports, viagra pills, ... take your pick) we took away the right to post links for users below a certain post count. Once you are over that threshold (and it is going to be soon) you can post links without problem. This way we drove the worst spammers away and with the rest we can deal. Sorry for this inconvenience, but there we foud no better alternative.

Thank you, i am glad i could be of help.

Thinking again, here are some other tips and a response to stomp:

First: yes, one needs a lot of discipline. It turns out that the "freedom" the shell grants you (like not having to declare your variables, etc.) is best not used at all. One should write shell as if it was just another high-level language (like C or FORTRAN - are there any others?). This is why i prefer Korn Shell over bash, why i always declare my variables before i use them and why i write like shell variables were as typed as C variables.

In C it is possible to create a string, use it as an integer and - if the value just happens to match - as a pointer thereafter. This "clever programming" usually gets you into deep kimchi sooner than later. The same is true for shell scripts. I think the "obfuscated C contest" is a funny pastime to show off your skills - but whatever you write professionally should be the direct opposite of that: clear, self-evident, easy to read.

Itmaywellbethatyoucanstilldecipherwhatiwritethisway but perhaps employing this style consistently would not help to make me your favourite author.

About the topic of scripting versus programming (high.level languages): both have their place. As a systems administrator you need all kinds of administrative scripts/procedures and because i like my procedures to be as reliant as possible probably 80% of my code deals with error handling and logging. If you have an environment with ~400 systems (my last project) and write a script to create user accounts you check:

- if the user already exists (and, in case it doesn't)
- if the user ID is already taken on that server
- if the user name is already taken
- if the parent directory for the intended HOME already exists
- if there is enough room to create the HOME
- add other possible problems here ad libitum....

And my script checks all these on every server before even attempting to create the account. A script doesn't need to be able to deal with all these problems, but if on server 347 the creation has failed i rather read a log entry of user name "foobar" already taken, cannot continue than "mkuser error: stop"

All the tasks above are easy to do in script and, after doing all these checks, writing a documentation in the header, etc., the final script (yes, i use this every day, upon request i'll post it) is about 550 lines long (i make heavy use of a self-written library of external shell functions, in bash these functions would add another ~1000 lines of code). But the same in some highlevel language would be more closely to 10k lines and would take several months to write. This was written in about 3 days.

It is, IMHO, not about the size of a programming project, but its scope, that will determine the best-suited language. Like nobody would write a bookeeping software in assembler (not any more) and nobody would write number-crunching programs in anything else than FORTRAN, nobody would write application programs in script or administrative scripts in any high-level language. Its just not the right tool for the task.

To sum up about script programming: take it serious. Take it as serious as you would write any other piece of software because that in fact it is. With a different scope and different means, but the principles of good software engineering apply here no less than in any other environment. Even more so because the language doesn't enforce any strict work so that you have to make up with discipline what the language lacks in strictness.

I already stressed to comment your code: declaring your variables gives you the opportunity to describe their contents, which i always find helpful. Consider this code:

Code:

#! /bin/yourshell

typeset    chLine=""                      # line to process
typeset -i iArrCnt=1                      # counter to current array element
typeset    fIn="/bla/foo/somewhere"       # default input file
typeset    fOut="/bla/foo/whatever"       # default output file

# main ()
... rest of code

You might not like the hungarian style notation i adopted and stuck with but suppose you get this script in hand and should change something: it might be helpful to understand immediately what in which variable is supposed to be. It is not relevant which standard you use - but use any one (even your own) and stick with it consistently.

My own style (but that is personal preference more than anything else and heavily influenced by old habits from years of programming in assembler and FORTRAN) is to declare all local variables at the start of each function, put comments at position 50 of the line and never let a line be longer than 80 characters if i can avoid it. I use 5 spaces indentation and write my loops and control-structures in one line:

Code:

while [ "$bla" = "$foo" ] ; do
     this "$bla"
     that "$foo"
done

case $bla in
     X)
          this X
          that Y
          ;;

     *)
          bla
          if [ -z "$whatever" ] ; then
               whatever
          else
               something ELSE
          fi
          foo
          bar
          ;;

esac

But it doesn't matter so much what you do as long as you do it consistently. Programming is about organising your thoughts in the strictest possible way and your code should show this organisation.

Finally: Unix programs should maintain some tenets of interoperability. For instance, your script:

Code:

#!/bin/bash
clear
srvice=("plexmediaserver" "plexconnect" "plexpy" "nzbdrone" "couchpotato" "headphones" "utserver" "qbittorrent-nox" "HTPC-Manager" "sabnzbdplus" "webmin");

for i in "${srvice[@]}";  do
   systemctl is-active $i > /dev/null 2>&1
   state=$(systemctl is-active $i >/dev/null && echo Alive || echo Dead)
   #state="${state/failed/inactive}"
   printf "%-18s : %-20s\n" "${i^}" "${state^}"
done

Suppose you would use this script in some sort of pipeline or with a redirection to a log file:

Code:

script.sh | ...
script.sh >/path/to/log 2>&1

Don't you think that clear-statement, as useful as it might be in interactive use, might maybe disturb the processing? I would leave that out for exactly this reason.

Another topic: strive to adhere to the POSIX standard as much as you can. This is sometimes cumbersome and bound to additional effort, especially when you use Linux, because there are lots of nice little extras in many commands that come in handy. Still, it might limit the usability of your script to a certain platform i you use the extensions this platform offers for the standard. But using the bare standard every time is oftenly rewarded when you take a script from platform X, copy it to a completely different platform and it runs without any change.

OK, enough for today and happy scripting.

I hope this helps.

bakunin

Hi bakunin,

Thanks for your time and comments - they are welcomed :-)

I did have a go at some more structured programming in formal programming in Pascal, C and basic (as well as some others) languages some 25 years ago, however, I didn't do anything series with the skills. These days whilst I enjoy the scripting aspect of having a problem and creating a solution to deal with it, I would tend to stick to scripting, however, I am always looking at streamlining and clean simple structure with my scripts. You have certainly given me some things to consider and remember
I was not aware of KSH - so on your mention i went and had a quick read - I like it! So i have installed it and will start doing some scripting using korn and see how I go :-)

I may have to ask you for some more tips from time to time in the future ;-)

All the best for the new year!

Cheers,
Darren

Yes! One more converted! ;-)

Seriously: i am glad you like it. What i appreciate most about Korn Shell is the possibilty to set the variable FPATH, which works similar to PATH - though not for binaries but shell functions. This way you can designate a directory with functions written in shell which you can use from any of your scripts. This way i built a "library" over the years with functions for repetitive tasks: writing a log, create/remove space for temporary files, check if the script is run with root authority ....

You are welcome. Ask as much as you wish. Here is a tip you haven't asked: Barry Rosenberg, "KornShell Programming Tutorial" and "Hand-On KornShell93 Programmming". These are not only very good books about the topic but also great fun to read. You might want to give them a try.

I hope this helps.

bakunin

Sounds good i am going to get into some more scripting over the next few weeks ;-)

The important thing i like about KSH is POSIX compliant!


Funny you mention books I was looking at a few online today I will add those to my book list to pick up in the near future.

Tell me bakunin, have you or do you use Conky or perhaps something else that you may suggest I look at. I have settled on Lubuntu as the Linux platform I prefer (at least at present). I have a number of Virtual machines that I run and Lubuntu if very good with low resource allocations. In the past, I have played with Solaris (when it was SUN and I liked it a great deal as a Unix Platform) and I did have a have a passing look at SUSE before Novel got hold of it.

Anyway, I digress - how this original script thread started was I was looking to use Conky to monitor some of the services I use on my HTPC Lubuntu system to so at a glance I can see if there is anything I need to give attention to...

And Stomp and other visitors are now reading and starting to roll their eyes as we have really gone off topic

Hi @bakunin

Ok, I have a new scripting requirement that I wanted to run by you...

I have a 3rd party script that pulls metadata from imdb.com and gives it to me in a clean text format. Now I need to modify the format and some of the fields to make the data ready for the next process I want to use it for - that is applying it to a mp4 file to so that my iTunes reads it correctly.

No, the question is - script in bash, ksh, sh or python? I think I would need to use grep or sed to manipulate the string i get from the 3rd party script output and if your suggestion is bash/ksh/sh then what do you suggest - grep or sed r a combination of both? OR do you think it could all be down via shell?

look forward to your thoughts

PS for anyone else reading, your thoughts and comments are welcome and sorry for going off topic - I would have sent bakunin a PM, however, i haven't posted 10 times yet...

Cheers,
Darren

---------- Post updated at 08:52 PM ---------- Previous update was at 08:30 PM ----------

Oh and I just thought about stomps comments - please see what I am attempting to accomplish below.

this is an output example I get from the 3rd party script (GitHub - bgr/imdb-cli: Command line tool for retrieving IMDb movie information)

Code:

Title:Deep Throat
Year:1993
Rated:TV-14
Released:17 Sep 1993
Season:1
Episode:2
Runtime:46 min
Genre:Drama
 Mystery
 Sci-Fi
Director:Daniel Sackheim
Writer:Chris Carter (created by)
 Chris Carter
Actors:David Duchovny
 Gillian Anderson
 Jerry Hardin
 Michael Bryan French
Plot:Mulder and Scully investigate the mysterious case of a military test pilot who disappeared after experiencing strange psychotic behaviour.
Language:English
Country:USA
Awards:N/A
Poster:http://ia.media-imdb.com/images/M/MV...._V1_SX300.jpg
Metascore:N/A
imdbRating:8.3
imdbVotes:3385
imdbID:tt0751099
seriesID:tt0106179
Type:episode
Response:True

and this is what I need to make it look like so that I can use FFmpeg to apply it to the mp4 file

Code:

;FFMETADATA1
major_brand=qt  
minor_version=512
compatible_brands=qt  
title=This Guilty Blood
album=Shadowhunters, Season 2
genre=Action
track=1/10
disc=2
date=2017-01-02
synopsis=Only hours have passed since Jace left with Valentine and all hell has broken loose at The Institute. Alec, Isabelle and Clary are desperate to find Jace, but are quickly stopped in their tracks with the arrival of Victor Aldertree, who means business about getting the Institute back on track. But getting the New York Shadowhunters in-line with The Clave may be counterintuitive to Alec, Isabelle and Clary's plan to rescue Jace. Meanwhile, Jocelyn has a lot to catch up on now that she is awake.
iTunEXTC=us-tv|TV-14|500
iTunMOVI=<?xml version\="1.0" encoding\="UTF-8"?><!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"><plist version\="1.0"><dict>\
<key>cast</key><array>\
<dict><key>name</key><string>Katherine McNamara</string></dict>\
<dict><key>name</key><string>Dominic Sherwood</string></dict>\
<dict><key>name</key><string>Matthew Daddario</string></dict>\
<dict><key>name</key><string>Alberto Rosende</string></dict>\
<dict><key>name</key><string>Isaiah Mustafa</string></dict>\
</array>\
<key>screenwriters</key><array>\
<dict><key>name</key><string>Michael Reisz</string></dict>\
</array>\
<key>directors</key><array>\
<dict><key>name</key><string>Matt Hastings</string></dict>\
</array>\
</dict></plist>\

media_type=10
show=Shadowhunters
episode_id=This Guilty Blood
season_number=2
episode_sort=1
network=Freeform
hd_video=0
description=Only hours have passed since Jace left with Valentine and all hell has broken loose at The Institute.
encoder=Lavf57.25.100

Please start a new thread to discuss a new topic (such as your new script) instead of just using one thread to discuss several topics. (Using a single thread makes it hard for readers to figure out to which topic future posts refer.)

And, please, when showing sample input and sample output, make the sample output be output that you want to produce when given the sample input you showed earlier. I don't see any clear way to transform input data for a show named Deep Throat into output for a show named ShadowHunters and assume that is not what you expect your script to do. With the examples you provided in post #13 we have no way to determine whether output fields like major_brand=qt and compatible_brands=qt are constants to be provided by the conversion process or are derived from data present in some input but not provided in others.

Please do not continue discussion on post #13 in this thread!