Visit Our UNIX and Linux User Community


On how to select the right tool for a given task


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting On how to select the right tool for a given task
# 22  
Old 10-20-2008
Quote:
Originally Posted by ghostdog74
like i said, if time is a concern, i won't write shell scripts.
Yes and no. Shell language has something compiled code might lack: architecture independence. A shell script (well: most shell scripts, there might be exceptions) won't care about being run in 32-bit or 64-bit environments, about being run on certain OS versions (try that with a program relying on some shared libraries), etc.

Quote:
Here's an awk version (which i hope its correct), if you consider awk an "external tool" as well.
Of course you are not wrong. What was discussed about mixing external programs and shell constructs. Of course you can solve this example completely in awk, but wether you write in awk and call shell functions via the system() subfunction call or you write in shell and call awk as an external program you pay dearly for it because the exec() and fork() calls which are necessary.

As i have understood cfajohnson and perderabo this was their concern, not which tool was faster if you never leave it.

bakunin
# 23  
Old 10-20-2008
Quote:
Originally Posted by ghostdog74
if real time is a concern, i don't write shell scripts.

If real time is a concern, using shell constructs, instead of external commands, can make a great deal of difference.
# 24  
Old 10-20-2008
Quote:
Originally Posted by cfajohnson

If real time is a concern, using shell constructs, instead of external commands, can make a great deal of difference.
An example to keep things in perspective:

If you need to parse a huge file and you need the results fast, would you use awk or the shell's construct to do it?
# 25  
Old 10-21-2008

To parse a large file, I would generally use awk (or occasionally sed).

If I needed to do operations on every line that could not be done with awk or sed internals, and required calling other commands, I would probably use the shell.
# 26  
Old 10-21-2008
My personal perspective:

1) I am not very good at memorizing stuff, nor do I think the human brain is optimized for that task. I'm much better at making inferences and deductions instead.
2) I know what parameter substitutions are -- in general. But I can't bring myself to remember the difference between
Code:
${parameter##pattern}
and
${parameter%%pattern}

Everytime I see one of those, I have to open a terminal and make a small test to figure out which is which. And thats only an example, there are many other similar looking constructs that do not give the slightest hint to what they actually do. My memory is weak, and when I can't use my deductive and inferential powers, I end up wasting time figuring out what they do. A classic example, if you only knew minimal bash, how much time do you need to understand what this does:

Code:
path="/home/neked/testfile"
s=${path##*/}

versus:

Code:
path="/home/neked/testfile"
s=$(basename $path)

The second code provides some semantics for you to infer what the code does. The first one relies on your memory. The brief way to sum my point is that parameter substitutions are NOT easily readable.

3) Even if you were a parameter substitution guru, and you used meaningful variable names and comments to make clearer what your parameter substitution tricks do, then future maintainers of the code might not be the same. This bit me a couple of days ago: I had to spend 20 minutes debugging a bash script riddled with those parameter substitution scripts. I estimate I would've spent closer to 5 minutes if the code was written using more obvious external tools (sed, basename, awk). This is about 15 minutes of human time wasted in order to save less than a few milliseconds of CPU time. Especially since the whole script runs only once a night, and does not exceed 0.030 seconds runtime on my modest 7 years old computer.

4) The conclusion for me is that parameter substitution should only be used when and only if the need arises. Anything else is premature optimization at the cost of more developer hours debugging and maintaining the code. If I have a script that takes 4 seconds to execute, which could be optimized into running within less than a second using parameter substitution, that still would not -- on its own -- make a convincing case to use param substitution. For a convincing case to be made, the need to reclaim the additional seconds of CPU time must be established and weighed against the loss in human seconds needed to maintain and develop the code.

Last edited by neked; 10-21-2008 at 04:50 PM..
# 27  
Old 10-21-2008
Quote:
Originally Posted by neked
3) Even if you were a parameter substitution guru, and you used meaningful variable names and comments to make clearer what your parameter substitution tricks do, then future maintainers of the code might not be the same. This bit me a couple of days ago: I had to spend 20 minutes debugging a bash script riddled with those parameter substitution scripts. I estimate I would've spent closer to 5 minutes if the code was written using more obvious external tools (sed, basename, awk). This is about 15 minutes of human time wasted in order to save less than a few milliseconds of CPU time. Especially since the whole script runs only once a night, and does not exceed 0.030 seconds runtime on my modest 7 years old computer.
I've just substituted all occurrences of parameter substitutions with equivalent sed and basename commands in the script I refer to above, the execution time went up from an average of 0.030 seconds to 0.061 seconds. That means parameter substitution use shaved an average of 0.03 seconds per run. Considering that I've had to spend roughly 15 more minutes understanding the code to debug it, this means the code has to run 15*60/0.03 times to pay off the extra time I invested to debug it. That's 30,000 times. Considering that it runs once a night, this means we'll have to wait more than 80 years!

Even after 80 years, those 0.020 seconds shaved off CPU time per night are not as worthy to me as 15 minutes of my time which I could've spent aternative activity redacted

Last edited by Perderabo; 10-21-2008 at 05:49 PM.. Reason: clean up language
# 28  
Old 10-21-2008
Quote:
Originally Posted by neked
Even after 80 years, those 0.020 seconds shaved off CPU time per night are not as worthy to me as 15 minutes of my time which I could've spent aternative activity redacted
Ironically, this post sounds more interesting after the censorship. Smilie

Previous Thread | Next Thread
Test Your Knowledge in Computers #36
Difficulty: Easy
Moving processes from the main memory of a memory-constrained Linux computer to disk is called 'swapping'.
True or False?

9 More Discussions You Might Find Interesting

1. Homework & Coursework Questions

[HELP] Easy task

I have a simple task for my school work. I'm new with unix, so i need help. I need to write a scenario. Task is. From created txt file read first 3 words and create a 3 catalogs with those 3 words. 2 of those new catalogs should be transferred to other directory. If someone could help me just... (1 Reply)
Discussion started by: justynykas
1 Replies

2. Shell Programming and Scripting

Task

Hi experts, I have a problem with the below shell task: I need to modify the file creatin a paired row , per each row which matches filter (e.g. number of nonempty columns = 5) Output should look like this: second row is original one from the input, first row(red) is pairing row, it's... (29 Replies)
Discussion started by: hernand
29 Replies

3. Shell Programming and Scripting

task

Hi all, I'm newbie and stuck here. Thanks for any help. Input(txt file) a b X c d Y e f Z g h W Requested output: a b X Y c d Y X e f Z W g h W Z Please use code tags when posting data and code samples! (10 Replies)
Discussion started by: hernand
10 Replies

4. Shell Programming and Scripting

last task for my script

hi, infile- create table salary ( occupation_code char(40), earnings decimal(10,2), occ_yearend integer ); outfile- salary:create table salary salary:( occupation_code char(40), salary: earnings decimal(10,2), salary: occ_yearend integer salary:); Thanks. (4 Replies)
Discussion started by: dvah
4 Replies

5. Boot Loaders

Reboot and Select Proper Boot device or insert Boot media in select Boot device and press a key

Hello, I have kubuntu on my laptop and now I decided to switch to Windows 7. I made the bios settings properly (first choice is boot from cd\vd) but I see the error " reboot and select proper Boot device or insert Boot media in select Boot device and press a key " I have tried CD and... (0 Replies)
Discussion started by: rpf
0 Replies

6. Programming

Help with task daemon

believe it or not but this is my first c program (i've worked with java, C#, php though) I am trying to make a daemon that checks if mplayer is running(it's for a projection room) and if it is not then to run mplayer with a file.. So far it's not working and I don't know why Help and comments... (5 Replies)
Discussion started by: james2432
5 Replies

7. Shell Programming and Scripting

Parse an XML task list to create each task.xml file

I have an task definition listing xml file that contains a list of tasks such as <TASKLIST <TASK definition="Completion date" id="Taskname1" Some other <CODE name="Code12" <Parameter pname="Dog" input="5.6" units="feet" etc /Parameter> <Parameter... (3 Replies)
Discussion started by: MissI
3 Replies

8. Shell Programming and Scripting

comment and Uncomment single task out of multiple task

I have a file contains TASK gsnmpproxy { CommandLine = $SMCHOME/bin/gsnmpProxy.exe } TASK gsnmpdbgui { CommandLine = $SMCHOME/bin/gsnmpdbgui.exe I would like to comment and than uncomment specific task eg TASK gsnmpproxy Pls suggest how to do in shell script (9 Replies)
Discussion started by: madhusmita
9 Replies

9. UNIX for Dummies Questions & Answers

process vs task

Hi, I am new to this forum and unix too. I have just started learning unix. As I was going through the first chapter, I read that unix is multitasking, multiprogramming, multiprocessing and multiuser OS. My question is: Is there any difference between a TASK and a PROCESS. How are PROCESS... (2 Replies)
Discussion started by: hana
2 Replies

Featured Tech Videos