Shell Programming and Scripting

I'll have to step in here and ask everyone to please stop arguing over whether bash, Perl, Python, Ruby, PHP, Whitespace, or BF (sorry if I forgot to list your language of choice here) is better. The OP asked for opinions specific to Perl. If you can answer that, good. If you only answer to provoke someone thinking different than you, stop it or infractions will be handed out.

Starting now, I'll give out infractions, and remove every post not directly relating to the first post in this thread. If you want to argue, do it via PMs.

Okay, so let me explain my position again:
Why do you want to convert these scripts in the first place. Performance ? Reliability ? Portability ? Curiosity ? Fun ?
Bash like ksh93 which I prefer and other shells are scripting languages which can be used both interactively and not. (Bourne) Shell syntax has been defined several decades ago and is doing well. Knowing it is kind of something mandatory when you work with Unix. Shells features have increased within the years but shells are still following the Unix philosophy where specialized programs using stdin/stdout/stderr and error codes are used to build complex tasks, instead of having a one fit all approach.
Perl on the other hand isn't designed to be used interactively and certainly cannot be used as a login shell (of course you can use a program written in any language including perl as your login shell, but that's off topic. plsh is written in perl but isn't perl itself). Perl was initially designed to be an improvement over sed and awk which is fine. IMHO, it has derailed when becoming a general purpose programming language.
Unlike shells which leverage them, Perl doesn't encourage to reuse available external utilities. Everything is simpler when using a dedicated perl library explaining the large span of perl modules available. This is an advantage when running in a Unix hostile environment like Windows but I don't see the point in building such a self-centric ecosystem on Unix / Gnu/Linux.
Compared to competing general purpose languages, perl is pretty famous for its cryptic syntax. Someone with a little background with programming languages can read and understand code written in other programming language he doesn't know with very little learning or even not at all. Perl is a big exception. It has many unique idioms you cannot guess or easily find in its documentation. While this might not be an issue for code you write and maintain yourself, this can prove disastrous for code that need to be updated by someone else while the original author is no more available years after that. Almost anyone can pick an old sysadmin or whatever shell script and adapt it to suit new needs. With perl, unless the code has been carefully written and commented to avoid that risk, the more productive way might be to rewrite the whole from scratch when nobody is able to understand it. That's what I meant with negative productivity, I wasn't referring to performance which is a different point. If you really demand performance, just use C for the critical parts. If you want cross platform portability, I would strongly prefer Java which is much more readable and has an even larger community and industry support.

To summarize, I see perl as an addictive/elitist language which doesn't that much worth converting existing scripts to. On the other hand, like most languages, it can be useful and efficient for some specific tasks and might still be a good choice for simple scripts called from the shell.

By the way, I'm not a mod here.

More and more of what I'm doing is advanced manipulation of data, so I'm using awk/sed/grep/whatever to bend my data and do things with it, I just have this nagging feeling with bash that I'm reaching the end of it's capability (and what it was intended for) and I may need to go some direction. I learned C a little bit and also python, C seemed like you had to write a lot more code (definitely just my opinion, don't know whether it's based on anything) to get a given job done. Python was nice, but I have this nagging feeling it wasn't going to scale down to bash and also up to web apps, at least not yet, though I guess there's a pretty active community hacking it right now. I like Java, but I'm not sure it has the flexibility of perl (another opinion of mine, possibly based on nothing), so I thought I'd possibly learn and use perl for a couple years, get some things done with it, and then look at python/java/whatever again. My theory is that the perl code I'd build could be made to "interface" with whatever language I may choose then, so I wouldn't lose anything, as long as I comment whatever I'm doing in my perl scripts. Feel free to correct any of my assumptions, I don't want to waste a couple of years. The comments so far have been VERY helpful, I appreciate both sides very much, it helps me make a better choice.

You're not, really. A lot of the places you're using awk and so forth you're doing things that could be done with bash builtins, and you're still making some beginner mistakes like cat instead of redirection. If you don't mind, I'll improve your script a little. Give me a bit.

---------- Post updated at 11:21 AM ---------- Previous update was at 10:55 AM ----------

Code:

# Don't need cat here.  See http://www.partmaps.org/era/unix/award.html
# Don't need awk either.  Bash is perfectly capable of splitting its own input.
for FILE in "${@:-$ll}"
do
        while IFS=";" read lat lon
        do
                # You can also do operations more compactly like this
                ((ref_num++))

                url="http://ws.geonames.org/findNearbyPlaceName?lat=$lat&lng=$lon"
                curl -s "$url" > xmlfile
                # Could maybe have xmlstarlet print both arguments instead of 
                # running it twice?  Not completely sure.
                city=`/usr/bin/xmlstarlet sel -t -m //geoname -v toponymName xmlfile`
                country_code=`/usr/bin/xmlstarlet sel -t -m //geoname -v countryCode xmlfile`

                # We do not need to run mysql N files * M rows times!  One instance can run EVERY query.
                # We're using cat here so we can use a here document for easier formatting.
                # Also note where I substitute ' for \' in case of funny-named cities.
                cat <<EOF
INSERT INTO somedb.sometable (id, ref_num, timestamp, lat,
        lon, city, country_code)
        VALUES (NULL,'${ref_num}','${lat}','${lon}',
        '${city//\'/\'}','${country_code//\'/\'}');
EOF
        done < "$FILE"
done | mysql --batch -u $USER_NAME --password=$PASSWORD -D "somedb"

You can do a lot with pipes. Bourne shell lets you put them wherever. The trick is to not to use them for one-liners if you can help it, they're inefficient that way, but excellent for manipulating arbitrary amounts of data -- connect long-running processes or code sections with them, not pipe a single word through sed. Here we're creating an entire list of insert operations and feeding it into one instance of mysql instead of running mysql N*M times. This is important since process creation times can be significant compared to other operations.

This is what I'd consider the biggest difference between shell scripting and perl, one of shell's most powerful features, and one of the last things people grasp about shells. To get a pipe in perl you have to make an explicit open call, it doesn't come naturally. Nor does it understand redirection and pipes on the statement or code block level and so forth.

thanks for the tips, this is why I support the site, it's a great place to learn from the pros who've been there/done that