If the script you came up with is working for you
then it's quite ok.
But because you asked maybe just some suggestions
(although this is probably only a matter of personal taste)
I think here you mixed up single quotes (embracing the whole ps command)
with backticks.
Though it may look daft it is quite ok to have nested backticks.
Nevertheless, I am not particularly fond of backticks because they have become obsolete in modern posix shells,
and thus their use is deprecated.
With bash, ksh, zsh, hp-ux sh etc. you should use $(...) instead.
So this would be how I would rewrite it

Code:

if (( $a < $(ps -p $(pgrep SpectroSERVER) -o rss=)) )); then

But this is still quite ugly.
With all those pairs of parens one easily could lose counting the closing ones.
So better split it up into several statements.
Also, what if the inner cmd expansion of pgrep fails?
Also, as you are going to calculate with these vars
you should better declare them as integers.
Also better use meaningful, imaginative variable names
Maybe one could rewrite it this way?

Code:

declare -i RSS_THRESHOLD=3942400
spectro_pid=$(pgrep SpectroSERVER)  # does its cmd name really appear in mixed case in the proc table?
[[ -z $spectro_pid ]] && exit 1
declare -i spectro_rss=$(ps -p $spectro_pid -o rss=)
if (( RSS_THRESHOLD < spectro_pid )); then

On the other hand you could parse the RSS in one proc table lookup like so

Code:

spectro_name=SpectroServer
set -- $(ps -e -o pid,ppid,rss,comm|awk -v pn=$spectro_name '$4~pn&&$2==1{print$1,$3}')
declare -i spectro_pid=$1 spectro_rss=$2

Yet another variant was to retrieve the RSS from the procfs like

Code:

spectro_rss=$(awk '$1~/^VmRSS:/{print$2}' $(printf /proc/%lu/status $(pgrep -P1 $spectro_name)))

These shall only serve as examples that there is more than one way to do it
(viz. TIMTOWTDI, spoken "timtowdy" as the Perl hailers use to say)
It's not clear to me how this should work
since it ought to produce a syntax error
though I think I can gather what the intension is;
viz. tail on the log until a certain expression appears
and then assign this line to variable b, right?