String manipulation for sheer speed...


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers String manipulation for sheer speed...
# 1  
Old 05-25-2013
String manipulation for sheer speed...

Hi all...

I have not forgotten the Scope project as this has been bugging me and WILL be part of it...
(Note that the 8192 byte binary file can be any binary file of your choice but the values "m" and "n" will vary.)
They are correct in the demo and correct in the _finished_at_present_ product, but......

I am having one hell of a problem trying to beat 10 seconds to scan an 8KB binary file for the scope. A derivative of this WILL be used to determine the polarity of the DC component of a signal... However I WAS trying to make it usable as an Audio Frequency Counter.

I HAVE successfully done it for Python 2x and 3x and reads the frequency to an accuracy of 0.1 per cent but Shell scripting - time wise ONLY - has so far eluded me but the accuracy is still the same.

Code:
#!/bin/bash

# "decimalstring" is a 24576 byte contiguous 3 byte decimal
# value from 255 to 000 generated from an 8KB binary file...
decimalstring=`hexdump -n8192 -s0 -v -e '1/1 "%03u"' 8KB.dat`

splicedstaing=""
somestring=""
n=0
m=0

# printf "$decimalstring" > 24KB.txt
# printf "$decimalstring"

printf "\nStart!"

# Start position of slow loop.
for splicedstring in $( seq 0 3 24575 )
do
	somestring="${decimalstring:$splicedstring:3}"
	if [ $somestring -ge 128 ]
	then
		m=$[ ( m + 1 ) ]
	fi
	if [ $somestring -le 127 ]
	then
		n=$[ ( n + 1 ) ]
	fi
done
# End position of slow loop.

printf "\n\nDone!, m=$m, n=$n...\n\n"

This is the result on screen and IS correct but it takes 10 seconds to get there... ;o(

Removing the comment and space from the line......
Code:
# printf "$decimalstring"

......shows how fast the resulting 24KiB file is printed to screen...

The results on my Macbook Pro:-
Code:
Last login: Sat May 25 17:26:11 on ttys000
Barrys-MacBook-Pro:~ barrywalker$ ./decimal_string.sh

Start!

Done!, m=4096, n=4096...

Barrys-MacBook-Pro:~ barrywalker$

I have tried other methods but this has shaved at least 4 seconds of off the others...

A pointer in the right direction for any other method to speed this up at least 10 fold between START and DONE would really be appreciated...

TIA...

Bazza...
# 2  
Old 05-25-2013
If you are convinced that the 3 digit substrings in "$decimalstring" are in the range 0 through 255, the loop should run faster if you replace the two if statements with:
Code:
	if [ $somestring -ge 128 ]
	then
		m=$[ ( m + 1 ) ]
	else
		n=$[ ( n + 1 ) ]
	fi

Have you compared the speed of $[ ( m + 1 ) ] versus $((m+1))?
This User Gave Thanks to Don Cragun For This Post:
# 3  
Old 05-25-2013
Hi Don Cragun...

Tried both separately and together...

The "if... then... else... fi" made no difference...

The "$((m+1))" seems to have shaved another 1/2 to 1 second, now down is single figures... ;o)
So it IS about 5 to 10 per cent faster...

Thanks for the heads up, hopefully there will be more ideas come my way but after using Python for the same thing I didn't realise how slow the shell could be in real time stuff...

The second part is now included... ;o)

Bazza...
# 4  
Old 05-25-2013
I could chop off another 5 - 10%, but far from cutting in half. As you seem to use a recent bash, this might work for you:
Code:
for splicedstring in $( seq 0 3 24575 )
  do a=$((10#${decimalstring:$splicedstring:3} & 128 >> 7))  # ANDing out bit 7; shift right to bit 1; result is 1 or 0
     m=$((m+=a ))                                            # increment m if bit was set (number >= 128)
     n=$((n+=(1-a)))                                         # increment n if bit was clear
  done

This User Gave Thanks to RudiC For This Post:
# 5  
Old 05-26-2013
Hi RudiC...

Liked the look of the code and tried it out...

Worked as expected but it was no faster than Don's reply.

But I do like the idea... That is logged in the old grey matter for future use...

Thanks...
# 6  
Old 05-26-2013
bash has a well-deserved reputation for being slow. A few years ago, its poor performance led Ubuntu Linux to abandon it for boot scripts (although bash remains the default interactive login shell for users).

If you aren't wedded to bash, and if the scripts in question aren't brimming with bashisms, you could try running them under ksh (which was installed by default on OS X when I last used the OS, a few years ago).

Regards,
Alister
This User Gave Thanks to alister For This Post:
# 7  
Old 05-26-2013
If you use ksh instead of bash, you can also use the built-in {init..limit..inc} instead of spawning a separate process to evaluate $(seq init inc limit}. I.e.:
Code:
for splicedstring in {0..24575..3}

This User Gave Thanks to Don Cragun For This Post:
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. OS X (Apple)

String manipulation

i have a string that am looking to extract all characters following 3 consecutiv numbers. Example my string is J1705PEAN038TDMN, i need to get TDMN My string can have multiple 3 consecutive numbers, i need what follows last occurance (9 Replies)
Discussion started by: gigagigosu
9 Replies

2. Shell Programming and Scripting

String Manipulation

I'm making a little game in Perl, and I am trying to remove the first instance of a character in an arbitrary string. For example, if the string is "cupcakes"and the user enters another string that contains letters from "cupcake" e.g: "sake"the original string will now look like this (below)... (3 Replies)
Discussion started by: whyte_rhyno
3 Replies

3. Shell Programming and Scripting

String manipulation

Hi , I am getting a string like aaa,bbb,sdsdad,sdfsdf,sdfsdfdsf,rtyrtyr,45654654,ddfdfdfgdfg,dfgdfgdg........... Now what I need is to format it. So after each nth comma I need one newline. So the above will look like when n=3 aaa,bbb,sdsdad, sdfsdf,sdfsdfdsf,rtyrtyr,... (4 Replies)
Discussion started by: Anupam_Halder
4 Replies

4. Shell Programming and Scripting

Deleting part of a string : string manipulation

i have something like this... echo "teCertificateId" | awk -F'Id' '{ print $1 }' | awk -F'te' '{ print $2 }' Certifica the awk should remove 'te' only if it is present at the start of the string.. anywhere else it should ignore it. expected output is Certificate (7 Replies)
Discussion started by: vivek d r
7 Replies

5. Shell Programming and Scripting

string manipulation

Hi, I have the followoing details in one file: opt/tra/domain/test/new/filename1 training/ear/help I need to manipulate the string in the following manner: filename1= opt/tra/domain/test/new/filename1 help=training/ear/help last string is the name and equal sign and then... (2 Replies)
Discussion started by: ckchelladurai
2 Replies

6. Shell Programming and Scripting

Help With String Manipulation

Hi Guru's, I need some help with data manipulation using shell scripting. I know how to replace the whole string but not part of the string. The value after aa= should be replaced with the value in the mail leaving ,OU=111,OU=222,DC=333 as is. Below are the inputs and expected outputs. Input:... (17 Replies)
Discussion started by: Samingla
17 Replies

7. Shell Programming and Scripting

string manipulation

if I have two string variable, how do I add one to anther. like a= "a" b="b" c=$a+$b but that doesn't work. Is there anyway to solve it.http://www.qtl.co.il/img/copy.pnghttp://www.google.com/favicon.icohttp://www.babylon.com/favicon.icohttp://www.morfix.com/favicon.ico (2 Replies)
Discussion started by: programAngel
2 Replies

8. Shell Programming and Scripting

string manipulation

i have a file that contains a pattern like this: ajay 1234 newyork available kumar 2345 denver singh 2345 newyork ajay 3456 denver kumar 3456 newyork singh 3456 delhi available ajay 4567 miami kumar 4567 miami singh 4567 delhi i want to search for each line... (5 Replies)
Discussion started by: ajay41aj
5 Replies

9. Shell Programming and Scripting

String Manipulation Help

Hey Guys, Right i know how to alter a word to begin with a capital letter, i know how to remove unwanted characters and replace them with the relevant character however i don't now if there is a way to do them all in one line. Code: echo -n ${string:0:1} | tr a-z A-Z #convert first letter... (4 Replies)
Discussion started by: shadow0001
4 Replies

10. Shell Programming and Scripting

string manipulation

Hello, I have a korn shell string variable str1 = "A,B,Z" I would like to create another korn shell string variable str2 = "letter = 'A' or letter = 'B' or letter = 'Z' " Please help! Thanks in advance an UNIX newbie! (13 Replies)
Discussion started by: hai1973
13 Replies
Login or Register to Ask a Question