String manipulation for sheer speed...


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers String manipulation for sheer speed...
# 8  
Old 05-26-2013
Have you tried reading the data into an array which would allow you to avoid the substring overhead which I believe is costing you. Your original code ran in about 4 seconds for me using Bash and less than a second with Kshell. The code below, which reads and processes the array, runs much faster under both:

Kshell .07s
Bash .17s

Maybe I'm missing something but this seems to perform pretty well.

Code:
# "decimalstring" is an array of the first 8192 bytes from the input file
# (the variable name is now misleading, but I left it the same as in the original code)
decimalstring=(  $( hexdump -n8192 -s0 -v -e '1/1 "%u "' input-file )  )

n=0
m=0

printf "\nStart!"

# Start position of slow loop.
for (( i = 0; i < 8192; i++ ))
do
        if (( ${decimalstring[$i]} >= 127 ))
        then
                ((m++))
        else
                ((n++))
        fi
done
# End position of slow loop.

printf "\n\nDone!, m=$m, n=$n...\n\n"


The raw output:
Code:
spot:[/home/scooter/src/test]time bash tt18

Start!

Done!, m=2063, n=6129...


real    0m0.17s
user    0m0.17s
sys     0m0.01s
spot:[/home/scooter/src/test]time ksh tt18

Start!

Done!, m=2063, n=6129...


real    0m0.07s
user    0m0.06s
sys     0m0.01s


Last edited by agama; 05-26-2013 at 11:22 PM.. Reason: code tag broken
This User Gave Thanks to agama For This Post:
# 9  
Old 05-27-2013
Hi agama...

No, I hadn't even thought about an array but I tried your code and now it is near instant...
(Macbook Pro, 13", circa August 2012, default is bash.)

This has helped me no-end and now I have to apply it to the frequency counter...
When it is inside the frequency counter code you will be mentioned...

If I couldn't find an alternative to my original code then I WAS going to spawn a child process and pass the results back to the parent code in its own time...

Many, many thanks...

@ Don...
Using ksh and your snippet shaved 6.5 seconds off of my original code...

@ alister...
Thanks for the heads up I will take careful note although I am sure Corona688 mentioned this problem to me some time ago...

Thanks to all for your time...

Bazza...

Last edited by wisecracker; 05-27-2013 at 05:56 AM.. Reason: Typos...
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. OS X (Apple)

String manipulation

i have a string that am looking to extract all characters following 3 consecutiv numbers. Example my string is J1705PEAN038TDMN, i need to get TDMN My string can have multiple 3 consecutive numbers, i need what follows last occurance (9 Replies)
Discussion started by: gigagigosu
9 Replies

2. Shell Programming and Scripting

String Manipulation

I'm making a little game in Perl, and I am trying to remove the first instance of a character in an arbitrary string. For example, if the string is "cupcakes"and the user enters another string that contains letters from "cupcake" e.g: "sake"the original string will now look like this (below)... (3 Replies)
Discussion started by: whyte_rhyno
3 Replies

3. Shell Programming and Scripting

String manipulation

Hi , I am getting a string like aaa,bbb,sdsdad,sdfsdf,sdfsdfdsf,rtyrtyr,45654654,ddfdfdfgdfg,dfgdfgdg........... Now what I need is to format it. So after each nth comma I need one newline. So the above will look like when n=3 aaa,bbb,sdsdad, sdfsdf,sdfsdfdsf,rtyrtyr,... (4 Replies)
Discussion started by: Anupam_Halder
4 Replies

4. Shell Programming and Scripting

Deleting part of a string : string manipulation

i have something like this... echo "teCertificateId" | awk -F'Id' '{ print $1 }' | awk -F'te' '{ print $2 }' Certifica the awk should remove 'te' only if it is present at the start of the string.. anywhere else it should ignore it. expected output is Certificate (7 Replies)
Discussion started by: vivek d r
7 Replies

5. Shell Programming and Scripting

string manipulation

Hi, I have the followoing details in one file: opt/tra/domain/test/new/filename1 training/ear/help I need to manipulate the string in the following manner: filename1= opt/tra/domain/test/new/filename1 help=training/ear/help last string is the name and equal sign and then... (2 Replies)
Discussion started by: ckchelladurai
2 Replies

6. Shell Programming and Scripting

Help With String Manipulation

Hi Guru's, I need some help with data manipulation using shell scripting. I know how to replace the whole string but not part of the string. The value after aa= should be replaced with the value in the mail leaving ,OU=111,OU=222,DC=333 as is. Below are the inputs and expected outputs. Input:... (17 Replies)
Discussion started by: Samingla
17 Replies

7. Shell Programming and Scripting

string manipulation

if I have two string variable, how do I add one to anther. like a= "a" b="b" c=$a+$b but that doesn't work. Is there anyway to solve it.http://www.qtl.co.il/img/copy.pnghttp://www.google.com/favicon.icohttp://www.babylon.com/favicon.icohttp://www.morfix.com/favicon.ico (2 Replies)
Discussion started by: programAngel
2 Replies

8. Shell Programming and Scripting

string manipulation

i have a file that contains a pattern like this: ajay 1234 newyork available kumar 2345 denver singh 2345 newyork ajay 3456 denver kumar 3456 newyork singh 3456 delhi available ajay 4567 miami kumar 4567 miami singh 4567 delhi i want to search for each line... (5 Replies)
Discussion started by: ajay41aj
5 Replies

9. Shell Programming and Scripting

String Manipulation Help

Hey Guys, Right i know how to alter a word to begin with a capital letter, i know how to remove unwanted characters and replace them with the relevant character however i don't now if there is a way to do them all in one line. Code: echo -n ${string:0:1} | tr a-z A-Z #convert first letter... (4 Replies)
Discussion started by: shadow0001
4 Replies

10. Shell Programming and Scripting

string manipulation

Hello, I have a korn shell string variable str1 = "A,B,Z" I would like to create another korn shell string variable str2 = "letter = 'A' or letter = 'B' or letter = 'Z' " Please help! Thanks in advance an UNIX newbie! (13 Replies)
Discussion started by: hai1973
13 Replies
Login or Register to Ask a Question