Slow FFT in ksh93 and awk.


Login or Register to Reply

 
Thread Tools Search this Thread
# 8  
Old 09-12-2018
Another thing you can do to speed it up is keep it running - feed it input and read it output at need, rather than starting and killing it all the time. How to do that depends on what your input data would look like and what you'd want for your output data.
This User Gave Thanks to Corona688 For This Post:
wisecracker (09-13-2018)
# 9  
Old 09-13-2018
Pur awk FFT.

Quote:
Originally Posted by Corona688
Another thing you can do to speed it up is keep it running - feed it input and read it output at need, rather than starting and killing it all the time. How to do that depends on what your input data would look like and what you'd want for your output data.
There is one error and corrected for the correct results, note the minus sign in red:
SUMIMAG += -REAL_ARRAY[T]*SIN_ANGLE + IMAG_ARRAY[T]*COS_ANGLE;
Correct results:
Code:
Real:
 8.50 3.64 -1.71 0.05 1.50 -0.17 -0.29 0.48 0.50 0.48 -0.29 -0.17 1.50 0.05 -1.71 3.64
Imaginary:
 0.00 -1.93 -0.50 -0.24 -1.00 0.47 0.50 -1.22 0.00 1.22 -0.50 -0.47 1.00 0.24 0.50 1.93

Otherwise phenominal.
Thanks a lot...
(This has taught me more about awk than ANY tutorial or info from the mighty WWW and for that I can only say thank you.)

Last edited by wisecracker; 09-14-2018 at 06:44 AM.. Reason: Add title.
This User Gave Thanks to wisecracker For This Post:
Corona688 (09-13-2018)
# 10  
Old 09-13-2018
Just as a side note - awka is an awk -> C translator

I think it was created out of similar frustrations we have all had with awk performance in major applications. I've tried it with varying success.

May be worth considering:
GitHub - noyesno/awka: Revive awka - AWK to C Conversion Tool
These 2 Users Gave Thanks to jim mcnamara For This Post:
Corona688 (09-13-2018) wisecracker (09-13-2018)
# 11  
Old 09-13-2018
Pure awk FFT.

OK a modified version of Corona688's awk translation:
Code:
# FFT_Awk.awk
# A modified translation by Corona688, WAY COOL!
# Don't feed it REAL/IMAG/K/T, those are local variables!
function fft(REAL_ARRAY, IMAG_ARRAY, K, T, SUMREAL, SUMIMAG,
        ANGLE)
{
        # PI to 14 decimal places.
        PI=3.14159265358979;
        for(K=1; K in REAL_ARRAY; K++)
        {
                SUMREAL=SUMIMAG=0;
                for(T=1; T in REAL_ARRAY; T++)
                {
                        ANGLE=(2*PI*(T-1)*(K-1))/N;
                        SUMREAL += REAL_ARRAY[T]*cos(ANGLE) + IMAG_ARRAY[T]*sin(ANGLE);
                        SUMIMAG += -REAL_ARRAY[T]*sin(ANGLE) + IMAG_ARRAY[T]*cos(ANGLE);
                }
                REAL[K]=SUMREAL
                IMAG[K]=SUMIMAG
        }
}

BEGIN {
        # REALSTR="1.0 1.0 1.0 1.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.5 0.5 0.5"
        # IMAGSTR="0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0"
        # 'IMAGSTR' can be builtin but not done in this DEMO.

        # Split into arrays.  INDEXES 1-16, not 0-15!
        N=split(REALSTR, REAL_ARRAY, " ");
        split(IMAGSTR, IMAG_ARRAY, " ");
        # Convert strings to floats.
        for(X in REAL_ARRAY) REAL_ARRAY[X] += 0.0
        for(X in IMAG_ARRAY) IMAG_ARRAY[X] += 0.0
        # N=16;    # Hardcoded 16 elements WITH '0.5' mid point padding for this DEMO!

        fft(REAL_ARRAY, IMAG_ARRAY);

        printf("Real:\n");
        for(X=1; X in REAL; X++)
                printf(" %.5f", REAL[X]);
        printf("\n\n");

        printf("Imaginary:\n");
        for(X=1; X in IMAG; X++)
                printf(" %.5f", IMAG[X]);
        printf("\n\n");

        printf("Absolute Values:\n");
        for(X=1; X<=N; X++)
		printf(" %.5f", sqrt(((REAL[X])^2)+((IMAG[X])^2)));
        printf("\n");

# awk -f FFT_Awk.awk -v N=16 -v REALSTR="1.0 1.0 1.0 1.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.5 0.5 0.5" IMAGSTR="0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0" /dev/null
#
# Real:
#  8.50000 3.63782 -1.70711 0.05143 1.50000 -0.17275 -0.29289 0.48350 0.50000 0.48350 -0.29289 -0.17275 1.50000 0.05143 -1.70711 3.63782
#
# Imaginary:
#  0.00000 -1.93071 -0.50000 -0.24147 -1.00000 0.46564 0.50000 -1.22361 0.00000 1.22361 -0.50000 -0.46564 1.00000 0.24147 0.50000 1.93071
#
# Absolute Values:
#  8.50000 4.11842 1.77882 0.24688 1.80278 0.49665 0.57947 1.31567 0.50000 1.31567 0.57947 0.49665 1.80278 0.24688 1.77882 4.11842
}

Is this a first?
Could it even pull views to UNIX from search engines?
Who knows but one thing I do know my eventual aim was to do this.
I chose the ksh-awk and ARexx versions earlier in this thread to see the possibilities of FFT on languages not really designed to do it, then to dig deep into awk programming to succeed at this also...
The awk only version, courtesy of Corona688, makes FFT available for all the shells floating point or not.
Again C688, superb, now to ether remove import dependencies for python, (already done a python version), or modify this for the builtin 'SPECAN' in AudioScope.sh...
Lastly, I hope this has been an interesting break from all the usual stuff posted on here.

Last edited by wisecracker; 09-14-2018 at 06:43 AM.. Reason: Add title...
These 2 Users Gave Thanks to wisecracker For This Post:
Corona688 (09-19-2018) Neo (09-16-2018)
# 12  
Old 09-15-2018
WOW!
The awk version works on an AMIGA A1200 with HDD and 4MB extra memory using ADE the UNIX _emulator_ for AMIGA OS-3.x running ksh88 on top of a default AMIGA A1200 shell.
Now that is pretty damn good!
Thanks to Corona688 for his original translation.

@Corona688 with your permission can I upload to AMINET for we AMIGA fanatics too?

Aminet - Latest packages, last 14 days
# 13  
Old 09-17-2018
Absolutely. I don't post code for people to not use it.

It's a kind of naive implementation though. Usually better to read values from stdin, but knowing how audioscope works I knew you'd need batches. That'd obscure how it really works a bit though. Maybe better as this, people can populate the arrays as they please.

It's not truly an FFT either, just a DFT -- which gives the same result but takes much more time to do so. Notice how it must do N*N loops for N input. FFT accomplishes the same with N log N loops.

Still, I hope it has its uses. And I'm grateful for you writing one in shell. That's the first DFT I've seen written in a language I really understand.

Last edited by Corona688; 09-17-2018 at 06:26 PM..
These 2 Users Gave Thanks to Corona688 For This Post:
Neo (09-17-2018) wisecracker (09-18-2018)
# 14  
Old 09-19-2018
Quote:
Originally Posted by Corona688
Absolutely. I don't post code for people to not use it.

It's a kind of naive implementation though. Usually better to read values from stdin, but knowing how audioscope works I knew you'd need batches. That'd obscure how it really works a bit though. Maybe better as this, people can populate the arrays as they please.

It's not truly an FFT either, just a DFT -- which gives the same result but takes much more time to do so. Notice how it must do N*N loops for N input. FFT accomplishes the same with N log N loops.

Still, I hope it has its uses. And I'm grateful for you writing one in shell. That's the first DFT I've seen written in a language I really understand.
Yes it is technically a DFT but most people don't know what the difference is and FFT is what they know most, so I used that term. I will upload to AMINET as a DFT however.
And naive it is too, but I could stay with python and use the builtin import cmath version I have got to eliminate the 'scipy' and 'scipyIO' dependencies. I really didn't want to use python although my code works from Python 1.4.0 to the current 3.7.0.
Thanks a lot matey, will post the URL for you to see when I have uploaded it...
Bazza...

------ Post updated 19-09-18 at 06:19 AM ------

Hi Corona688...

Uploaded to the AMINET site at around 9:30pm UK local time last night, 18-09-2018.

At around 6:00am today 19-09-2018 UK local time it was added and has had 2 downloads, dls, already. It will be on show for 14 days and I expect at least 150 dls in that time.

Aminet - dev/gcc/DFT-FFT.awk.txt

Thanks a lot...

Bazza.

(Consider this thread wrapped up.)

I have no idea why it added my separate post to the end of this one...
This User Gave Thanks to wisecracker For This Post:
Corona688 (09-19-2018)
Login or Register to Reply

|
Thread Tools Search this Thread
Search this Thread:
Advanced Search

More UNIX and Linux Forum Topics You Might Find Helpful
FFT for Python 2.0.x to 3.7.0. wisecracker OS X (Apple) 0 08-20-2018 09:54 AM
Ksh93/AIX compatibility majinfrede AIX 2 04-26-2018 09:07 AM
FFT for the AMIGA through ksh88 shell. wisecracker OS X (Apple) 0 11-10-2017 04:16 AM
Possible ksh93 Bug Expanding Variables? bakunin Shell Programming and Scripting 4 07-04-2017 01:54 AM
Ksh93 on Linux compatible with ksh93 on AIX Keith Turley UNIX for Advanced & Expert Users 6 10-22-2016 01:37 AM
Making a faster alternative to a slow awk command s052866 Shell Programming and Scripting 10 07-12-2012 05:54 AM
ksh93 different results using -x option lthorson Shell Programming and Scripting 8 10-30-2011 10:54 AM
cut, sed, awk too slow to retrieve line - other options? fzd Shell Programming and Scripting 7 12-30-2010 07:34 PM
ksh93 newline character problem neeto UNIX for Dummies Questions & Answers 4 06-08-2010 05:14 AM
The builtin split function in AWK is too slow kevintse Shell Programming and Scripting 14 05-23-2010 10:15 AM
Install ksh93 for cygwin devtakh UNIX for Advanced & Expert Users 1 07-10-2009 07:32 AM
Performance degradation with KSH93 i.f.schulz Shell Programming and Scripting 2 04-02-2009 10:27 AM
ksh88 or ksh93 Nestor Shell Programming and Scripting 3 03-13-2009 10:33 AM
ksh93 deprecation... curleb Shell Programming and Scripting 2 03-17-2008 08:36 PM
echo is too slow. HELP with Awk eja UNIX for Dummies Questions & Answers 4 05-14-2007 09:34 AM