Slow FFT in ksh93 and awk.


 
Thread Tools Search this Thread
Operating Systems OS X (Apple) Slow FFT in ksh93 and awk.
# 8  
Old 09-12-2018
Another thing you can do to speed it up is keep it running - feed it input and read it output at need, rather than starting and killing it all the time. How to do that depends on what your input data would look like and what you'd want for your output data.
This User Gave Thanks to Corona688 For This Post:
# 9  
Old 09-13-2018
Pur awk FFT.

Quote:
Originally Posted by Corona688
Another thing you can do to speed it up is keep it running - feed it input and read it output at need, rather than starting and killing it all the time. How to do that depends on what your input data would look like and what you'd want for your output data.
There is one error and corrected for the correct results, note the minus sign in red:
SUMIMAG += -REAL_ARRAY[T]*SIN_ANGLE + IMAG_ARRAY[T]*COS_ANGLE;
Correct results:
Code:
Real:
 8.50 3.64 -1.71 0.05 1.50 -0.17 -0.29 0.48 0.50 0.48 -0.29 -0.17 1.50 0.05 -1.71 3.64
Imaginary:
 0.00 -1.93 -0.50 -0.24 -1.00 0.47 0.50 -1.22 0.00 1.22 -0.50 -0.47 1.00 0.24 0.50 1.93

Otherwise phenominal.
Thanks a lot...
(This has taught me more about awk than ANY tutorial or info from the mighty WWW and for that I can only say thank you.)

Last edited by wisecracker; 09-14-2018 at 07:44 AM.. Reason: Add title.
This User Gave Thanks to wisecracker For This Post:
# 10  
Old 09-13-2018
Just as a side note - awka is an awk -> C translator

I think it was created out of similar frustrations we have all had with awk performance in major applications. I've tried it with varying success.

May be worth considering:
GitHub - noyesno/awka: Revive awka - AWK to C Conversion Tool
These 2 Users Gave Thanks to jim mcnamara For This Post:
# 11  
Old 09-13-2018
Pure awk FFT.

OK a modified version of Corona688's awk translation:
Code:
# FFT_Awk.awk
# A modified translation by Corona688, WAY COOL!
# Don't feed it REAL/IMAG/K/T, those are local variables!
function fft(REAL_ARRAY, IMAG_ARRAY, K, T, SUMREAL, SUMIMAG,
        ANGLE)
{
        # PI to 14 decimal places.
        PI=3.14159265358979;
        for(K=1; K in REAL_ARRAY; K++)
        {
                SUMREAL=SUMIMAG=0;
                for(T=1; T in REAL_ARRAY; T++)
                {
                        ANGLE=(2*PI*(T-1)*(K-1))/N;
                        SUMREAL += REAL_ARRAY[T]*cos(ANGLE) + IMAG_ARRAY[T]*sin(ANGLE);
                        SUMIMAG += -REAL_ARRAY[T]*sin(ANGLE) + IMAG_ARRAY[T]*cos(ANGLE);
                }
                REAL[K]=SUMREAL
                IMAG[K]=SUMIMAG
        }
}

BEGIN {
        # REALSTR="1.0 1.0 1.0 1.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.5 0.5 0.5"
        # IMAGSTR="0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0"
        # 'IMAGSTR' can be builtin but not done in this DEMO.

        # Split into arrays.  INDEXES 1-16, not 0-15!
        N=split(REALSTR, REAL_ARRAY, " ");
        split(IMAGSTR, IMAG_ARRAY, " ");
        # Convert strings to floats.
        for(X in REAL_ARRAY) REAL_ARRAY[X] += 0.0
        for(X in IMAG_ARRAY) IMAG_ARRAY[X] += 0.0
        # N=16;    # Hardcoded 16 elements WITH '0.5' mid point padding for this DEMO!

        fft(REAL_ARRAY, IMAG_ARRAY);

        printf("Real:\n");
        for(X=1; X in REAL; X++)
                printf(" %.5f", REAL[X]);
        printf("\n\n");

        printf("Imaginary:\n");
        for(X=1; X in IMAG; X++)
                printf(" %.5f", IMAG[X]);
        printf("\n\n");

        printf("Absolute Values:\n");
        for(X=1; X<=N; X++)
		printf(" %.5f", sqrt(((REAL[X])^2)+((IMAG[X])^2)));
        printf("\n");

# awk -f FFT_Awk.awk -v N=16 -v REALSTR="1.0 1.0 1.0 1.0 1.0 1.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.5 0.5 0.5" IMAGSTR="0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0" /dev/null
#
# Real:
#  8.50000 3.63782 -1.70711 0.05143 1.50000 -0.17275 -0.29289 0.48350 0.50000 0.48350 -0.29289 -0.17275 1.50000 0.05143 -1.70711 3.63782
#
# Imaginary:
#  0.00000 -1.93071 -0.50000 -0.24147 -1.00000 0.46564 0.50000 -1.22361 0.00000 1.22361 -0.50000 -0.46564 1.00000 0.24147 0.50000 1.93071
#
# Absolute Values:
#  8.50000 4.11842 1.77882 0.24688 1.80278 0.49665 0.57947 1.31567 0.50000 1.31567 0.57947 0.49665 1.80278 0.24688 1.77882 4.11842
}

Is this a first?
Could it even pull views to UNIX from search engines?
Who knows but one thing I do know my eventual aim was to do this.
I chose the ksh-awk and ARexx versions earlier in this thread to see the possibilities of FFT on languages not really designed to do it, then to dig deep into awk programming to succeed at this also...
The awk only version, courtesy of Corona688, makes FFT available for all the shells floating point or not.
Again C688, superb, now to ether remove import dependencies for python, (already done a python version), or modify this for the builtin 'SPECAN' in AudioScope.sh...
Lastly, I hope this has been an interesting break from all the usual stuff posted on here.

Last edited by wisecracker; 09-14-2018 at 07:43 AM.. Reason: Add title...
These 2 Users Gave Thanks to wisecracker For This Post:
# 12  
Old 09-15-2018
WOW!
The awk version works on an AMIGA A1200 with HDD and 4MB extra memory using ADE the UNIX _emulator_ for AMIGA OS-3.x running ksh88 on top of a default AMIGA A1200 shell.
Now that is pretty damn good!
Thanks to Corona688 for his original translation.

@Corona688 with your permission can I upload to AMINET for we AMIGA fanatics too?

Aminet - Latest packages, last 14 days
# 13  
Old 09-17-2018
Absolutely. I don't post code for people to not use it.

It's a kind of naive implementation though. Usually better to read values from stdin, but knowing how audioscope works I knew you'd need batches. That'd obscure how it really works a bit though. Maybe better as this, people can populate the arrays as they please.

It's not truly an FFT either, just a DFT -- which gives the same result but takes much more time to do so. Notice how it must do N*N loops for N input. FFT accomplishes the same with N log N loops.

Still, I hope it has its uses. And I'm grateful for you writing one in shell. That's the first DFT I've seen written in a language I really understand.

Last edited by Corona688; 09-17-2018 at 07:26 PM..
These 2 Users Gave Thanks to Corona688 For This Post:
# 14  
Old 09-19-2018
Quote:
Originally Posted by Corona688
Absolutely. I don't post code for people to not use it.

It's a kind of naive implementation though. Usually better to read values from stdin, but knowing how audioscope works I knew you'd need batches. That'd obscure how it really works a bit though. Maybe better as this, people can populate the arrays as they please.

It's not truly an FFT either, just a DFT -- which gives the same result but takes much more time to do so. Notice how it must do N*N loops for N input. FFT accomplishes the same with N log N loops.

Still, I hope it has its uses. And I'm grateful for you writing one in shell. That's the first DFT I've seen written in a language I really understand.
Yes it is technically a DFT but most people don't know what the difference is and FFT is what they know most, so I used that term. I will upload to AMINET as a DFT however.
And naive it is too, but I could stay with python and use the builtin import cmath version I have got to eliminate the 'scipy' and 'scipyIO' dependencies. I really didn't want to use python although my code works from Python 1.4.0 to the current 3.7.0.
Thanks a lot matey, will post the URL for you to see when I have uploaded it...
Bazza...

------ Post updated 19-09-18 at 06:19 AM ------

Hi Corona688...

Uploaded to the AMINET site at around 9:30pm UK local time last night, 18-09-2018.

At around 6:00am today 19-09-2018 UK local time it was added and has had 2 downloads, dls, already. It will be on show for 14 days and I expect at least 150 dls in that time.

Aminet - dev/gcc/DFT-FFT.awk.txt

Thanks a lot...

Bazza.

(Consider this thread wrapped up.)

I have no idea why it added my separate post to the end of this one...
This User Gave Thanks to wisecracker For This Post:
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. OS X (Apple)

FFT for Python 2.0.x to 3.7.0.

Hi guys... This is code that was originally designed to work on an upgraded AMIGA A1200 using Python 2.0.x. Unfortunately it broke inside much later versions, NOT because of the print statement/function but other minor subtleties. So this is the final result tested on various machines including... (0 Replies)
Discussion started by: wisecracker
0 Replies

2. AIX

Ksh93/AIX compatibility

Hi everyone ! Im trying to know from wich version of AIX KSH93 is available ? Internet tell me 6.x and 7.x AIX are available, bue what about 5.x ? Is KSH93 available on AIX 5.x ? Is it the same way to manipulate variables as KSH93 on 7.x ? Thanks for your support and have a nice day ! (2 Replies)
Discussion started by: majinfrede
2 Replies

3. OS X (Apple)

FFT for the AMIGA through ksh88 shell.

I don't know if anyone is interested but I have been meddling with FFT for the AMIGA. (Sadly we AMIGAns don't have these luxuries through any scripting language. Below is a Python snippet that uses the builtin 'cmath' module to work with the lowly Python 2.0.1 for the AMIGA. It is part of a... (0 Replies)
Discussion started by: wisecracker
0 Replies

4. UNIX for Advanced & Expert Users

Ksh93 on Linux compatible with ksh93 on AIX

Hi Experts, I have several shell scripts that have been developed on a Linux box for korn ksh93. If we want to run this software on an AIX 6.1 box that runs ksh88 by default can we just change the she-bang line to reference /bin/ksh93 which ships with AIX as its "enhanced shell" to ensure... (6 Replies)
Discussion started by: Keith Turley
6 Replies

5. Shell Programming and Scripting

Making a faster alternative to a slow awk command

Hi, I have a large number of input files with two columns of numbers. For example: 83 1453 99 3255 99 8482 99 7372 83 175 I only wish to retain lines where the numbers fullfil two requirements. E.g: =83 1000<=<=2000 To do this I use the following... (10 Replies)
Discussion started by: s052866
10 Replies

6. Shell Programming and Scripting

cut, sed, awk too slow to retrieve line - other options?

Hi, I have a script that, basically, has two input files of this type: file1 key1=value1_1_1 key2=value1_2_1 key4=value1_4_1 ... file2 key2=value2_2_1 key2=value2_2_2 key3=value2_3_1 key4=value2_4_1 ... My files are 10k lines big each (approx). The keys are strings that don't... (7 Replies)
Discussion started by: fzd
7 Replies

7. Shell Programming and Scripting

The builtin split function in AWK is too slow

I have a text file that contains 4 million lines, each line contains 2 fields(colon as field separator). as shown: 123:444,555,666,777,888,345 233:5444,555,666,777,888,345 623:454,585,664,773,888,345 ...... Here I have to split the second field(can be up to 40,000 fields) by comma into an... (14 Replies)
Discussion started by: kevintse
14 Replies

8. Shell Programming and Scripting

ksh88 or ksh93

Hi all! Does anybody know how can I check if any UNIX installation has implemented ksh88 or ksh93? Thanks in advance. Néstor. (3 Replies)
Discussion started by: Nestor
3 Replies

9. Shell Programming and Scripting

ksh93 deprecation...

Any means of running ksh93 in a ksh88-mode? Might sound odd, but I want/need to restrict U/Win-developed scripts to correspond to the ksh88 version on my Solaris environment(s). Thanks. (2 Replies)
Discussion started by: curleb
2 Replies

10. UNIX for Dummies Questions & Answers

echo is too slow. HELP with Awk

Hello All, Below is a simple script i worte to find the 208th char in a file. If the char = "C" then I re-direct the line to a file called change.txt. If it is not "C" then I re-direct it to a file called delete.txt. My problem is I have a file 0f 500K lines. this script is very slow. I am... (4 Replies)
Discussion started by: eja
4 Replies
Login or Register to Ask a Question