input ("/" delimited fields):

style1/book1 (author_C)/editor1/2000
style1/book2 (author_A)/editor2/2004
style1/book3 (author_B)/editor3/2001
style2/book8 (author_B)/editor4/2010
style2/book5 (author_A)/editor2/1998

style1/book2 (author_A)/editor2/2004
style1/book3 (author_B)/editor3/2001
style1/book1 (author_C)/editor1/2000
style2/book5 (author_A)/editor2/1998
style2/book8 (author_B)/editor4/2010

BEGIN{FS=OFS="/"}

{
    array[$1] = $0

    split ($2, aut, " ")

    asort(array)

    o = asort(aut)

    for (o in aut)
        print array[aut[o]]

}

The versions of awk that I use (on OS X) don't have the asort() and asorti() functions, but I have read the gawk man page. Unlike the sort utility, there is no way to specify a sort key for these functions; they always sort the array using the entire contents of the string as the sort key. If you want to use asort() in gawk to sort with field 1 as your primary sort key and the second part of field 2 as your secondary key; you need to prepend each line in your array with primary and secondary sort fields, use asort() or asorti() to sort the modified records, and then strip off the added sort fields when you print (or otherwise process) the results.

So you mean I should
1st) sort by field1 and generate a first output
2nd) use this output to sort by subfield 2 and generate the final output.

I tried things like below but still doesn't work.

BEGIN{FS=OFS="/"}

{
# sort by field1
    array[$1] = $0

    asort(array)

# first output
    for (i in array)
        $0 = array[i]

# redefine fields in first output    
        split($0, rec, FS)
        rec[$2] = $0

        split($0, sub, " ")
        aut[++a] = sub[2]

# sort by subfield2 
        n = asort(aut)

# print final output
        for (j=1; j<=n; j++)
            print array[aut[j]]
    
}

What you have above sorts the accumulated input twice each time you read a line from your input file. Clearly that isn't what you want. Even if you were doing the sorts in an END clause instead of in a clause that is executed every time you read an input line, performing a sort on the entire line and then performing a sort on the second sort key is not the same as performing a single sort with a primary and secondary sort key.

I don't have access to a system running gawk, but just using standard interfaces, I get the output:
which I think is what you're trying to get, when I use the script:

#!/bin/ksh
awk 'BEGIN{FS=OFS="/"
    tmpfile = "asorti.out"
    sortcommand = "sort -t/ -o " tmpfile
    cleanup = "rm " tmpfile
}
{   split ($2, sub, " ")
    array[$1 "/" sub[2] "/" $0] = $0
}
END{for (i in array) print i | sortcommand
    close(sortcommand)
    while(getline i < tmpfile) print array[i]
    close(tmpfile)
    system(cleanup)
}' in

#!/bin/ksh
gawk 'BEGIN{FS = OFS = "/" }
{   split ($2, sub, " ")
    array[$1 "/" sub[2] "/" $0] = $0
}   
END{n = asorti(array)
    for(i = 1; i <= n; print array[i++]);
}' in