Please let me know if Google search appliance supports JSON or XML interface? If yes please provide some references

You should ask Google about their products.

http://groups.google.com/group/Google-Search-Appliance

XML can be returned though. Just remove the proxystylesheet name/value param from your query string, and you'll see it.

Is it anyway possible that from a result set of queried pages generated by search engine a query on that result pages can be executed?

We need to pass a new parameter: swrnum to Google Search Appliance Engine. When we pass the new parameter the search is set to query again to the number of results found in our initial search. The new search term is passed in as as_q (as = advanced search).The original term, q, is passed in as a hidden field. The results will show both terms combined, plus the new results

Now my next question is how to capture the Search Engine result set because I do not want the user to redirect to GSA page, instead I want to control the final display through my code? What tool functions supports is?

Please advice.