Sponsored Content
Full Discussion: Search Engine
Top Forums Web Development Search Engine Post 302486102 by pludi on Friday 7th of January 2011 03:22:33 AM
Old 01-07-2011
I'd say you need (at least) 3 components:
  1. A crawler that downloads pages, and follows links on those pages.
  2. An indexer that builds a list of words used on each page (maybe in relation to other words nearby), and saves that to a database.
  3. A front-end to query the database.
For the crawler you can use just about any language since the main limitation is the network speed. For the indexer I'd recommend either C/C++ (for speed) or a language geared towards natural language processing (like Perl). For the front-end you can again choose whatever language you're comfortable with.
 

5 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Search Engine

How do you write a search engline to search offline scripts? (3 Replies)
Discussion started by: hawaiifiver
3 Replies

2. Programming

Search Engine in C

Hello everybody, I need help with this, I need to design a CGI search engine in C but i have no idea on what or how to do it. Do i have to open all the html files one by one and search for the given strings? i think this process will be slow, and will take too much of the server processing... (11 Replies)
Discussion started by: semash!
11 Replies

3. Shell Programming and Scripting

About search engine in unix

Hello, How to create a search engine in unix using commands?...Atleast guidelines to craete this search engine...Thank you in advance. (10 Replies)
Discussion started by: Sindhu R
10 Replies

4. Homework & Coursework Questions

About search engine in unix

Use and complete the template provided. The entire template must be completed. If you don't, your post may be deleted! 1. The problem statement, all variables and given/known data: How to create a search engine in unix using commands?...Atleast guidelines to craete this search engine...Thank... (1 Reply)
Discussion started by: Sindhu R
1 Replies

5. What is on Your Mind?

YouTube: Search Engine Optimization | How To Fix Soft 404 Errors and A.I. Tales from Google Search

Getting a bit more comfortable making quick YT videos in 4K, here is: Search Engine Optimization | How To Fix Soft 404 Errors and A.I. Tales from Google Search Console https://youtu.be/I6b9T2qcqFo (0 Replies)
Discussion started by: Neo
0 Replies
HTTP::Proxy::Engine::Legacy(3pm)			User Contributed Perl Documentation			  HTTP::Proxy::Engine::Legacy(3pm)

NAME
HTTP::Proxy::Engine::Legacy - The "older" HTTP::Proxy engine SYNOPSIS
my $proxy = HTTP::Proxy->new( engine => 'Legacy' ); DESCRIPTION
This engine reproduces the older child creation algorithm of HTTP::Proxy. Angelos Karageorgiou "<angelos@unix.gr>" reports: I got the Legacy engine to work really fast under "Win32" with the following trick: max_keep_alive_requests(1); max_clients(120); $HTTP::VERSION(1.0); # just in case and it smokes. It seems that forked children are really slow when calling select for handling "keep-alive"d requests! METHODS
The module defines the following methods, used by HTTP::Proxy main loop: start() Initialise the engine. run() Implements the forking logic: a new process is forked for each new incoming TCP connection. stop() Reap remaining child processes. The following method is used by the engine internally: reap_zombies() Process the dead child processes. SEE ALSO
HTTP::Proxy, HTTP::Proxy::Engine. AUTHOR
Philippe "BooK" Bruhat, "<book@cpan.org>". COPYRIGHT
Copyright 2005, Philippe Bruhat. LICENSE
This module is free software; you can redistribute it or modify it under the same terms as Perl itself. perl v5.12.4 2011-07-03 HTTP::Proxy::Engine::Legacy(3pm)
All times are GMT -4. The time now is 02:45 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy