Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: indexing list of words in a file
Top Forums Shell Programming and Scripting indexing list of words in a file Post 302576282 by m.d.ludwig on Thursday 24th of November 2011 07:31:28 AM
Old 11-24-2011
I read the problem as determine if a word from the list exists in the content of the webpage:
Code:
#! /usr/bin/perl

use strict;
use warnings;
use LWP::Simple;

$\ = "\n";

my $url  = shift(@ARGV);

unless (defined $url && '' lt $url) {
    print STDERR $0, ': missing url';
    exit(1);
}

my $content = get($url);

unless (defined $content) {
    print STDERR $url, ': has no content';
    exit(1);
}

while (<>) {
    chomp;
    print index($content, $_) < 0 ? 'false' : 'true';
}

which would be invoked as:
Code:
perl hindex http://www.online.net/serveur-dedie/offre-dedibox-qc.xhtml wordlist

which results in:
Code:
true
true
false
false
false
true
false

Now this returns 'true' or 'false' depending on the existence of a sequence of characters in the content of the webpage, not splitting out words, removing html tags, and the like. You would need something like HTML::Parser to do that:
Code:
#! /usr/bin/perl

use strict;
use warnings;

use LWP::Simple;
use HTML::Parser;

$\ = "\n";

my %WORDS = ();

sub text {
    my $text = shift(@_);
    return unless defined $text;
    foreach my $w (split ' ', $text) { $WORDS{lc $w}++; }
}

my $url = shift(@ARGV);

unless (defined $url && '' lt $url) {
    print STDERR 'USAGE: ', $0, ' <url> [<wordlist>]';
    exit(1);
}

my $content = get($url);

unless (defined $content) {
    print STDERR $url, ': has no content';
    exit(1);
}

HTML::Parser->new(text_h => [ \&text, 'text' ])->parse($content);

while (<>) {
    chomp;
    print defined $WORDS{$_} ? 'true' : 'false';
}

Which, when invoked, returns:
Code:
true
false
false
false
false
false
false

Please note that this example does not skip over the contents of <script> tags and the like.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

List all file names that contain two specific words.

Hi, all: I would like to search all files under "./" and its subfolders recursively to find out those files contain both word "A" and word "B", and list the filenames finally. How to realize that? Cheers JIA (18 Replies)
Discussion started by: jiapei100
18 Replies

2. Shell Programming and Scripting

Splitting Concatenated Words in Input File with Words from a Master File

Hello, I have a complex problem. I have a file in which words have been joined together: Theboy ranslowly I want to be able to correctly split the words using a lookup file in which all the words occur: the boy ran slowly slow put child ly The lookup file which is meant for look up... (21 Replies)
Discussion started by: gimley
21 Replies

3. Shell Programming and Scripting

[ask]filtering file to indexing...

dear all, i have file with format like this file_master.txt 20110212|231213|rio|apri|23112|222222 20110212|312311|jaka|dino|31223|543234 20110301|343322|alfan|budi|32131|333311 ... i want filter with output like this index_nm.txt rio|apri jaka|dino ... index_years.txt 20110212... (7 Replies)
Discussion started by: zvtral
7 Replies

4. Shell Programming and Scripting

indexing a file

hello guys, I have a file like this: input.dat Push-to-talk No Coonection IP support Support for IP telephony Yes Built-in SIP stack Yes Support via software Yes Microsoft Support for Microsoft Exchange Yes UMA (5 Replies)
Discussion started by: Johanni
5 Replies

5. Shell Programming and Scripting

Split a file using 2-D indexing system

I have a file and want to split it using a 2-D index system for example if the file is p.dat with 6 data sets separated by ">". I want to set nx=3, ny=2. I need to create files p.dat.1.1 p.dat.1.2 p.dat.1.3 p.dat.2.1 p.dat.2.2 p.dat.2.3 I have tried using a single index and want... (3 Replies)
Discussion started by: kristinu
3 Replies

6. Shell Programming and Scripting

Grepping a list of words from one file in a master database of homophones

Hello, I am sorry if the title is confusing, but I need a script to grep a list of Names from a Source file in a Master database in which all the homophonic variants of the name are listed along with a single indexing key and store all of these in an output file. I need this because I am testing... (4 Replies)
Discussion started by: gimley
4 Replies

7. UNIX for Advanced & Expert Users

List all file names that contain two specific words. ( follow up )

Being new to the forum, I tried finding a solution to find files containing 2 words not necessarily on the same line. This thread "List all file names that contain two specific words." answered it in part, but I was looking for a more concise solution. Here's a one-line suggestion... (8 Replies)
Discussion started by: Symbo53
8 Replies

8. Shell Programming and Scripting

Replace particular words in file based on if finds another words in that line

Hi All, I need one help to replace particular words in file based on if finds another words in that file . i.e. my self is peter@king. i am staying at north sydney. we all are peter@king. How to replace peter to sham if it finds @king in any line of that file. Please help me... (8 Replies)
Discussion started by: Rajib Podder
8 Replies

9. Shell Programming and Scripting

Awk- Indexing a list of numbers in file2 to print certain rows in file1

Hi Does anyone know of an efficient way to index a column of data in file2 to print the coresponding row in file1 which corresponds to the data in file2 AND 30 rows preceding and after the row in file1. For example suppose you have a list of numbers in file2 (single column) as follows:... (6 Replies)
Discussion started by: Geneanalyst
6 Replies

10. Shell Programming and Scripting

Deleting a list of words from a text file

Hello, I have a list of words separated by spaces I am trying to delete from a text file, and I could not figure out what is the best way to do this. what I tried (does not work) : delete="password key number verify" arr=($delete) for i in arr { sed "s/\<${arr}\>]*//g" in.txt } >... (5 Replies)
Discussion started by: Hawk4520
5 Replies

LEARN ABOUT SUSE

lwp-request

LWP-REQUEST(1)						User Contributed Perl Documentation					    LWP-REQUEST(1)

NAME

       lwp-request, GET, POST, HEAD - Simple command line user agent

SYNOPSIS

       lwp-request [-afPuUsSedvhx] [-m method] [-b base URL] [-t timeout]
		   [-i if-modified-since] [-c content-type]
		   [-C credentials] [-p proxy-url] [-o format] url...

DESCRIPTION

       This program can be used to send requests to WWW servers and your local file system. The request content for POST and PUT methods is read
       from stdin.  The content of the response is printed on stdout.  Error messages are printed on stderr.  The program returns a status value
       indicating the number of URLs that failed.

       The options are:

       -m <method>
	   Set which method to use for the request.  If this option is not used, then the method is derived from the name of the program.

       -f  Force request through, even if the program believes that the method is illegal.  The server might reject the request eventually.

       -b <uri>
	   This URI will be used as the base URI for resolving all relative URIs given as argument.

       -t <timeout>
	   Set the timeout value for the requests.  The timeout is the amount of time that the program will wait for a response from the remote
	   server before it fails.  The default unit for the timeout value is seconds.	You might append "m" or "h" to the timeout value to make
	   it minutes or hours, respectively.  The default timeout is '3m', i.e. 3 minutes.

       -i <time>
	   Set the If-Modified-Since header in the request. If time is the name of a file, use the modification timestamp for this file. If time
	   is not a file, it is parsed as a literal date. Take a look at HTTP::Date for recognized formats.

       -c <content-type>
	   Set the Content-Type for the request.  This option is only allowed for requests that take a content, i.e. POST and PUT.  You can force
	   methods to take content by using the "-f" option together with "-c".  The default Content-Type for POST is
	   "application/x-www-form-urlencoded".  The default Content-type for the others is "text/plain".

       -p <proxy-url>
	   Set the proxy to be used for the requests.  The program also loads proxy settings from the environment.  You can disable this with the
	   "-P" option.

       -P  Don't load proxy settings from environment.

       -H <header>
	   Send this HTTP header with each request. You can specify several, e.g.:

	       lwp-request 
		   -H 'Referer: http://other.url/' 
		   -H 'Host: somehost' 
		   http://this.url/

       -C <username>:<password>
	   Provide credentials for documents that are protected by Basic Authentication.  If the document is protected and you did not specify the
	   username and password with this option, then you will be prompted to provide these values.

       The following options controls what is displayed by the program:

       -u  Print request method and absolute URL as requests are made.

       -U  Print request headers in addition to request method and absolute URL.

       -s  Print response status code.	This option is always on for HEAD requests.

       -S  Print response status chain. This shows redirect and authorization requests that are handled by the library.

       -e  Print response headers.  This option is always on for HEAD requests.

       -d  Do not print the content of the response.

       -o <format>
	   Process HTML content in various ways before printing it.  If the content type of the response is not HTML, then this option has no
	   effect.  The legal format values are; text, ps, links, html and dump.

	   If you specify the text format then the HTML will be formatted as plain latin1 text.  If you specify the ps format then it will be
	   formatted as Postscript.

	   The links format will output all links found in the HTML document.  Relative links will be expanded to absolute ones.

	   The html format will reformat the HTML code and the dump format will just dump the HTML syntax tree.

	   Note that the "HTML-Tree" distribution needs to be installed for this option to work.  In addition the "HTML-Format" distribution needs
	   to be installed for -o text or -o ps to work.

       -v  Print the version number of the program and quit.

       -h  Print usage message and quit.

       -a  Set text(ascii) mode for content input and output.  If this option is not used, content input and output is done in binary mode.

       Because this program is implemented using the LWP library, it will only support the protocols that LWP supports.

SEE ALSO

       lwp-mirror, LWP

COPYRIGHT

       Copyright 1995-1999 Gisle Aas.

       This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

AUTHOR

       Gisle Aas <gisle@aas.no>

perl v5.12.1							    2009-11-21							    LWP-REQUEST(1)
All times are GMT -4. The time now is 11:50 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy