Home Man
Search
Today's Posts
Register

BSD, Linux, and UNIX shell scripting — Post awk, bash, csh, ksh, perl, php, python, sed, sh, shell scripts, and other shell scripting languages questions here.

PERL or SHELL Scrript to search in Directories by taking line by line from a text file

Tags
shell scripts

Login to Reply

 
Thread Tools Search this Thread
# 1  
Old 12-16-2011
Question PERL or SHELL Scrript to search in Directories by taking line by line from a text file

Unix box server version
***********
Code:
>uname  -r
B.11.00

>echo $SHELL
/usr/bin/ksh

--> in this server, I have the path like /IMbuild/dev/im0serv1
---> in that directory I have the folders startup(.jsp files nearly 100 jsp's ) and scripts(contains .js files nearly 100 files)
So, in .jsp,.js files there are two methods calling the Application ID like
returnMMCObjectContent("chiru");//
getDCCObjectContent("chiru");

like chiru I need to search for 22,000 articles.
till now I am using one command
Code:
find . -name  -exec  grep -l "chiru" {}\; | -exec  grep -n  "chiru" {}\;//

then I vl get the result like that method where ever the chiru was used and the file name with path like ../start/loutout.js,../filenames/login.jsp

so with this I can make a note that chiru was used name bcoz reference files were displayed.
If any reference file not came,that means chiru was unUsed name....

This whole part is MANUAL kind of thing I have to do CTRL+C and CTRL+V ..........for 22,000 times

MY REQUIREMENT IS:

--------------------
I want to take 10,000 names in a text file (test.txt) and directly pass as input file to the script and the script has to read the file line by line and searching in the sub directories(start and scripts) ,if reference has found,then those names has to go into one output file with that name and reference file(.jsp, .js) (out1.txt) , if any reference was not found then those names should go into another output file(out2.txt).
So I am asking for that script ;

either perl script or any UNIX shell script that can work in version:B.11.00



Another Unix box server
***********
Code:
>uname -r
2.6.9-89.0.20.ELsmp

>echo  $SHELL
ksh:echo: not found

This is different version of UNIX
The same script for the above server I want to execute in this server....


Thanks in Advance ........................

Last edited by vbe; 12-16-2011 at 05:51 AM.. Reason: used code tags next time for your data and code
# 2  
Old 12-16-2011
I have to admit some puzzlement from:
Quote:
Originally Posted by pasam
Code:
find . -name  -exec  grep -l "chiru" {}\; | -exec  grep -n  "chiru" {}\;//

This will run grep on every file and directory found - one at a time. Also, the output of these greps is then piped to -exec, which I am pretty sure does not exist. Restricting this to 'jsp' and 'js' files would help:
Code:
find . -type f \( -name '*.jsp' -o -name '*.js' \) -exec ....

And instead of using find to invoked grep on each file, use xargs:
Code:
find . -type f \( -name '*.jsp' -o -name '*.js' \) -print0 | xargs -0 grep -n chiru

The two greps are not needed, because grep -l regex filename will read the entire file it it does not find regex, and grep -n regex filename reads the entire file to find matches. By not using two greps, you are saving a extra (partial) read of files that contain regex.

Now the problem of finding matches of 10,000 patterns. Re-reading each file 10,000 times would be time consuming at best. But given that the pattern can occur anywhere in the file, that is what you are stuck with:
Code:
FILE=$(mktemp)
trap "rm -f ${FILE}" EXIT

# built list of files
find . -type f \( -name '*.jsp' -o -name '*.js' \) -print0 > ${FILE}

cat "${@}" | while read pattern; do
    xargs -0 grep -n "${pattern}" < ${FILE}
done

Now if the patterns occur only in lines like:
Quote:
Originally Posted by pasam
returnMMCObjectContent("pattern");//
getDCCObjectContent("pattern");
then you use some perl to make this run faster.
Code:
#! /path/to/perl
# USAGE: patternfind patternlist [filelist]...

my $patternlist = shift @ARGV;
die unless defined $patternlist;

my %OBJECTS;
# key = object
# value = list of file/line pairs

while (<>) {
    chomp;
    unless (-e $_) { print STDERR $_, ': no such file or directory'; next; }
    unless (-f $_) { print STDERR $_, ': not a file';                next; }
    unless (-r $_) { print STDERR $_, ': permission denied';         next; }

    open FH, '<', $_ or die $_;

    $. = 0;

    while (<FH>) {
        my ($object) = m{ObjectContent\s*\(\s*"([^"]+)"\s*\)};

        if (defined $object) {
            push @{$OBJECT{$object}}, { file => $_, line => $. };
        }
    }

    close FH;
}

open FH, '<', $patternfile or die $patternfile;

$\ = "\n";
$, = '';

while (<FH>) {
    chomp;

    if (defined $OBJECT{$_}) {
        foreach my $X (@$OBJECT{$_}) {
            print $_, ': ', $X->{file}, '(', $X->{line}, ')';
        }
    }
}

close FH;

This will read in every file in filelist (or read from stdin), search every file for the 'ObjectContents' function calls - saving the filename and location of every pattern found. Once that is done, the pattern file is read in and checked for matching patterns.
# 3  
Old 12-16-2011
MySQL

Thanks Ludwig,
Actually I am new to this shell script and PERL and I want the file content which have to print line by line and reference file (.jsp, .js,.xsl )
For Example:
if in the input text file app.txt ,4 names are there like below
Code:
                      pasam
                      nithin 
                      anil 
                      eric

then I want the output file containing the content like

Code:
Starting the  New Name:******************
pasam
/start/login.jsp
/start/header.js
Starting the  New Name:******************
nithin
/start/logout.jsp
/start/header.js
Starting the  New Name:******************
 anil
Starting the  New Name:******************
eric
/scripts/log/login.xsl

Thanks a lot Ludwig.

Last edited by zxmaus; 12-19-2011 at 10:17 AM.. Reason: added tags
# 4  
Old 12-19-2011
Question

Hi Ludwig,
when I tried this command,
Code:
Code:
  find . -type f \( -name '*.jsp' -o -name '*.js' \) -print0 | xargs -0 grep -n chiru
I got the below errors :
find : missing conjunction
xargs: unknown option: -0

when I tried the below script by creating a file with name Plan.sh
Code:
Code:
  FILE=$(mktemp) trap "rm -f ${FILE}" EXIT  # built list of files find . -type f \( -name '*.jsp' -o -name '*.js' \) -print0 > ${FILE}  cat "${@}" | while read pattern; do     xargs -0 grep -n "${pattern}" < ${FILE} done

in the above code, I have given Input filename: testfile in place of < $ {FILE} here.

when I tried to run the script by giving >sh plan.sh, I got the following errors:

Code:
find: bad option -print0
plan.sh[8]: "${ @ }" the specified substitution is not valid for this command

when I tried the Perl code by replacing the inputfile name (testfile) instead of FH

Code:
#!/usr/bin/perl
# USAGE: patternfind patternlist [filelist]...

my $patternlist = shift @ARGV;
die unless defined $patternlist;

my %OBJECTS;
# key = object
# value = list of file/line pairs

while (<>) {
    chomp;
    unless (-e $_) { print STDERR $_, ': no such file or directory'; next; }
    unless (-f $_) { print STDERR $_, ': not a file';                next; }
    unless (-r $_) { print STDERR $_, ': permission denied';         next; }

    open FH, '<', $_ or die $_;

    $. = 0;

    while (<FH>) {
        my ($object) = m{ObjectContent\s*\(\s*"([^"]+)"\s*\)};

        if (defined $object) {
            push @{$OBJECT{$object}}, { file => $_, line => $. };
        }
    }

    close FH;
}

open FH, '<', $patternfile or die $patternfile;

$\ = "\n";
$, = '';

while (<FH>) {
    chomp;

    if (defined $OBJECT{$_}) {
        foreach my $X (@$OBJECT{$_}) {
            print $_, ': ', $X->{file}, '(', $X->{line}, ')';
        }
    }
}

close FH;

Then I got the following errors:
Code:
syntax error plan1.pl  line 39 at EOF
Execution of plan1.pl aborted due to compilation errors

when You are writing the solution for this , plz give the input filename with INPUT and Output filename with OUTPUT .

Thanks.

It is very urgent .Please help me in this .
# 5  
Old 12-19-2011
Hi there.

You wrote:
Quote:
Originally Posted by pasam
When I tried this command,
Code:
find . -type f \( -name '*.jsp' -o -name '*.js' \) -print0 | xargs -0 grep -n chiru

I got the below errors :
Code:
find : missing conjunction
xargs: unknown option: -0

I made a poor assumption that HP-UX 11 was more similar to a linux system, where find does have a -print0 option. But since you are running a system-V unix, you can use:
Code:
find . -type f \( -name '*.jsp' -o -name '*.js' \) -print | xargs grep -n chiru

which will work provided none of your file or directories contains whitespace or quotes -- see this for more information. If your files might contain whitespace or quotes, you will have to use:
Code:
find . -type f \( -name '*.jsp' -o -name '*.js' \) -print | sed -e 's/[ "'"'"']/\\&/g' | xargs grep -n chiru

to escape the separators.
Quote:
Originally Posted by pasam
When I tried the Perl code by replacing the inputfile name (testfile) instead of FH
Can you please explain what you mean by this?
Also, what version of perl are you running?
# 6  
Old 12-20-2011
Hi, Ludwing,
in one of my UNIX box
which I have the Perl version 5.8.2 built for PA-RISC1.1-thread-multi(with 25 registered patches).In this box I want to search for .jsp and .js reference file(with paths) for the Object names (20000 object names read line by line from a text file and to get the reference file along with Object Name )
in another UNIX box
I have the Perl version, v5.8.5 built for i386-linux-thread-multi
In this box, I want to search for reference files of .xsl, .xml for the ObjectNames.
The requirement is same as above.

I need the PERL code for the above two versions :
the requirement is: The code has to read the text file(which is having 20000 object names ) and print the OUTPUT file with reference file.

The output file have to print like this
ObjectName1
../startup/registration/login.jsp
../script/code/val.js
ObjectName2
../.................jsp
../..................jsp
../...............js







Thanks a lot .
# 7  
Old 12-20-2011
Something like the following?
Code:
#!/usr/bin/perl
use strict;
use warnings;
use File::Find;
# usage $0 <directory list>
open (my $input, '<', "names.txt")
    ||die "Couldn't open names.txt file:\n\t$!\n";
while (<$input>){
    print "$_\n";
    find(sub {wanted($_)} , @ARGV);
}

sub wanted{
    if (/\.jsp?$/){
        my $regex=shift;
        open(my $current_file, '<', $_) || print "Could not read $_:\n\t$!\n";
        FILE:
        while(<$current_file>){
            if(/$regex/){
                print "$File::Find::name";
                next FILE;
            }
        }
        close($current_file);
    }
}

As far as I know File::Find is a core module in both Perl versions you list.

Last edited by Skrynesaver; 12-20-2011 at 06:13 AM.. Reason: Anchored the file selection regex and added a warning for un-readable files
Login to Reply

« Previous Thread | Next Thread »
Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Shell script UNIX to read text file line by line tester111 Shell Programming and Scripting 4 11-05-2017 12:59 PM
Shell script to read a text file line by line & process it... new_to_shell Shell Programming and Scripting 8 09-13-2011 04:45 AM
search all file for particular text and make changes to line 3 alok.behria UNIX for Dummies Questions & Answers 10 06-30-2011 06:08 PM
perl search and replace - search in first line and replance in 2nd line arvindng Shell Programming and Scripting 3 02-04-2011 09:50 AM
get the fifth line of a text file into a shell script and trim the line to extract a WORD venu Shell Programming and Scripting 3 12-08-2010 08:46 PM
Search text from a file and print text and one previous line too kamranjalal Shell Programming and Scripting 6 01-06-2009 02:27 AM
Perl script to search a line and copy it to another line ammu Shell Programming and Scripting 3 12-29-2008 04:12 PM
reverse search a text file from a specified line PacificWonder Shell Programming and Scripting 14 08-28-2008 04:28 PM
Perl: Search for string on line then search and replace text Crypto Shell Programming and Scripting 4 01-04-2008 09:24 AM
Perl: taking text from a .txt file perleo Shell Programming and Scripting 2 06-19-2003 10:23 PM


All times are GMT -4. The time now is 10:48 AM.

Unix & Linux Forums Content Copyright©1993-2018. All Rights Reserved.
UNIX.COM Login
Username:
Password:  
Show Password