Unix/Linux Go Back    


Shell Programming and Scripting BSD, Linux, and UNIX shell scripting — Post awk, bash, csh, ksh, perl, php, python, sed, sh, shell scripts, and other shell scripting languages questions here.

PERL or SHELL Scrript to search in Directories by taking line by line from a text file

Shell Programming and Scripting


Closed    
 
Thread Tools Search this Thread Display Modes
    #1  
Old Unix and Linux 12-16-2011   -   Original Discussion by pasam
pasam pasam is offline
Registered User
 
Join Date: Dec 2011
Last Activity: 9 January 2012, 2:38 AM EST
Location: India
Posts: 9
Thanks: 0
Thanked 0 Times in 0 Posts
Unix or Linux Question PERL or SHELL Scrript to search in Directories by taking line by line from a text file

Unix box server version
***********

Code:
>uname  -r
B.11.00

>echo $SHELL
/usr/bin/ksh

--> in this server, I have the path like /IMbuild/dev/im0serv1
---> in that directory I have the folders startup(.jsp files nearly 100 jsp's ) and scripts(contains .js files nearly 100 files)
So, in .jsp,.js files there are two methods calling the Application ID like
returnMMCObjectContent("chiru");//
getDCCObjectContent("chiru");

like chiru I need to search for 22,000 articles.
till now I am using one command

Code:
find . -name  -exec  grep -l "chiru" {}\; | -exec  grep -n  "chiru" {}\;//

then I vl get the result like that method where ever the chiru was used and the file name with path like ../start/loutout.js,../filenames/login.jsp

so with this I can make a note that chiru was used name bcoz reference files were displayed.
If any reference file not came,that means chiru was unUsed name....

This whole part is MANUAL kind of thing I have to do CTRL+C and CTRL+V ..........for 22,000 times

MY REQUIREMENT IS:

--------------------
I want to take 10,000 names in a text file (test.txt) and directly pass as input file to the script and the script has to read the file line by line and searching in the sub directories(start and scripts) ,if reference has found,then those names has to go into one output file with that name and reference file(.jsp, .js) (out1.txt) , if any reference was not found then those names should go into another output file(out2.txt).
So I am asking for that script ;

either perl script or any UNIX shell script that can work in version:B.11.00



Another Unix box server
***********
Code:
>uname -r
2.6.9-89.0.20.ELsmp

>echo  $SHELL
ksh:echo: not found

This is different version of UNIX
The same script for the above server I want to execute in this server....


Thanks in Advance ........................

Last edited by vbe; 12-16-2011 at 06:51 AM.. Reason: used code tags next time for your data and code
Sponsored Links
    #2  
Old Unix and Linux 12-16-2011   -   Original Discussion by pasam
m.d.ludwig m.d.ludwig is offline
Registered User
 
Join Date: Dec 2010
Last Activity: 7 December 2012, 7:50 PM EST
Location: Albany, NY
Posts: 313
Thanks: 15
Thanked 60 Times in 60 Posts
I have to admit some puzzlement from:
Quote:
Originally Posted by pasam View Post
Code:
find . -name  -exec  grep -l "chiru" {}\; | -exec  grep -n  "chiru" {}\;//

This will run grep on every file and directory found - one at a time. Also, the output of these greps is then piped to -exec, which I am pretty sure does not exist. Restricting this to 'jsp' and 'js' files would help:

Code:
find . -type f \( -name '*.jsp' -o -name '*.js' \) -exec ....

And instead of using find to invoked grep on each file, use xargs:

Code:
find . -type f \( -name '*.jsp' -o -name '*.js' \) -print0 | xargs -0 grep -n chiru

The two greps are not needed, because grep -l regex filename will read the entire file it it does not find regex, and grep -n regex filename reads the entire file to find matches. By not using two greps, you are saving a extra (partial) read of files that contain regex.

Now the problem of finding matches of 10,000 patterns. Re-reading each file 10,000 times would be time consuming at best. But given that the pattern can occur anywhere in the file, that is what you are stuck with:

Code:
FILE=$(mktemp)
trap "rm -f ${FILE}" EXIT

# built list of files
find . -type f \( -name '*.jsp' -o -name '*.js' \) -print0 > ${FILE}

cat "${@}" | while read pattern; do
    xargs -0 grep -n "${pattern}" < ${FILE}
done

Now if the patterns occur only in lines like:
Quote:
Originally Posted by pasam View Post
returnMMCObjectContent("pattern");//
getDCCObjectContent("pattern");
then you use some perl to make this run faster.

Code:
#! /path/to/perl
# USAGE: patternfind patternlist [filelist]...

my $patternlist = shift @ARGV;
die unless defined $patternlist;

my %OBJECTS;
# key = object
# value = list of file/line pairs

while (<>) {
    chomp;
    unless (-e $_) { print STDERR $_, ': no such file or directory'; next; }
    unless (-f $_) { print STDERR $_, ': not a file';                next; }
    unless (-r $_) { print STDERR $_, ': permission denied';         next; }

    open FH, '<', $_ or die $_;

    $. = 0;

    while (<FH>) {
        my ($object) = m{ObjectContent\s*\(\s*"([^"]+)"\s*\)};

        if (defined $object) {
            push @{$OBJECT{$object}}, { file => $_, line => $. };
        }
    }

    close FH;
}

open FH, '<', $patternfile or die $patternfile;

$\ = "\n";
$, = '';

while (<FH>) {
    chomp;

    if (defined $OBJECT{$_}) {
        foreach my $X (@$OBJECT{$_}) {
            print $_, ': ', $X->{file}, '(', $X->{line}, ')';
        }
    }
}

close FH;

This will read in every file in filelist (or read from stdin), search every file for the 'ObjectContents' function calls - saving the filename and location of every pattern found. Once that is done, the pattern file is read in and checked for matching patterns.
Sponsored Links
    #3  
Old Unix and Linux 12-16-2011   -   Original Discussion by pasam
pasam pasam is offline
Registered User
 
Join Date: Dec 2011
Last Activity: 9 January 2012, 2:38 AM EST
Location: India
Posts: 9
Thanks: 0
Thanked 0 Times in 0 Posts
MySQL

Thanks Ludwig,
Actually I am new to this shell script and PERL and I want the file content which have to print line by line and reference file (.jsp, .js,.xsl )
For Example:
if in the input text file app.txt ,4 names are there like below

Code:
                      pasam
                      nithin 
                      anil 
                      eric

then I want the output file containing the content like


Code:
Starting the  New Name:******************
pasam
/start/login.jsp
/start/header.js
Starting the  New Name:******************
nithin
/start/logout.jsp
/start/header.js
Starting the  New Name:******************
 anil
Starting the  New Name:******************
eric
/scripts/log/login.xsl

Thanks a lot Ludwig.

Last edited by zxmaus; 12-19-2011 at 11:17 AM.. Reason: added tags
    #4  
Old Unix and Linux 12-19-2011   -   Original Discussion by pasam
pasam pasam is offline
Registered User
 
Join Date: Dec 2011
Last Activity: 9 January 2012, 2:38 AM EST
Location: India
Posts: 9
Thanks: 0
Thanked 0 Times in 0 Posts
Unix or Linux Question

Hi Ludwig,
when I tried this command,
Code:

Code:
  find . -type f \( -name '*.jsp' -o -name '*.js' \) -print0 | xargs -0 grep -n chiru
I got the below errors :
find : missing conjunction
xargs: unknown option: -0

when I tried the below script by creating a file with name Plan.sh
Code:

Code:
  FILE=$(mktemp) trap "rm -f ${FILE}" EXIT  # built list of files find . -type f \( -name '*.jsp' -o -name '*.js' \) -print0 > ${FILE}  cat "${@}" | while read pattern; do     xargs -0 grep -n "${pattern}" < ${FILE} done

in the above code, I have given Input filename: testfile in place of < $ {FILE} here.

when I tried to run the script by giving >sh plan.sh, I got the following errors:


Code:
find: bad option -print0
plan.sh[8]: "${ @ }" the specified substitution is not valid for this command

when I tried the Perl code by replacing the inputfile name (testfile) instead of FH


Code:
#!/usr/bin/perl
# USAGE: patternfind patternlist [filelist]...

my $patternlist = shift @ARGV;
die unless defined $patternlist;

my %OBJECTS;
# key = object
# value = list of file/line pairs

while (<>) {
    chomp;
    unless (-e $_) { print STDERR $_, ': no such file or directory'; next; }
    unless (-f $_) { print STDERR $_, ': not a file';                next; }
    unless (-r $_) { print STDERR $_, ': permission denied';         next; }

    open FH, '<', $_ or die $_;

    $. = 0;

    while (<FH>) {
        my ($object) = m{ObjectContent\s*\(\s*"([^"]+)"\s*\)};

        if (defined $object) {
            push @{$OBJECT{$object}}, { file => $_, line => $. };
        }
    }

    close FH;
}

open FH, '<', $patternfile or die $patternfile;

$\ = "\n";
$, = '';

while (<FH>) {
    chomp;

    if (defined $OBJECT{$_}) {
        foreach my $X (@$OBJECT{$_}) {
            print $_, ': ', $X->{file}, '(', $X->{line}, ')';
        }
    }
}

close FH;

Then I got the following errors:

Code:
syntax error plan1.pl  line 39 at EOF
Execution of plan1.pl aborted due to compilation errors

when You are writing the solution for this , plz give the input filename with INPUT and Output filename with OUTPUT .

Thanks.

It is very urgent .Please help me in this .
Sponsored Links
    #5  
Old Unix and Linux 12-19-2011   -   Original Discussion by pasam
m.d.ludwig m.d.ludwig is offline
Registered User
 
Join Date: Dec 2010
Last Activity: 7 December 2012, 7:50 PM EST
Location: Albany, NY
Posts: 313
Thanks: 15
Thanked 60 Times in 60 Posts
Hi there.

You wrote:
Quote:
Originally Posted by pasam View Post
When I tried this command,

Code:
find . -type f \( -name '*.jsp' -o -name '*.js' \) -print0 | xargs -0 grep -n chiru

I got the below errors :

Code:
find : missing conjunction
xargs: unknown option: -0

I made a poor assumption that HP-UX 11 was more similar to a linux system, where find does have a -print0 option. But since you are running a system-V unix, you can use:

Code:
find . -type f \( -name '*.jsp' -o -name '*.js' \) -print | xargs grep -n chiru

which will work provided none of your file or directories contains whitespace or quotes -- see this for more information. If your files might contain whitespace or quotes, you will have to use:

Code:
find . -type f \( -name '*.jsp' -o -name '*.js' \) -print | sed -e 's/[ "'"'"']/\\&/g' | xargs grep -n chiru

to escape the separators.
Quote:
Originally Posted by pasam View Post
When I tried the Perl code by replacing the inputfile name (testfile) instead of FH
Can you please explain what you mean by this?
Also, what version of perl are you running?
Sponsored Links
    #6  
Old Unix and Linux 12-20-2011   -   Original Discussion by pasam
pasam pasam is offline
Registered User
 
Join Date: Dec 2011
Last Activity: 9 January 2012, 2:38 AM EST
Location: India
Posts: 9
Thanks: 0
Thanked 0 Times in 0 Posts
Hi, Ludwing,
in one of my UNIX box
which I have the Perl version 5.8.2 built for PA-RISC1.1-thread-multi(with 25 registered patches).In this box I want to search for .jsp and .js reference file(with paths) for the Object names (20000 object names read line by line from a text file and to get the reference file along with Object Name )
in another UNIX box
I have the Perl version, v5.8.5 built for i386-linux-thread-multi
In this box, I want to search for reference files of .xsl, .xml for the ObjectNames.
The requirement is same as above.

I need the PERL code for the above two versions :
the requirement is: The code has to read the text file(which is having 20000 object names ) and print the OUTPUT file with reference file.

The output file have to print like this
ObjectName1
../startup/registration/login.jsp
../script/code/val.js
ObjectName2
../.................jsp
../..................jsp
../...............js







Thanks a lot .
Sponsored Links
    #7  
Old Unix and Linux 12-20-2011   -   Original Discussion by pasam
Skrynesaver's Unix or Linux Image
Skrynesaver Skrynesaver is offline Forum Advisor  
Grumpy old geek
 
Join Date: Mar 2011
Last Activity: 13 November 2017, 5:30 AM EST
Location: Éire
Posts: 931
Thanks: 25
Thanked 223 Times in 217 Posts
Something like the following?

Code:
#!/usr/bin/perl
use strict;
use warnings;
use File::Find;
# usage $0 <directory list>
open (my $input, '<', "names.txt")
    ||die "Couldn't open names.txt file:\n\t$!\n";
while (<$input>){
    print "$_\n";
    find(sub {wanted($_)} , @ARGV);
}

sub wanted{
    if (/\.jsp?$/){
        my $regex=shift;
        open(my $current_file, '<', $_) || print "Could not read $_:\n\t$!\n";
        FILE:
        while(<$current_file>){
            if(/$regex/){
                print "$File::Find::name";
                next FILE;
            }
        }
        close($current_file);
    }
}

As far as I know File::Find is a core module in both Perl versions you list.

Last edited by Skrynesaver; 12-20-2011 at 07:13 AM.. Reason: Anchored the file selection regex and added a warning for un-readable files
Sponsored Links
Closed

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Linux More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Shell script to read a text file line by line & process it... new_to_shell Shell Programming and Scripting 8 09-13-2011 05:45 AM
perl search and replace - search in first line and replance in 2nd line arvindng Shell Programming and Scripting 3 02-04-2011 10:50 AM
get the fifth line of a text file into a shell script and trim the line to extract a WORD venu Shell Programming and Scripting 3 12-08-2010 09:46 PM
Search text from a file and print text and one previous line too kamranjalal Shell Programming and Scripting 6 01-06-2009 03:27 AM
Perl: Search for string on line then search and replace text Crypto Shell Programming and Scripting 4 01-04-2008 10:24 AM



All times are GMT -4. The time now is 01:47 AM.