PERL or SHELL Scrript to search in Directories by taking line by line from a text file


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting PERL or SHELL Scrript to search in Directories by taking line by line from a text file
# 1  
Old 12-16-2011
Question PERL or SHELL Scrript to search in Directories by taking line by line from a text file

Unix box server version
***********
Code:
>uname  -r
B.11.00

>echo $SHELL
/usr/bin/ksh

--> in this server, I have the path like /IMbuild/dev/im0serv1
---> in that directory I have the folders startup(.jsp files nearly 100 jsp's ) and scripts(contains .js files nearly 100 files)
So, in .jsp,.js files there are two methods calling the Application ID like
returnMMCObjectContent("chiru");//
getDCCObjectContent("chiru");

like chiru I need to search for 22,000 articles.
till now I am using one command
Code:
find . -name  -exec  grep -l "chiru" {}\; | -exec  grep -n  "chiru" {}\;//

then I vl get the result like that method where ever the chiru was used and the file name with path like ../start/loutout.js,../filenames/login.jsp

so with this I can make a note that chiru was used name bcoz reference files were displayed.
If any reference file not came,that means chiru was unUsed name....

This whole part is MANUAL kind of thing I have to do CTRL+C and CTRL+V ..........for 22,000 times

MY REQUIREMENT IS:

--------------------
I want to take 10,000 names in a text file (test.txt) and directly pass as input file to the script and the script has to read the file line by line and searching in the sub directories(start and scripts) ,if reference has found,then those names has to go into one output file with that name and reference file(.jsp, .js) (out1.txt) , if any reference was not found then those names should go into another output file(out2.txt).
So I am asking for that script ;

either perl script or any UNIX shell script that can work in version:B.11.00



Another Unix box server
***********
Code:
>uname -r
2.6.9-89.0.20.ELsmp

>echo  $SHELL
ksh:echo: not found

This is different version of UNIX
The same script for the above server I want to execute in this server....


Thanks in Advance ........................

Last edited by vbe; 12-16-2011 at 06:51 AM.. Reason: used code tags next time for your data and code
# 2  
Old 12-16-2011
I have to admit some puzzlement from:
Quote:
Originally Posted by pasam
Code:
find . -name  -exec  grep -l "chiru" {}\; | -exec  grep -n  "chiru" {}\;//

This will run grep on every file and directory found - one at a time. Also, the output of these greps is then piped to -exec, which I am pretty sure does not exist. Restricting this to 'jsp' and 'js' files would help:
Code:
find . -type f \( -name '*.jsp' -o -name '*.js' \) -exec ....

And instead of using find to invoked grep on each file, use xargs:
Code:
find . -type f \( -name '*.jsp' -o -name '*.js' \) -print0 | xargs -0 grep -n chiru

The two greps are not needed, because grep -l regex filename will read the entire file it it does not find regex, and grep -n regex filename reads the entire file to find matches. By not using two greps, you are saving a extra (partial) read of files that contain regex.

Now the problem of finding matches of 10,000 patterns. Re-reading each file 10,000 times would be time consuming at best. But given that the pattern can occur anywhere in the file, that is what you are stuck with:
Code:
FILE=$(mktemp)
trap "rm -f ${FILE}" EXIT

# built list of files
find . -type f \( -name '*.jsp' -o -name '*.js' \) -print0 > ${FILE}

cat "${@}" | while read pattern; do
    xargs -0 grep -n "${pattern}" < ${FILE}
done

Now if the patterns occur only in lines like:
Quote:
Originally Posted by pasam
returnMMCObjectContent("pattern");//
getDCCObjectContent("pattern");
then you use some perl to make this run faster.
Code:
#! /path/to/perl
# USAGE: patternfind patternlist [filelist]...

my $patternlist = shift @ARGV;
die unless defined $patternlist;

my %OBJECTS;
# key = object
# value = list of file/line pairs

while (<>) {
    chomp;
    unless (-e $_) { print STDERR $_, ': no such file or directory'; next; }
    unless (-f $_) { print STDERR $_, ': not a file';                next; }
    unless (-r $_) { print STDERR $_, ': permission denied';         next; }

    open FH, '<', $_ or die $_;

    $. = 0;

    while (<FH>) {
        my ($object) = m{ObjectContent\s*\(\s*"([^"]+)"\s*\)};

        if (defined $object) {
            push @{$OBJECT{$object}}, { file => $_, line => $. };
        }
    }

    close FH;
}

open FH, '<', $patternfile or die $patternfile;

$\ = "\n";
$, = '';

while (<FH>) {
    chomp;

    if (defined $OBJECT{$_}) {
        foreach my $X (@$OBJECT{$_}) {
            print $_, ': ', $X->{file}, '(', $X->{line}, ')';
        }
    }
}

close FH;

This will read in every file in filelist (or read from stdin), search every file for the 'ObjectContents' function calls - saving the filename and location of every pattern found. Once that is done, the pattern file is read in and checked for matching patterns.
# 3  
Old 12-16-2011
MySQL

Thanks Ludwig,
Actually I am new to this shell script and PERL and I want the file content which have to print line by line and reference file (.jsp, .js,.xsl )
For Example:
if in the input text file app.txt ,4 names are there like below
Code:
                      pasam
                      nithin 
                      anil 
                      eric

then I want the output file containing the content like

Code:
Starting the  New Name:******************
pasam
/start/login.jsp
/start/header.js
Starting the  New Name:******************
nithin
/start/logout.jsp
/start/header.js
Starting the  New Name:******************
 anil
Starting the  New Name:******************
eric
/scripts/log/login.xsl

Thanks a lot Ludwig.

Last edited by zxmaus; 12-19-2011 at 11:17 AM.. Reason: added tags
# 4  
Old 12-19-2011
Question

Hi Ludwig,
when I tried this command,
Code:
Code:
  find . -type f \( -name '*.jsp' -o -name '*.js' \) -print0 | xargs -0 grep -n chiru
I got the below errors :
find : missing conjunction
xargs: unknown option: -0

when I tried the below script by creating a file with name Plan.sh
Code:
Code:
  FILE=$(mktemp) trap "rm -f ${FILE}" EXIT  # built list of files find . -type f \( -name '*.jsp' -o -name '*.js' \) -print0 > ${FILE}  cat "${@}" | while read pattern; do     xargs -0 grep -n "${pattern}" < ${FILE} done

in the above code, I have given Input filename: testfile in place of < $ {FILE} here.

when I tried to run the script by giving >sh plan.sh, I got the following errors:

Code:
find: bad option -print0
plan.sh[8]: "${ @ }" the specified substitution is not valid for this command

when I tried the Perl code by replacing the inputfile name (testfile) instead of FH

Code:
#!/usr/bin/perl
# USAGE: patternfind patternlist [filelist]...

my $patternlist = shift @ARGV;
die unless defined $patternlist;

my %OBJECTS;
# key = object
# value = list of file/line pairs

while (<>) {
    chomp;
    unless (-e $_) { print STDERR $_, ': no such file or directory'; next; }
    unless (-f $_) { print STDERR $_, ': not a file';                next; }
    unless (-r $_) { print STDERR $_, ': permission denied';         next; }

    open FH, '<', $_ or die $_;

    $. = 0;

    while (<FH>) {
        my ($object) = m{ObjectContent\s*\(\s*"([^"]+)"\s*\)};

        if (defined $object) {
            push @{$OBJECT{$object}}, { file => $_, line => $. };
        }
    }

    close FH;
}

open FH, '<', $patternfile or die $patternfile;

$\ = "\n";
$, = '';

while (<FH>) {
    chomp;

    if (defined $OBJECT{$_}) {
        foreach my $X (@$OBJECT{$_}) {
            print $_, ': ', $X->{file}, '(', $X->{line}, ')';
        }
    }
}

close FH;

Then I got the following errors:
Code:
syntax error plan1.pl  line 39 at EOF
Execution of plan1.pl aborted due to compilation errors

when You are writing the solution for this , plz give the input filename with INPUT and Output filename with OUTPUT .

Thanks.

It is very urgent .Please help me in this .
# 5  
Old 12-19-2011
Hi there.

You wrote:
Quote:
Originally Posted by pasam
When I tried this command,
Code:
find . -type f \( -name '*.jsp' -o -name '*.js' \) -print0 | xargs -0 grep -n chiru

I got the below errors :
Code:
find : missing conjunction
xargs: unknown option: -0

I made a poor assumption that HP-UX 11 was more similar to a linux system, where find does have a -print0 option. But since you are running a system-V unix, you can use:
Code:
find . -type f \( -name '*.jsp' -o -name '*.js' \) -print | xargs grep -n chiru

which will work provided none of your file or directories contains whitespace or quotes -- see this for more information. If your files might contain whitespace or quotes, you will have to use:
Code:
find . -type f \( -name '*.jsp' -o -name '*.js' \) -print | sed -e 's/[ "'"'"']/\\&/g' | xargs grep -n chiru

to escape the separators.
Quote:
Originally Posted by pasam
When I tried the Perl code by replacing the inputfile name (testfile) instead of FH
Can you please explain what you mean by this?
Also, what version of perl are you running?
# 6  
Old 12-20-2011
Hi, Ludwing,
in one of my UNIX box
which I have the Perl version 5.8.2 built for PA-RISC1.1-thread-multi(with 25 registered patches).In this box I want to search for .jsp and .js reference file(with paths) for the Object names (20000 object names read line by line from a text file and to get the reference file along with Object Name )
in another UNIX box
I have the Perl version, v5.8.5 built for i386-linux-thread-multi
In this box, I want to search for reference files of .xsl, .xml for the ObjectNames.
The requirement is same as above.

I need the PERL code for the above two versions :
the requirement is: The code has to read the text file(which is having 20000 object names ) and print the OUTPUT file with reference file.

The output file have to print like this
ObjectName1
../startup/registration/login.jsp
../script/code/val.js
ObjectName2
../.................jsp
../..................jsp
../...............js







Thanks a lot .
# 7  
Old 12-20-2011
Something like the following?
Code:
#!/usr/bin/perl
use strict;
use warnings;
use File::Find;
# usage $0 <directory list>
open (my $input, '<', "names.txt")
    ||die "Couldn't open names.txt file:\n\t$!\n";
while (<$input>){
    print "$_\n";
    find(sub {wanted($_)} , @ARGV);
}

sub wanted{
    if (/\.jsp?$/){
        my $regex=shift;
        open(my $current_file, '<', $_) || print "Could not read $_:\n\t$!\n";
        FILE:
        while(<$current_file>){
            if(/$regex/){
                print "$File::Find::name";
                next FILE;
            }
        }
        close($current_file);
    }
}

As far as I know File::Find is a core module in both Perl versions you list.

Last edited by Skrynesaver; 12-20-2011 at 07:13 AM.. Reason: Anchored the file selection regex and added a warning for un-readable files
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Shell script UNIX to read text file line by line

i have a text file as belows, it includes 2 columns, 1st is the column name, 2nd is the file_name data_file.txt column_name file_name col1 file1 col2 file2 col3 file1 col4 file1 col5 file2 now, i would like to... (4 Replies)
Discussion started by: tester111
4 Replies

2. Shell Programming and Scripting

Shell script to read a text file line by line & process it...

Hi , I am trying to write an shell, which reads a text file (from a location) having a list of numbers of strictly 5 digits only ex: 33144 Now my script will check : 1) that each entry is only 5 digits & numeric only, no alphabets, & its not empty. 2)then it executes a shell script called... (8 Replies)
Discussion started by: new_to_shell
8 Replies

3. UNIX for Dummies Questions & Answers

search all file for particular text and make changes to line 3

Hi All, I am sitting on HPUX. I want to change the exit into #exit, which appears into 3red line of code in shell scripting, wondering how shell script to be called up to perform action. I have following code in all files. Now, I need to find the text exit and replace into #exit. #!/sbin/sh... (10 Replies)
Discussion started by: alok.behria
10 Replies

4. Shell Programming and Scripting

perl search and replace - search in first line and replance in 2nd line

Dear All, i want to search particular string and want to replance next line value. following is the test file. search string is tmp,??? ,10:1 "???" may contain any 3 character it should remain the same and next line replace with ,10:50 tmp,123 --- if match tmp,??? then... (3 Replies)
Discussion started by: arvindng
3 Replies

5. Shell Programming and Scripting

get the fifth line of a text file into a shell script and trim the line to extract a WORD

FOLKS , i have a text file that is generated automatically of an another korn shell script, i want to bring in the fifth line of the text file in to my korn shell script and look for a particular word in the line . Can you all share some thoughts on this one. thanks... Venu (3 Replies)
Discussion started by: venu
3 Replies

6. Shell Programming and Scripting

Search text from a file and print text and one previous line too

Hi, Please let me know how to find text and print text and its previous line. Please don't get irritated few days back I asked text and next line. I am using HP-UX 11.11 Thanks for your help. (6 Replies)
Discussion started by: kamranjalal
6 Replies

7. Shell Programming and Scripting

Perl script to search a line and copy it to another line

Hi I have a log file (say log.txt). I have to search for a line which has the string ( say ERROR) in the log file and copy 15 lines after this into another file (say error.txt). Can someone give me the code and this has to be in PERL Thanks in advance Ammu (3 Replies)
Discussion started by: ammu
3 Replies

8. Shell Programming and Scripting

reverse search a text file from a specified line

Hello All, I have a following task that I need to accomplish through a script or program and I am looking for some help as I have exhausted my ideas. 1. given: a text file with thousands of lines 2. find: pattern A in file and get line number ( grep -n works) 3. find: the first occurence of... (14 Replies)
Discussion started by: PacificWonder
14 Replies

9. UNIX for Dummies Questions & Answers

how can search a String in one text file and replace the whole line in another file

i am very new to UNIX plz help me in this scenario i have two text files as below file1.txt name=Rajakumar. Discipline=Electronics and communication. Designation=software Engineer. file2.txt name=Kannan. Discipline=Mechanical. Designation=CADD Design Engineer. ... (6 Replies)
Discussion started by: kkraja
6 Replies

10. Shell Programming and Scripting

Perl: Search for string on line then search and replace text

Hi All, I have a file that I need to be able to find a pattern match on a line, search that line for a text pattern, and replace that text. An example of 4 lines in my file is: 1. MatchText_randomNumberOfText moreData ReplaceMe moreData 2. MatchText_randomNumberOfText moreData moreData... (4 Replies)
Discussion started by: Crypto
4 Replies
Login or Register to Ask a Question