Search for patterns in thousands of files | Unix Linux Forums | Shell Programming and Scripting

  Go Back    


Shell Programming and Scripting Post questions about KSH, CSH, SH, BASH, PERL, PHP, SED, AWK and OTHER shell scripts and shell scripting languages here.

Search for patterns in thousands of files

Shell Programming and Scripting


Closed Thread    
 
Thread Tools Search this Thread Display Modes
    #1  
Old 03-26-2013
danish0909 danish0909 is offline
Registered User
 
Join Date: Dec 2011
Last Activity: 15 December 2013, 1:20 PM EST
Posts: 37
Thanks: 8
Thanked 0 Times in 0 Posts
Search for patterns in thousands of files

Hi All,


I want to search for a certain string in thousands of files and these files are distributed over different directories created daily. For that I created a small script in bash but while running it I am getting the below error:

/ms.sh: xrealloc: subst.c:5173: cannot allocate 268435456 bytes (536977408 bytes allocated)

Pasting the code that I wrote:


Code:
#!/usr/local/bin/bash

for i in `cat msisdn_u.txt`
do

cd /comptel4/elink/backup1/output/vas/NG0/20130301
find ./*GPX.Z|xargs zcat|grep $i; cd ..
cd /comptel4/elink/backup1/output/vas/NG0/20130302
find ./*GPX.Z|xargs zcat|grep $i; cd ..
cd /comptel4/elink/backup1/output/vas/NG0/20130303
find ./*GPX.Z|xargs zcat|grep $i; cd ..
cd /comptel4/elink/backup1/output/vas/NG0/20130304
find ./*GPX.Z|xargs zcat|grep $i; cd ..
cd /comptel4/elink/backup1/output/vas/NG0/20130305
find ./*GPX.Z|xargs zcat|grep $i; cd ..
cd /comptel4/elink/backup1/output/vas/NG0/20130306
find ./*GPX.Z|xargs zcat|grep $i; cd ..
cd /comptel4/elink/backup1/output/vas/NG0/20130307
find ./*GPX.Z|xargs zcat|grep $i; cd ..
..
..
..
done

This is in the patterns file:


Code:
more msisdn_u.txt
0564891888
0500555401
0563433343
0561132174
0562714661
0543210172
0503588147
0541400224
0564445889
0544998887
0564543055
0544095240
0563211334

Please advise as I need to find out and report it to the management.

Thanks

Danish

Last edited by radoulov; 03-26-2013 at 07:21 AM..
Sponsored Links
    #2  
Old 03-26-2013
PikK45 PikK45 is offline
Registered User
 
Join Date: Jul 2012
Last Activity: 7 February 2014, 8:50 AM EST
Location: Chennai
Posts: 560
Thanks: 41
Thanked 72 Times in 70 Posts
Why are not using a single find command for all the directories??


Code:
find /comptel4/elink/backup1/output/vas/NG0/ -name "*.GPX.Z" -exec zgrep -il $i {} \;

or

Code:
find /comptel4/elink/backup1/output/vas/NG0/ -name "*.GPX.Z" -print | xargs zgrep -il $i

Hope this helps
Sponsored Links
    #3  
Old 03-26-2013
danish0909 danish0909 is offline
Registered User
 
Join Date: Dec 2011
Last Activity: 15 December 2013, 1:20 PM EST
Posts: 37
Thanks: 8
Thanked 0 Times in 0 Posts
Thanks for your suggestions PikK45..but is the command descending into directories..because I dont see any output..the command returns back to the command prompt
    #4  
Old 03-26-2013
PikK45 PikK45 is offline
Registered User
 
Join Date: Jul 2012
Last Activity: 7 February 2014, 8:50 AM EST
Location: Chennai
Posts: 560
Thanks: 41
Thanked 72 Times in 70 Posts
can you show us what you did??
Sponsored Links
    #5  
Old 03-26-2013
danish0909 danish0909 is offline
Registered User
 
Join Date: Dec 2011
Last Activity: 15 December 2013, 1:20 PM EST
Posts: 37
Thanks: 8
Thanked 0 Times in 0 Posts
I am on hp ux


Code:
/comptel/elink> find /comptel4/elink/backup1/output/vas/NG0/ -name "*.GPX.Z" -exec zgrep -il $i {} \;
/comptel/elink>


Code:
/comptel4/elink/backup1/output/vas/NG0> find /comptel4/elink/backup1/output/vas/NG0/ -name "*.GPX.Z" -print
/comptel4/elink/backup1/output/vas/NG0>

Sponsored Links
    #6  
Old 03-26-2013
PikK45 PikK45 is offline
Registered User
 
Join Date: Jul 2012
Last Activity: 7 February 2014, 8:50 AM EST
Location: Chennai
Posts: 560
Thanks: 41
Thanked 72 Times in 70 Posts
Are there files with .GPX.Z extension in the "/comptel4/elink/backup1/output/vas/NG0/" directory or its sub-directories??
Sponsored Links
    #7  
Old 03-26-2013
danish0909 danish0909 is offline
Registered User
 
Join Date: Dec 2011
Last Activity: 15 December 2013, 1:20 PM EST
Posts: 37
Thanks: 8
Thanked 0 Times in 0 Posts
I have run it again but I am getting the same error. There are subdirectories under it so I have given a patter for it.

Code:
/comptel4/elink/backup1/output/vas/NG0> ./ms.sh
./ms.sh: xrealloc: subst.c:5173: cannot allocate 268435456 bytes (536936448 bytes allocated)
/comptel4/elink/backup1/output/vas/NG0> more ms.sh
#!/usr/local/bin/bash

for i in `cat msisdn_u.txt`
do

find /comptel4/elink/backup1/output/vas/NG0/201303* -name "*GPX.Z" -print | xargs zgrep -il $i
done


Last edited by Franklin52; 03-26-2013 at 07:41 AM.. Reason: Please use code tags for data and code samples
Sponsored Links
Closed Thread

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

More UNIX and Linux Forum Topics You Might Find Helpful
Thread Thread Starter Forum Replies Last Post
Search all files based on first and in all listed files search the second patterns Black-Linux SuSE 1 05-01-2011 06:10 AM
to read two files, search for patterns and store the output in third file vaibhavkorde Shell Programming and Scripting 3 04-14-2011 03:48 AM
script to search patterns inside list of files dr46014 UNIX for Dummies Questions & Answers 2 10-19-2010 01:02 PM
Best way to search for patterns in huge text files andy2000 UNIX for Advanced & Expert Users 8 01-09-2010 05:39 PM
how to search files efficiently using patterns arunsubbhian UNIX Desktop for Dummies Questions & Answers 2 09-07-2007 02:42 AM



All times are GMT -4. The time now is 06:12 AM.