Finding a specific pattern from thousands of files ????


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Finding a specific pattern from thousands of files ????
# 1  
Old 02-17-2006
Finding a specific pattern from thousands of files ????

Hi All,

I want to find a specific pattern from approximately 400000 files on solaris platform. Its very heavy for me to grep that pattern to each file individually.

Can anybody suggest me some way to search for specific pattern (alpha numeric) from these forty thousand files. Please note that out of 40000 files only few files has this specific pattern.

I would really be greatful. This really can make my work easy.

Thanks,
Aru
# 2  
Old 02-17-2006
one way i could think of is to run multiple instance of grep together dividing the number of files to each of the instance.

If you are running it in live env take care of the no of instance, the load your system could take on
# 3  
Old 02-17-2006
Try to use find command coupled with -exec option.
If you know the pattern of filenames you can narrow down no. of files to be searched
with wild card characters.
# 4  
Old 02-17-2006
Quote:
Originally Posted by shriashishpatil
Try to use find command coupled with -exec option.
If you know the pattern of filenames you can narrow down no. of files to be searched
with wild card characters.
request was to search for a pattern within the file and the user is not sure of the file pattern which contain the search pattern
only input is the filenames and the pattern to be searched within the file

moreover find with exec is not an effective way for large runs
use find piped to xargs
# 5  
Old 02-17-2006
Ya matrixmadhan...... you are right . I am not sure which files out of those 40000 file contains a search pattern..

suppose that I have files 1x,2x,3x........,40000x .
So, how are you going to use find piped to xargs , say to search pattern "N1cd56" in the above 40000 files.

please tell me the usage.

Thanks,
Aru
# 6  
Old 02-17-2006
that was just a comparison what i said about exec with find and xargs with find

suggested was:
instance 1:
for files in <file 1... file10000>
do
grep <pattern> files
done

instance 2:
for files in <file 10001... file20000>
do
grep <pattern> files
done

similarly for instance 3 and 4.
you can just have list of filenames in a particular file and just furnish the required filenames (file 1.. file 10000) to each of the instance

hope i am clear
# 7  
Old 02-17-2006
Let me try once......

Thnx.
Aru
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Finding lines of specific size in files using sed

i am using sed to detect any lines that are not exactly 21. the following gives me the lines that ARE exactly 21. i want the opposite , i want the two lines that are not size 21 (shown in bold) type a.a 000008050110010201NNN 000008060810010201NNN 21212000008070110010201NNN... (5 Replies)
Discussion started by: boncuk
5 Replies

2. Shell Programming and Scripting

Finding all files based on pattern

Hi All, I need to find all files in a directory which are containing specific pattern. Thing is that file name should not consider if pattern is only in commented area. all contents which are under /* */ are commented all lines which are starting with -- or if -- is a part of some sentence... (13 Replies)
Discussion started by: Lakshman_Gupta
13 Replies

3. UNIX for Dummies Questions & Answers

Finding Files only under a specific FileSystem

Hi, I am using AIX and one of my file systems is getting filled up and I need to track with files are occupying more volume. Filesystem GB blocks Free %Used Iused %Iused Mounted on /dev/nhdb_lv 2110.00 63.80 97% 76525 1% /nhdb under the Mount Point /nhdb... (1 Reply)
Discussion started by: zulfi123786
1 Replies

4. Shell Programming and Scripting

Finding log files that match number pattern

I have logs files which are generated each day depending on how many processes are running. Some days it could spin up 30 processes. Other days it could spin up 50. The log files all have the same pattern with the number being the different factor. e.g. LOG_FILE_1.log LOG_FILE_2.log etc etc ... (2 Replies)
Discussion started by: atelford
2 Replies

5. Shell Programming and Scripting

Finding duplicates in a file excluding specific pattern

I have unix file like below >newuser newuser <hello hello newone I want to find the unique values in the file(excluding <,>),so that the out put should be >newuser <hello newone can any body tell me what is command to get this new file. (7 Replies)
Discussion started by: shiva2985
7 Replies

6. Shell Programming and Scripting

Finding 4 current files having specific File Name pattern

Hi All, I am trying to find 4 latest files inside one folder having following File Name pattern and store them into 4 different variables and then use for processing in my shell script. File name is fixed length. 1) Each file starts with = ABCJmdmfbsjop letters + 7 Digit Number... (6 Replies)
Discussion started by: lancesunny
6 Replies

7. UNIX for Dummies Questions & Answers

finding all files that do not match a certain pattern

I hope I'm asking this the right way -- I've been sending out a lot of resumes and some of them I saw on Craigslist -- so I named the file as 'Craigslist -- (filename)'. Well I noticed that at least one of the files was misspelled as 'Craigslit.' I want to eventually try to write a shell... (5 Replies)
Discussion started by: Straitsfan
5 Replies

8. Shell Programming and Scripting

Finding conserved pattern in different files

Hi power user, For examples, I have three different files: file 1: file2: file 3: AAA CCC ZZZ BBB BBB CCC CCC DDD DDD DDD TTT AAA EEE AAA XXX I... (8 Replies)
Discussion started by: anjas
8 Replies

9. UNIX for Dummies Questions & Answers

help with finding specific files

before i get to it, i would like to say this is the greatest unix site ive ever seen, and im glad to see so many people are out there to help. thanks well, im trying to make myself a script where i can specify a directory and a file size so that my script will show me any files larger than the... (5 Replies)
Discussion started by: linuxlaptop
5 Replies

10. Shell Programming and Scripting

finding duplicate files by size and finding pattern matching and its count

Hi, I have a challenging task,in which i have to find the duplicate files by its name and size,then i need to take anyone of the file.Then i need to open the file and find for more than one pattern and count of that pattern. Note:These are the samples of two files,but i can have more... (2 Replies)
Discussion started by: jerome Sukumar
2 Replies
Login or Register to Ask a Question