Sponsored Content
Top Forums Shell Programming and Scripting grep 1000s of files with 1000s of grep values Post 302720311 by mantis on Tuesday 23rd of October 2012 06:20:05 PM
Old 10-23-2012
grep 1000s of files with 1000s of grep values

Hi,

I have around 200,000 files in a given directory.

I need to cat each of these files and grep them for thousands of identifier values (or strings) in a given text file.

The text file looks something like this:

1234
1243545
1234353
121324

etc with thousands of entries.

Can you please assist how I can do this, and in the most efficient manner possible because this script will no doubt take a long time to run.

Thanks in advance.

Mantis
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

grep a list of values

Hi everybody! :) :D :D :) it's great to be here since this is my first post. touch /base/oracle/FRA/XMUT00/RMAN_FLAG touch /base/oracle/FRA/XRLL00/RMAN_FLAG find directory name containing RMAN_FLAG : $ find /base/oracle/FRA -name RMAN_FLAG -print|xargs -n1 dirname |sort -u... (3 Replies)
Discussion started by: jolan_louve
3 Replies

2. UNIX for Dummies Questions & Answers

grep using ASCII values

machine: HPUX file: a.dat contents: decimal 1 decimal 2 string 1 string 2 ASCII value of 'd': 100. to grep lines that have 'd', I use the following command grep d a.dat My requirement: I should grep for lines that contain 'd'. But I should use ASCII value of 'd' in the command... (1 Reply)
Discussion started by: sriksama
1 Replies

3. Shell Programming and Scripting

MEM=`ps v $PPID| grep -i db2 | grep -v grep| awk '{ if ( $7 ~ " " ) { print 0 } else

Hi Guys, I need to set the value of $7 to zero in case $7 is NULL. I've tried the below command but doesn't work. Any ideas. thanks guys. MEM=`ps v $PPID| grep -i db2 | grep -v grep| awk '{ if ( $7 ~ " " ) { print 0 } else { print $7}}' ` Harby. (4 Replies)
Discussion started by: hariza
4 Replies

4. Shell Programming and Scripting

grep two values together.

Hi... I have a file abc.txt , havin more then 10,000 lines, each field separated by '#'. I want to grep 9914699895 and 999 from abc.txt I am trying cat abc.txt | grep 9914699895 | grep 999 but i am also getting data like 9991111111 or 9991010101 I want to grep "999" exactly and... (1 Reply)
Discussion started by: tushar_tus
1 Replies

5. UNIX for Advanced & Expert Users

Moving 1000s of files to another folder

Hi, I need to move 1000s of files from one folder to another. Actually there are 100K+ files. Source dir : source1 Target dir : target1 Now if try cp or mv commands I am getting an error message : Argument List too long. I tried to do it by the time the files are created in the... (1 Reply)
Discussion started by: unx100
1 Replies

6. Shell Programming and Scripting

grep for certain files using a file as input to grep and then move

Hi All, I need to grep few files which has words like the below in the file name , which i want to put it in a file and and grep for the files which contain these names and move it to a new directory , full file name -C20091210.1000-20091210.1100_SMGBSC3:1000... (2 Replies)
Discussion started by: anita07
2 Replies

7. Shell Programming and Scripting

grep distinct values

this is a little more complex than that. I have a text file and I need to find all the distinct words that appear in a line after the word TABLESPACE when I grep for just the word tablespace, I get: how do i parse this a little better so i have a smaller file to read? This is just an... (4 Replies)
Discussion started by: guessingo
4 Replies

8. Cybersecurity

1000s of undelivered email messages

Hi, My boss has suddenly started receiving 1000s of messages in his inbox. They are undelivered messages that are bouncing back, though the emails weren't coming from him. I guess either these are fake undelivered messages and are just scam emails. Or they are real emails being sent with spoofed... (1 Reply)
Discussion started by: timgolding
1 Replies

9. UNIX Desktop Questions & Answers

How do you [e]grep for multiple values within multiple files?

Hi I'm sure there's a way to do this, but I ran out of caffeine/talent before getting the answer in a long winded alternate way (don't ask ;) ) The task I was trying to do was scan a directory of files and show only files that contained 3 values: I940 5433309 2181 I tried many variations... (4 Replies)
Discussion started by: callumw
4 Replies

10. Shell Programming and Scripting

Inconsistent `ps -eaf -o args | grep -i sfs_pcard_load_file.ksh | grep -v grep | wc -l`

i have this line of code that looks for the same file if it is currently running and returns the count. `ps -eaf -o args | grep -i sfs_pcard_load_file.ksh | grep -v grep | wc -l` basically it is assigned to a variable ISRUNNING=`ps -eaf -o args | grep -i sfs_pcard_load_file.ksh |... (6 Replies)
Discussion started by: wtolentino
6 Replies
fileutil(n)							  file utilities						       fileutil(n)

NAME
fileutil - Procedures implementing some file utilities SYNOPSIS
package require Tcl 8 package require fileutil ?1.4? ::fileutil::cat filename ::fileutil::fileType filename ::fileutil::find ?basedir ?filtercmd?? ::fileutil::findByPattern basedir ?-regexp|-glob? ?--? patterns ::fileutil::foreachLine var filename cmd ::fileutil::grep pattern ?files? ::fileutil::stripN path n ::fileutil::stripPwd path ::fileutil::touch ?-a? ?-c? ?-m? ?-r ref_file? ?-t time? filename ?...? DESCRIPTION
This package provides implementations of standard unix utilities. ::fileutil::cat filename A tcl implementation of the UNIX cat command. Returns the contents of the specified file. The first argument is the name of the file to read. ::fileutil::fileType filename An implementation of the UNIX file command, which uses various heuristics to guess the type of a file. Returns a list specifying as much type information as can be determined about the file, from most general (eg, "binary" or "text") to most specific (eg, "gif"). For example, the return value for a GIF file would be "binary graphic gif". The command will detect the following types of files: directory, empty, binary, text, script (with interpreter), executable elf, graphic gif, graphic jpeg, html, xml (with doctype if available), message pgp, and link. ::fileutil::find ?basedir ?filtercmd?? An implementation of the unix command find. Adapted from the Tcler's Wiki. Takes at most two arguments, the path to the directory to start searching from and a command to use to evaluate interest in each file. The path defaults to ".", i.e. the current directory. The command defaults to the empty string, which means that all files are of interest. The command takes care not to loose itself in infinite loops upon encountering circular link structures. The result of the command is a list containing the paths to the inter- esting files. ::fileutil::findByPattern basedir ?-regexp|-glob? ?--? patterns This command is based upon the TclX command recursive_glob, except that it doesn't allow recursion over more than one directory at a time. It uses ::fileutil::find internally and is thus able to and does follow symbolic links, something the TclX command does not do. First argument is the directory to start the search in, second argument is a list of patterns. The command returns a list of all files reachable through basedir whose names match at least one of the patterns. The options before the pattern-list determine the style of matching, either regexp or glob. glob-style matching is the default if no options are given. Usage of the option -- stops option processing. This allows the use of a leading '-' in the patterns. ::fileutil::foreachLine var filename cmd The command reads the file filename and executes the script cmd for every line in the file. During the execution of the script the variable var is set to the contents of the current line. The return value of this command is the result of the last invocation of the script cmd or the empty string if the file was empty. ::fileutil::grep pattern ?files? Implementation of grep. Adapted from the Tcler's Wiki. The first argument defines the pattern to search for. This is followed by a list of files to search through. The list is optional and stdin will be used if it is missing. The result of the procedures is a list containing the matches. Each match is a single element of the list and contains filename, number and contents of the matching line, separated by a colons. ::fileutil::stripN path n Removes the first n elements from the specified path and returns the modified path. If n is greater than the number of components in path an empty string is returned. ::fileutil::stripPwd path If the path is inside of the directory returned by [pwd] it is made relative to that directory. In other words, the current working directory is stripped from the path. The possibly modified path is returned as the result of the command. ::fileutil::touch ?-a? ?-c? ?-m? ?-r ref_file? ?-t time? filename ?...? Implementation of touch. Alter the atime and mtime of the specified files. If -c, do not create files if they do not already exist. If -r, use the atime and mtime from ref_file. If -t, use the integer clock value time. It is illegal to specify both -r and -t. If -a, only change the atime. If -m, only change the mtime. KEYWORDS
file utilities fileutil 1.4 fileutil(n)
All times are GMT -4. The time now is 06:00 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy