06-11-2013
Searching 8000 keywords in 300 large files is quite something, but the program you show can be optimized for speed.
a) Don't open and reread the keyword file line by line for every file matching your pattern.
b) Don't run the grep process for every single keyword/file combination (300 x 8000 = 2.4 million times!)
c) Don't use wc -l piped to the greps (again 2.4 million times)
d) Don't run the sql command including login for every single keyword/file combination; collect the results into a file and insert & update afterwards.
9 More Discussions You Might Find Interesting
1. UNIX for Dummies Questions & Answers
Hi,
how does the Unix File System perform with large directories (containing ~30.000 files)?
What kind of structure is used for the organization of a directory's content, linear lists, (binary) trees?
I hope the description 'Unix File System' is exact enough, I don't know more about the file... (3 Replies)
Discussion started by: dive
3 Replies
2. Shell Programming and Scripting
I found another problem with my disk-adding script today. When looking for disks, I use grep.
When I grep for the following disk sizes:
5242880
I also pick up these as well:
524288000
How do I specifically pick out one or the other, using grep, without resorting to the -v option?
... (9 Replies)
Discussion started by: LinuxRacr
9 Replies
3. Shell Programming and Scripting
Hello Gurus,
We are facing some performance issue in UNIX. If someone had faced such kind of issue in past please provide your suggestions on this .
Problem Definition:
/Few of load processes of our Finance Application are facing issue in UNIX when they uses a shell script having below... (19 Replies)
Discussion started by: KRAMA
19 Replies
4. Shell Programming and Scripting
I have the following problem:
I have two files: S containing sentences (one in each row) and W containing files (one in each row). It might look like this:
S:
a b c apple d.
e f orange g.
h banana i j.
W:
orange
banana
apple
My task is to replace in S all words that appear in W... (2 Replies)
Discussion started by: tootles564
2 Replies
5. Shell Programming and Scripting
Background
-------------
The Unix flavor can be any amongst Solaris, AIX, HP-UX and Linux. I have below 2 flat files.
File-1
------
Contains 50,000 rows with 2 fields in each row, separated by pipe.
Row structure is like Object_Id|Object_Name, as following:
111|XXX
222|YYY
333|ZZZ
... (6 Replies)
Discussion started by: Souvik
6 Replies
6. Red Hat
Hi,
I've some directory that I used as working directory for a program. At the end of the procedure, the content is deleted. This directory, when I do a ls -l, appears to still take up some space. After a little research, I've seen on a another board of this forum that it's not really taking... (5 Replies)
Discussion started by: bdx
5 Replies
7. Shell Programming and Scripting
Hi All,
I need help to know the exact command when I grep large list of files. Either using ls or find command. However I do not want to find in the subdirectories as the number of subdirectories are not fixed. How do I achieve that.
I want something like this:
find ./ -name "MYFILE*.txt"... (2 Replies)
Discussion started by: angshuman
2 Replies
8. Shell Programming and Scripting
I want to extract verbal forms from a large corpus of English. I have identified a certain number of patterns. Each pattern has the following structure
SPACE word_CATEGORY
where word refers to the verbal form and CATEGORY refers to the class of the verb
The categories are identified as per the... (4 Replies)
Discussion started by: gimley
4 Replies
9. Shell Programming and Scripting
Hello,
For several of our scripts we are using awk to search patterns in files with data from other files. This works almost perfectly except that it takes ages to run on larger files. I am wondering if there is a way to speed up this process or have something else that is quicker with the... (15 Replies)
Discussion started by: SDohmen
15 Replies
LEARN ABOUT LINUX
pam_lastlog
PAM_LASTLOG(8) Linux-PAM Manual PAM_LASTLOG(8)
NAME
pam_lastlog - PAM module to display date of last login
SYNOPSIS
pam_lastlog.so [debug] [silent] [never] [nodate] [nohost] [noterm] [nowtmp] [noupdate] [showfailed]
DESCRIPTION
pam_lastlog is a PAM module to display a line of information about the last login of the user. In addition, the module maintains the
/var/log/lastlog file.
Some applications may perform this function themselves. In such cases, this module is not necessary.
OPTIONS
debug
Print debug information.
silent
Don't inform the user about any previous login, just update the /var/log/lastlog file.
never
If the /var/log/lastlog file does not contain any old entries for the user, indicate that the user has never previously logged in with
a welcome message.
nodate
Don't display the date of the last login.
noterm
Don't display the terminal name on which the last login was attempted.
nohost
Don't indicate from which host the last login was attempted.
nowtmp
Don't update the wtmp entry.
noupdate
Don't update any file.
showfailed
Display number of failed login attempts and the date of the last failed attempt from btmp. The date is not displayed when nodate is
specified.
MODULE TYPES PROVIDED
Only the session module type is provided.
RETURN VALUES
PAM_SUCCESS
Everything was successful.
PAM_SERVICE_ERR
Internal service module error.
PAM_USER_UNKNOWN
User not known.
EXAMPLES
Add the following line to /etc/pam.d/login to display the last login time of an user:
session required pam_lastlog.so nowtmp
FILES
/var/log/lastlog
Lastlog logging file
SEE ALSO
pam.conf(5), pam.d(5), pam(7)
AUTHOR
pam_lastlog was written by Andrew G. Morgan <morgan@kernel.org>.
Linux-PAM Manual 06/04/2011 PAM_LASTLOG(8)