Sponsored Content
Top Forums Shell Programming and Scripting Finding 50k Keywords in 3k files Post 302521153 by rjains on Tuesday 10th of May 2011 11:06:14 AM
Old 05-10-2011
What Operating System and version do you have? -- Linux .....2.6.9-42.ELsmp #1 SMP Wed Jul 12 23:27:17 EDT 2006 i686 i686 i386 GNU/Linux

What Shell do you prefer? /bin/ksh

How big is the keywords file? 50k keywords (each keyword upto 30-40 char)
How big is the total of the 3k data files? Each file will have about 300-400 lines
Are these all normal unix text files with a reasonable record size? All text files

You appear to be attempting 150,000,000 serial file passes (15,000 x 3,000) -- Yes :-(

Is this a one-off or something which will be run again and again? -- Not one off. This will done regularly

Do you have a full-works database engine such as Oracle? -- if found to be more efficient we can get a database engine such as Oracle
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

renaming 50k files, whats the best way?

Because I am not creative, I did this: find . -type f -name '*.GIF'|cut -d'/' -f2|awk -F. '{print "mv "$1".GIF "$1".gif --reply=yes"}' > case.sh Then ran the case.sh - I was wondering if you guys could come up with something more efficient? Or even limit CPU useage? It is killing my poor ext3... (3 Replies)
Discussion started by: r0sc0
3 Replies

2. Shell Programming and Scripting

finding duplicate files by size and finding pattern matching and its count

Hi, I have a challenging task,in which i have to find the duplicate files by its name and size,then i need to take anyone of the file.Then i need to open the file and find for more than one pattern and count of that pattern. Note:These are the samples of two files,but i can have more... (2 Replies)
Discussion started by: jerome Sukumar
2 Replies

3. Shell Programming and Scripting

Finding files

How can we find "latest files which have been recently updated/changed/created" in solaris 10??? (3 Replies)
Discussion started by: asadlone
3 Replies

4. UNIX for Dummies Questions & Answers

finding keywords in many files using grep

Hi to all Sorry for the confusion because I did not explain the task clearly. There are many .hhr files in a folder There are so many lines in these .hhr files but I want only the following 2 lines to be transferred to the output file. The keyword No 1 and all the words in the next line They... (5 Replies)
Discussion started by: raghulrajan
5 Replies

5. Shell Programming and Scripting

Finding files with wc -l results = 1 then moving the files to another folder

Hi guys can you please help me with a script to find files with one row/1 line of content then move the file to another directory my script below runs but nothing happens to the files....Alternatively Ca I get a script to find the *.csv files with "wc -1" results = 1 then create a list of those... (5 Replies)
Discussion started by: Dj Moi
5 Replies

6. UNIX for Advanced & Expert Users

Need to search for keywords within files modified at a certain time

I have a huge list of files in an Unix directory (around 10000 files). I need to be able to search for a certain keyword only within files that are modified between certain date and time, say for e.g 2012-08-20 12:30 to 2012-08-20 12:40 Can someone let me know what would be the fastest way... (10 Replies)
Discussion started by: virtual123
10 Replies

7. Shell Programming and Scripting

Search files in directory for keywords using bash

I have ~100 text files in a directory that I am trying to parse and output to a new file. I am looking for the words chr,start,stop,ref,alt in each of the files. Those fields should appear somewhere in those files. The first two fields of each new set of rows is also printed. Since this is on a... (7 Replies)
Discussion started by: cmccabe
7 Replies

8. UNIX for Dummies Questions & Answers

Find keywords in multiple log files

The Problem that I am having is when the code ran and populated the progflag.csv file, columns MEMSIZE, SECOND and SASEXE were blank. The next problems are the IF else statement isn't working and the email function isn't sending the progflag.csv attachment. a. What I want the program to do is to... (2 Replies)
Discussion started by: dellanicholson
2 Replies

9. Shell Programming and Scripting

Find keywords in multiple log files

I have several problems with my program: I hope you can help me. 1) the If else statement isn't working . The IF Else syntax is: If MEMSIZE OR sasfoundation (SASEXE) OR Real Time(second) >1.0 and Filename, output column name and value to csv or else nothing Example progflag,cvs:... (13 Replies)
Discussion started by: dellanicholson
13 Replies

10. UNIX for Beginners Questions & Answers

Compare 2 files with different keywords : use server health-check tool

I have two files to be compared to get the output of the differences. File1 has a lot more lists than File2. After searching a lot on this thread I'am unable to find the exact code that im willing to get. This will be used as 'pre-check'/post-check utility (health check Tool) to compare... (1 Reply)
Discussion started by: GeekyJimmy
1 Replies
PYGETTEXT(1)						      General Commands Manual						      PYGETTEXT(1)

NAME
pygettext - Python equivalent of xgettext(1) SYNOPSIS
pygettext [OPTIONS] INPUTFILE ... DESCRIPTION
pygettext is deprecated. The current version of xgettext supports many languages, including Python. pygettext uses Python's standard tokenize module to scan Python source code, generating .pot files identical to what GNU xgettext generates for C and C++ code. From there, the standard GNU tools can be used. pygettext searches only for _() by default, even though GNU xgettext recognizes the following keywords: gettext, dgettext, dcgettext, and gettext_noop. See the -k/--keyword flag below for how to augment this. OPTIONS
-a, --extract-all Extract all strings. -d, --default-domain=NAME Rename the default output file from messages.pot to name.pot. -E, --escape Replace non-ASCII characters with octal escape sequences. -D, --docstrings Extract module, class, method, and function docstrings. These do not need to be wrapped in _() markers, and in fact cannot be for Python to consider them docstrings. (See also the -X option). -h, --help Print this help message and exit. -k, --keyword=WORD Keywords to look for in addition to the default set, which are: _ You can have multiple -k flags on the command line. -K, --no-default-keywords Disable the default set of keywords (see above). Any keywords explicitly added with the -k/--keyword option are still recognized. --no-location Do not write filename/lineno location comments. -n, --add-location Write filename/lineno location comments indicating where each extracted string is found in the source. These lines appear before each msgid. The style of comments is controlled by the -S/--style option. This is the default. -o, --output=FILENAME Rename the default output file from messages.pot to FILENAME. If FILENAME is `-' then the output is sent to standard out. -p, --output-dir=DIR Output files will be placed in directory DIR. -S, --style=STYLENAME Specify which style to use for location comments. Two styles are supported: o Solaris # File: filename, line: line-number o GNU #: filename:line The style name is case insensitive. GNU style is the default. -v, --verbose Print the names of the files being processed. -V, --version Print the version of pygettext and exit. -w, --width=COLUMNS Set width of output to columns. -x, --exclude-file=FILENAME Specify a file that contains a list of strings that are not be extracted from the input files. Each string to be excluded must appear on a line by itself in the file. -X, --no-docstrings=FILENAME Specify a file that contains a list of files (one per line) that should not have their docstrings extracted. This is only useful in conjunction with the -D option above. If `INPUTFILE' is -, standard input is read. BUGS
pygettext attempts to be option and feature compatible with GNU xgettext where ever possible. However some options are still missing or are not fully implemented. Also, xgettext's use of command line switches with option arguments is broken, and in these cases, pygettext just defines additional switches. AUTHOR
pygettext is written by Barry Warsaw <barry@zope.com>. Joonas Paalasmaa <joonas.paalasmaa@iki.fi> put this manual page together based on "pygettext --help". pygettext 1.4 PYGETTEXT(1)
All times are GMT -4. The time now is 08:42 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy