9 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Hey everyone,
I have a bunch of lines with values in field 4 that I am interested in.
If these values are between 1 and 3 I want it to count all these values to all be counted together and then have the computer print out
LOW and the number of lines with those values in between 1 and 3,... (2 Replies)
Discussion started by: VagabondGold
2 Replies
2. Shell Programming and Scripting
I am trying to figure out a way in nawk to 1) get a count of the number of times a value appears in field 1 and 2) count each time the same value appears in field 2 for each value of field 1. So for example, if I have a text file with the following:
grapes, purple
apples, green
squash, yellow... (2 Replies)
Discussion started by: he204035
2 Replies
3. Shell Programming and Scripting
Hey Unix gurus,
I would like to count the number occurrences of all the words (regardless of case) across multiple files, preferably outputting them in descending order of occurrence. This is well beyond my paltry shell scripting ability.
Researching, I can find many scripts/commands that... (4 Replies)
Discussion started by: twjolson
4 Replies
4. Shell Programming and Scripting
Hi,
I have a big file (~960MB) having epoch time values (~50 million entries) which looks like
897393601
897393601
897393601
897393601
897393602
897393602
897393602
897393602
897393602
897393603
897393603
897393603
897393603
and so on....each time stamp has more than one... (6 Replies)
Discussion started by: sajal.bhatia
6 Replies
5. Shell Programming and Scripting
Hi,
I have couple of .txt files (say 50 files) in a folder.
For each file:
I need to get the number of lines in each file and then that count -1 (I wanted to exclude the header.
Then sum the counts of all files and output the total sum.
Is there an efficient way to do this using shell... (7 Replies)
Discussion started by: Lucky Ali
7 Replies
6. Shell Programming and Scripting
I have 10 appservers and each appserver has 4 jvms . Each of these logs is archived and stored on a nfs directory . For example the files are
/logs/200907/ap1-jvm1.server.log.20090715.gz
/logs/200907/ap2-jvm2.server.log.20090714.gz
/logs/200908/ap1-jvm1.server.log.20090812.gz
I want to... (3 Replies)
Discussion started by: gubbu
3 Replies
7. Solaris
Hi,
I have a load of if statements that look for files in a directory, I want to be able to count them up and the total files confirmed in an email? I ahve tried expr but i this does not work and it only reads in the first if and ignores the rest.
Please see script,
#!/bin/ksh
... (2 Replies)
Discussion started by: Pablo_beezo
2 Replies
8. Shell Programming and Scripting
Hi all. If I have a unix directory with multiple files, lets say, I have some with .dat extensions, some with .txt extensions, etc etc.
How in a script would I provide a count of all the different file types (so, the different extensions, I guess) in the directory??
So if I had:
test.dat... (6 Replies)
Discussion started by: gerard1
6 Replies
9. UNIX for Dummies Questions & Answers
Which one line command can count and print to the screen the number of files containing "a" or "A" in their name (5 Replies)
Discussion started by: Edy
5 Replies
funidx(7) SAORD Documentation funidx(7)
NAME
Funidx - Using Indexes to Filter Rows in a Table
SYNOPSIS
This document contains a summary of the user interface for filtering rows in binary tables with indexes.
DESCRIPTION
Funtools Table Filtering allows rows in a table to be selected based on the values of one or more columns in the row. Because the actual
filter code is compiled on the fly, it is very efficient. However, for very large files (hundreds of Mb or larger), evaluating the filter
expression on each row can take a long time. Therefore, funtools supports index files for columns, which are used automatically during fil-
tering to reduce dramatically the number of row evaluations performed. The speed increase for indexed filtering can be an order of magni-
tude or more, depending on the size of the file.
The funindex program creates an index on one or more columns in a binary table. For example, to create an index for the column pi in the
file huge.fits, use:
funindex huge.fits pi
This will create an index named huge_pi.idx.
When a filter expression is initialized for row evaluation, funtools looks for an index file for each column in the filter expression. If
found, and if the file modification date of the index file is later than that of the data file, then the index will be used to reduce the
number of rows that are evaluated in the filter. When Spatial Region Filtering is part of the expression, the columns associated with the
region are checked for index files.
If an index file is not available for a given column, then in general, all rows must be checked when that column is part of a filter
expression. This is not true, however, when a non-indexed column is part of an AND expression. In this case, only the rows that pass the
other part of the AND expression need to be checked. Thus, in some cases, filtering speed can increase significantly even if all columns
are not indexed.
Also note that certain types of filter expression syntax cannot make use of indices. For example, calling functions with column names as
arguments implies that all rows must be checked against the function value. Once again, however, if this function is part of an AND expres-
sion, then a significant improvement in speed still is possible if the other part of the AND expression is indexed.
For example, note below the dramatic speedup in searching a 1 Gb file using an AND filter, even when one of the columns (pha) has no index:
time fundisp
huge.fits'[idx_activate=0,idx_debug=1,pha=2348&&cir 4000 4000 1]'
"x y pha"
x y pha
---------- ----------- ----------
3999.48 4000.47 2348
3999.48 4000.47 2348
3999.48 4000.47 2348
3999.48 4000.47 2348
3999.48 4000.47 2348
3999.48 4000.47 2348
3999.48 4000.47 2348
3999.48 4000.47 2348
3999.48 4000.47 2348
3999.48 4000.47 2348
3999.48 4000.47 2348
3999.48 4000.47 2348
3999.48 4000.47 2348
3999.48 4000.47 2348
3999.48 4000.47 2348
3999.48 4000.47 2348
42.36u 13.07s 6:42.89 13.7%
time fundisp
huge.fits'[idx_activate=1,idx_debug=1,pha=2348&&cir 4000 4000 1]'
"x y pha"
x y pha
---------- ----------- ----------
idxeq: [INDEF]
idxand sort: x[ROW 8037025:8070128] y[ROW 5757665:5792352]
idxand(1): INDEF [IDX_OR_SORT]
idxall(1): [IDX_OR_SORT]
3999.48 4000.47 2348
3999.48 4000.47 2348
3999.48 4000.47 2348
3999.48 4000.47 2348
3999.48 4000.47 2348
3999.48 4000.47 2348
3999.48 4000.47 2348
3999.48 4000.47 2348
3999.48 4000.47 2348
3999.48 4000.47 2348
3999.48 4000.47 2348
3999.48 4000.47 2348
3999.48 4000.47 2348
3999.48 4000.47 2348
3999.48 4000.47 2348
3999.48 4000.47 2348
1.55u 0.37s 1:19.80 2.4%
When all columns are indexed, the increase in speed can be even more dramatic:
time fundisp
huge.fits'[idx_activate=0,idx_debug=1,pi=770&&cir 4000 4000 1]'
"x y pi"
x y pi
---------- ----------- ----------
3999.48 4000.47 770
3999.48 4000.47 770
3999.48 4000.47 770
3999.48 4000.47 770
3999.48 4000.47 770
3999.48 4000.47 770
3999.48 4000.47 770
3999.48 4000.47 770
3999.48 4000.47 770
3999.48 4000.47 770
3999.48 4000.47 770
3999.48 4000.47 770
3999.48 4000.47 770
3999.48 4000.47 770
3999.48 4000.47 770
3999.48 4000.47 770
42.60u 12.63s 7:28.63 12.3%
time fundisp
huge.fits'[idx_activate=1,idx_debug=1,pi=770&&cir 4000 4000 1]'
"x y pi"
x y pi
---------- ----------- ----------
idxeq: pi start=9473025,stop=9492240 => pi[ROW 9473025:9492240]
idxand sort: x[ROW 8037025:8070128] y[ROW 5757665:5792352]
idxor sort/merge: pi[ROW 9473025:9492240] [IDX_OR_SORT]
idxmerge(5): [IDX_OR_SORT] pi[ROW]
idxall(1): [IDX_OR_SORT]
3999.48 4000.47 770
3999.48 4000.47 770
3999.48 4000.47 770
3999.48 4000.47 770
3999.48 4000.47 770
3999.48 4000.47 770
3999.48 4000.47 770
3999.48 4000.47 770
3999.48 4000.47 770
3999.48 4000.47 770
3999.48 4000.47 770
3999.48 4000.47 770
3999.48 4000.47 770
3999.48 4000.47 770
3999.48 4000.47 770
3999.48 4000.47 770
1.67u 0.30s 0:24.76 7.9%
The miracle of indexed filtering (and indeed, of any indexing) is the speed of the binary search on the index, which is of order log2(n)
instead of n. (The funtools binary search method is taken from http://www.tbray.org/ongoing/When/200x/2003/03/22/Binary, to whom grateful
acknowledgement is made.) This means that the larger the file, the better the performance. Conversely, it also means that for small files,
using an index (and the overhead involved) can slow filtering down somewhat. Our tests indicate that on a file containing a few tens of
thousands of rows, indexed filtering can be 10 to 20 percent slower than non-indexed filtering. Of course, your mileage will vary with con-
ditions (disk access speed, amount of available memory, process load, etc.)
Any problem encountered during index processing will result in indexing being turned off, and replaced by filtering all rows. You can turn
filtering off manually by setting the idx_activate variable to 0 (in a filter expression) or the FILTER_IDX_ACTIVATE environment variable
to 0 (in the global environment). Debugging output showing how the indexes are being processed can be displayed to stderr by setting the
idx_debug variable to 1 (in a filter expression) or the FILTER_IDX_DEBUG environment variable to 1 (in the global environment).
Currently, indexed filtering only works with FITS binary tables and raw event files. It does not work with text files. This restriction
might be removed in a future release.
SEE ALSO
See funtools(7) for a list of Funtools help pages
version 1.4.2 January 2, 2008 funidx(7)