12-02-2011
search a number in very very huge amount of data
Hi,
I have to search a number in a very long listing of files.the total size of the files in which I have to search is 10 Tera Bytes.
How to search a number in such a huge amount of data effectively.I used fgrep but it is taking many hours to search. Is there any other feasible solution to search in such huge data. Any way to serach effectively?
10 More Discussions You Might Find Interesting
1. UNIX for Dummies Questions & Answers
folks,
In my working directory, there a multiple large files which only contain one line in the file. The line is too long to use "grep", so any help?
For example, if I want to find if these files contain a string like "93849", what command I should use?
Also, there is oder_id number... (1 Reply)
Discussion started by: ting123
1 Replies
2. Programming
Hi,
I'm trying to figure out the best solution to the following problem, and I'm not
yet that much experienced like you. :-)
Basically I have to read a fairly large file, composed of "messages" , in order
to display all of them through an user interface (made with QT).
The messages that... (3 Replies)
Discussion started by: emitrax
3 Replies
3. UNIX for Advanced & Expert Users
I have the following situation:
a text file with 50000 string patterns:
abc2344536
gvk6575556
klo6575556
....
and 3 text files each with more than 1 million lines:
...
000000 abc2344536 46575 0000
000000 abc2344536 46575 4444
000000 abc2344555 46575 1234
...
I... (8 Replies)
Discussion started by: andy2000
8 Replies
4. Shell Programming and Scripting
I met a problem on HPUX with 64G RAM and 20 CPU.
There are 5 million files with file name from file0000001.dat to file9999999.dat, in the same directory, and with some other files with random names.
I was trying to remove all the files from file0000001.dat to file9999999.dat at the same time.... (9 Replies)
Discussion started by: lisp21
9 Replies
5. Shell Programming and Scripting
How to search a string which has occured numerous times in a single row. I tried many options, I am facing issue with the file size. Anything I go for, it says it is huge.. File is 82MB.
Assume, the file contains the string 'Name' in many places.. Something Like below.
... (5 Replies)
Discussion started by: Muthuraj K
5 Replies
6. AIX
Hi
When i copy 300GB of data from one filesystem to the other filesystem in AIX I get the error :
tar: 0511-825 The file 'SAPBRD.dat' is too large.
The command I used is :
# tar -cf - . | (cd /sapbackup ; tar -xf - )
im copying as root
The below is my ulimit -a output :
... (3 Replies)
Discussion started by: samsungsamsung
3 Replies
7. Shell Programming and Scripting
Hello Friends,
I have the below scenario in my current project. Suggest me which tool ( perl,python etc) is best to this scenario. Or should I go for Programming language ( C/Java )..
(1) I will be having a very big file ( information about 200million subscribers will be stored in it ). This... (5 Replies)
Discussion started by: panyam
5 Replies
8. Shell Programming and Scripting
This basic code works.
I have a very long list, almost 10000 lines that I am building into the array. Each line has either 2 or 3 fields as shown in the code snippit. The array elements are static (for a few reasons that out of scope of this question) the list has to be "built in".
It... (5 Replies)
Discussion started by: sumguy
5 Replies
9. Shell Programming and Scripting
Hi Friends,
I have a file with sample amount data as follows:
-89990.3456
8788798.990000128
55109787.20
-12455558989.90876
I need to exclude the '-' symbol in order to treat all values as an absolute one and then I need to sum up.The record count is around 1 million.
How... (8 Replies)
Discussion started by: Ravichander
8 Replies
10. Shell Programming and Scripting
I have nginx web server logs with all requests that were made and I'm filtering them by date and time.
Each line has the following structure:
127.0.0.1 - xyz.com GET 123.ts HTTP/1.1 (200) 0.000 s 3182 CoreMedia/1.0.0.15F79 (iPhone; U; CPU OS 11_4 like Mac OS X; pt_br)
These text files are... (21 Replies)
Discussion started by: brenoasrm
21 Replies
LEARN ABOUT SUSE
ausearch_add_expression
AUSEARCH_ADD_expression(3) Linux Audit API AUSEARCH_ADD_expression(3)
NAME
ausearch_add_expression - build up search expression
SYNOPSIS
#include <auparse.h>
int ausearch_add_expression(auparse_state_t *au, const char *expression, char **error, ausearch_rule_t how);
DESCRIPTION
ausearch_add_item adds an expression to the current audit search expression. The search conditions can then be used to scan logs, files,
or buffers for something of interest. The expression parameter contains an expression, as specified in ausearch-expression(5).
The how parameter determines how this search expression will affect the existing search expression, if one is already defined. The possi-
ble values are:
AUSEARCH_RULE_CLEAR
Clear the current search expression, if any, and use only this search expression.
AUSEARCH_RULE_OR
If a search expression E is already configured, replace it by (E || this_search_expression).
AUSEARCH_RULE_AND
If a search expression E is already configured, replace it by (E && this_search_expression).
RETURN VALUE
If successful, ausearch_add_expression returns 0. Otherwise, it returns -1, sets errno and it may set *error to an error message; the
caller must free the error message using free(3). If an error message is not available or can not be allocated, *error is set to NULL.
SEE ALSO
ausearch_add_item(3), ausearch_add_interpreted_item(3), ausearch_add_timestamp_item(3), ausearch_add_regex(3), ausearch_set_stop(3), ause-
arch_clear(3), ausearch_next_event(3), ausearch-expression(5).
AUTHOR
Miloslav Trmac
Red Hat Feb 2008 AUSEARCH_ADD_expression(3)