Sponsored Content
Top Forums Shell Programming and Scripting Search for patterns in thousands of files Post 302785683 by danish0909 on Tuesday 26th of March 2013 07:18:17 AM
Old 03-26-2013
I am on hp ux

Code:
/comptel/elink> find /comptel4/elink/backup1/output/vas/NG0/ -name "*.GPX.Z" -exec zgrep -il $i {} \;
/comptel/elink>

Code:
/comptel4/elink/backup1/output/vas/NG0> find /comptel4/elink/backup1/output/vas/NG0/ -name "*.GPX.Z" -print
/comptel4/elink/backup1/output/vas/NG0>

 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Finding a specific pattern from thousands of files ????

Hi All, I want to find a specific pattern from approximately 400000 files on solaris platform. Its very heavy for me to grep that pattern to each file individually. Can anybody suggest me some way to search for specific pattern (alpha numeric) from these forty thousand files. Please note that... (6 Replies)
Discussion started by: aarora_98
6 Replies

2. UNIX Desktop Questions & Answers

how to search files efficiently using patterns

hi friens, :) if i need to find files with extension .c++,.C++,.cpp,.Cpp,.CPp,.cPP,.CpP,.cpP,.c,.C wat is the pattern for finding them :confused: (2 Replies)
Discussion started by: arunsubbhian
2 Replies

3. UNIX for Advanced & Expert Users

Copying Thousands of Tiny or Empty Files?

There is a procedure I do here at work where I have to synchronize file systems. The source file system always has three or four directories of hundreds of thousands of tiny (1k or smaller) or empty files. Whenever my rsync command reaches these directories, I'm waiting for hours for those files... (3 Replies)
Discussion started by: deckard
3 Replies

4. UNIX for Advanced & Expert Users

Best way to search for patterns in huge text files

I have the following situation: a text file with 50000 string patterns: abc2344536 gvk6575556 klo6575556 .... and 3 text files each with more than 1 million lines: ... 000000 abc2344536 46575 0000 000000 abc2344536 46575 4444 000000 abc2344555 46575 1234 ... I... (8 Replies)
Discussion started by: andy2000
8 Replies

5. Shell Programming and Scripting

help to parallelize work on thousands of files

I need to find a smarter way to process about 60,000 files in a single directory. Every night a script runs on each file generating a output on another directory; this used to take 5 hours, but as the data grows it is taking 7 hours. The files are of different sizes, but there are 16 cores... (10 Replies)
Discussion started by: vhope07
10 Replies

6. UNIX for Dummies Questions & Answers

script to search patterns inside list of files

>testfile while read x do if then echo $x >> testfile else fi if then echo $x >> testfile else fi done < list_of_files is there any efficient way to search abc.dml and xyz.dml ? (2 Replies)
Discussion started by: dr46014
2 Replies

7. Shell Programming and Scripting

to read two files, search for patterns and store the output in third file

hello i have two files temp.txt and temp_unique.text the second file consists the unique fields from the temp.txt file the strings stored are in the following form 4,4 17,12 15,65 4,4 14,41 15,65 65,89 1254,1298i'm able to run the following script to get the total count of a... (3 Replies)
Discussion started by: vaibhavkorde
3 Replies

8. SuSE

Search all files based on first and in all listed files search the second patterns

Hello Linux Masters, I am not a linux expert therefore i need help from linux gurus. Well i have a requirement where i need to search all files based on first patterns and after seraching all files then serach second pattern in all files which i have extracted based on first pattern.... (1 Reply)
Discussion started by: Black-Linux
1 Replies

9. Shell Programming and Scripting

Bash-awk to process thousands of files

Hi to all, I have thousand of files in a folder with names with format "FILE-YYYY-MM-DD-HHMM" for what I want to send the following AWK command awk '/Code.*/' FILE-2014* I'd like to separate all files that have the same date to a folder named with the corresponding date. For example, if I... (7 Replies)
Discussion started by: Ophiuchus
7 Replies

10. Shell Programming and Scripting

Bash - Find files excluding file patterns and subfolder patterns

Hello. For a given folder, I want to select any files find $PATH1 -f \( -name "*" but omit any files like pattern name ! -iname "*.jpg" ! -iname "*.xsession*" ..... \) and also omit any subfolder like pattern name -type d \( -name "/etc/gconf/gconf.*" -o -name "*cache*" -o -name "*Cache*" -o... (2 Replies)
Discussion started by: jcdole
2 Replies
Bio::Tools::EUtilities(3pm)				User Contributed Perl Documentation			       Bio::Tools::EUtilities(3pm)

NAME
Bio::Tools::EUtilities - NCBI eutil XML parsers SYNOPSIS
# from file or fh my $parser = Bio::Tools::EUtilities->new( -eutil => 'einfo', -file => 'output.xml' ); # or HTTP::Response object... my $parser = Bio::Tools::EUtilities->new( -eutil => 'esearch', -response => $response ); # esearch, esummary, elink @ids = $parser->get_ids(); # returns array or array ref of IDs # egquery, espell $term = $parser->get_term(); # returns array or array ref of IDs # elink, einfo $db = $parser->get_database(); # returns database # Query-related methods (esearch, egquery, espell data) # eutil data centered on use of search terms my $ct = $parser->get_count; # uses optional database for egquery count my $translation = $parser->get_count; my $corrected = $parser->get_corrected_query; # espell while (my $gquery = $parser->next_GlobalQuery) { # iterates through egquery data } # Info-related methods (einfo data) # database-related information my $desc = $parser->get_description; my $update = $parser->get_last_update; my $nm = $parser->get_menu_name; my $ct = $parser->get_record_count; while (my $field = $parser->next_FieldInfo) { # ... } while (my $field = $parser->next_LinkInfo) { # ... } # History methods (epost data, some data returned from elink) # data which enables one to retrieve and query against user-stored # information on the NCBI server while (my $cookie = $parser->next_History) { # ... } my @hists = $parser->get_Histories; # Bio::Tools::EUtilities::Summary (esummary data) # information on a specific database record # retrieve nested docsum data while (my $docsum = $parser->next_DocSum) { print "ID:",$docsum->get_ids," "; while (my $item = $docsum->next_Item) { # do stuff here... while (my $listitem = $docsum->next_ListItem) { # do stuff here... while (my $listitem = $docsum->next_Structure) { # do stuff here... } } } } # retrieve flattened item list per DocSum while (my $docsum = $parser->next_DocSum) { my @items = $docsum->get_all_DocSum_Items; } DESCRIPTION
Parses NCBI eutils XML output for retrieving IDs and other information. Part of the BioPerl EUtilities system. This is a general parser for eutils XML; data from efetch is NOT parsed (this requires separate format-dependent parsers). All other XML for eutils is parsed. These modules can be used independently of Bio::DB::EUtilities and Bio::Tools::EUtilities::EUtilParameters; if used in this way, only data present in the XML will be parsed out (other bits are retrieved from a passed-in Bio::Tools::EUtilities::EUtilParameters instance used while querying the database) TODO
This module is largely complete. However there are a few holes which will eventually be filled in. TranslationSets from esearch are not currently parsed, for instance. FEEDBACK
Mailing Lists User feedback is an integral part of the evolution of this and other Bioperl modules. Send your comments and suggestions preferably to one of the Bioperl mailing lists. Your participation is much appreciated. bioperl-l@lists.open-bio.org - General discussion http://www.bioperl.org/wiki/Mailing_lists - About the mailing lists Support Please direct usage questions or support issues to the mailing list: bioperl-l@bioperl.org rather than to the module maintainer directly. Many experienced and reponsive experts will be able look at the problem and quickly address it. Please include a thorough description of the problem with code and data examples if at all possible. Reporting Bugs Report bugs to the Bioperl bug tracking system to help us keep track the bugs and their resolution. Bug reports can be submitted via the web. https://redmine.open-bio.org/projects/bioperl/ AUTHOR
Email cjfields at bioperl dot org APPENDIX
The rest of the documentation details each of the object methods. Internal methods are usually preceded with a _ Constructor methods new Title : new Usage : my $parser = Bio::Tools::EUtilities->new(-file => 'my.xml', -eutil => 'esearch'); Function : create Bio::Tools::EUtilities instance Returns : new Bio::Tools::EUtilities instance Args : -file/-fh - File or filehandle -eutil - eutil parser to use (supports all but efetch) -response - HTTP::Response object (optional) Bio::Tools::EUtilities methods cache_response Title : cache_response Usage : $parser->cache_response(1) Function : sets flag to cache response object (off by default) Returns : value eval'ing to TRUE or FALSE Args : value eval'ing to TRUE or FALSE Note : must be set prior to any parsing run response Title : response Usage : my $response = $parser->response; Function : Get/Set HTTP::Response object Returns : HTTP::Response Args : HTTP::Response Note : to prevent object from destruction set cache_response() to TRUE parameter_base Title : parameter_base Usage : my $response = $parser->parameter_base; Function : Get/Set Bio::ParameterBaseI object (should be Bio::Tools::EUtilities::EUtilParameters) Returns : Bio::Tools::EUtilities::EUtilParameters || undef Args : (optional) Bio::Tools::EUtilities::EUtilParameters Note : If this object is present, it may be used as a last resort for some data values if parsed XML does not contain said values (for instance, database, term, IDs, etc). data_parsed Title : data_parsed Usage : if ($parser->data_parsed) {...} Function : returns TRUE if data has been parsed Returns : value eval'ing to TRUE or FALSE Args : none (set within parser) Note : mainly internal method (set in case user wants to check whether parser is exhausted). is_lazy Title : is_lazy Usage : if ($parser->is_lazy) {...} Function : returns TRUE if parser is set to lazy parsing mode (only affects elink/esummary) Returns : Boolean Args : none Note : Permanently set in constructor. Still highly experimental. Don't stare directly at happy fun ball... parse_data Title : parse_data Usage : $parser->parse_data Function : direct call to parse data; normally implicitly called Returns : none Args : none to_string Title : to_string Usage : $foo->to_string() Function : converts current object to string Returns : none Args : (optional) simple data for text formatting Note : Implemented in plugins print_all Title : print_all Usage : $info->print_all(); $info->print_all(-fh => $fh, -cb => $coderef); Function : prints (dumps) all data in parser. Unless a coderef is supplied, this just dumps the parser-specific to_string method to either a file/fh or STDOUT Returns : none Args : [optional] -file : file to print to -fh : filehandle to print to (cannot be used concurrently with file) -cb : coderef to use in place of default print method. This is passed in the parser object -wrap : number of columns to wrap default text output to (def = 80) Notes : only applicable for einfo. If -file or -fh are not defined, prints to STDOUT Bio::Tools::EUtilities::EUtilDataI methods eutil Title : eutil Usage : $eutil->$foo->eutil Function : Get/Set eutil Returns : string Args : string (eutil) Throws : on invalid eutil datatype Title : datatype Usage : $type = $foo->datatype; Function : Get/Set data object type Returns : string Args : string Methods useful for multiple eutils get_ids Title : get_ids Usage : my @ids = $parser->get_ids Function : returns array of requested IDs (see Notes for more specifics) Returns : array Args : [conditional] not required except when running elink queries against multiple databases. In case of the latter, the database name is optional but recommended when retrieving IDs as the ID list will be globbed together. In such cases, if a db name isn't provided a warning is issued as a reminder. Notes : esearch : returned ID list elink : returned ID list (see Args above for caveats) all others : from parameter_base->id or undef get_database Title : get_database Usage : my $db = $info->get_database; Function : returns single database name (eutil-compatible). This is the queried database. For most eutils this is straightforward. For elinks (which have 'db' and 'dbfrom') this is db/dbto, for egquery, it is the first db in the list (you probably want get_databases instead) Returns : string Args : none Notes : egquery : first db in the query (you probably want get_databases) einfo : the queried database espell : the queried database all others : from parameter_base->db or undef get_db (alias for get_database) get_databases Title : get_databases Usage : my @dbs = $parser->get_databases Function : returns list of databases Returns : array of strings Args : none Notes : This is guaranteed to return a list of databases. For a single database use the convenience method get_db/get_database egquery : list of all databases in the query einfo : the queried database, or the available databases espell : the queried database elink : collected from each LinkSet all others : from parameter_base->db or undef get_dbs (alias for get_databases) next_History Title : next_History Usage : while (my $hist=$parser->next_History) {...} Function : returns next HistoryI (if present). Returns : Bio::Tools::EUtilities::HistoryI (Cookie or LinkSet) Args : none Note : esearch, epost, and elink are all capable of returning data which indicates search results (in the form of UIDs) is stored on the remote server. Access to this data is wrapped up in simple interface (HistoryI), which is implemented in two classes: Bio::DB::EUtilities::History (the simplest) and Bio::DB::EUtilities::LinkSet. In general, calls to epost and esearch will only return a single HistoryI object (formerly known as a Cookie), but calls to elink can generate many depending on the number of IDs, the correspondence, etc. Hence this iterator, which allows one to retrieve said data one piece at a time. next_cookie (alias for next_History) get_Histories Title : get_Histories Usage : my @hists = $parser->get_Histories Function : returns list of HistoryI objects. Returns : list of Bio::Tools::EUtilities::HistoryI (History or LinkSet) Args : none Query-related methods get_count Title : get_count Usage : my $ct = $parser->get_count Function : returns the count (hits for a search) Returns : integer Args : [CONDITIONAL] string with database name - used to retrieve count from specific database when using egquery Notes : egquery : count for specified database (specified above) esearch : count for last search all others : undef get_term Title : get_term Usage : $st = $qd->get_term; Function : retrieve the term for the global search Returns : string Args : none Notes : egquery : search term espell : search term esearch : from parameter_base->term or undef all others : undef get_translation_from Title : get_translation_from Usage : $string = $qd->get_translation_from(); Function: portion of the original query replaced with translated_to() Returns : string Args : none Note : only applicable for esearch get_translation_to Title : get_translation_to Usage : $string = $qd->get_translation_to(); Function: replaced string used in place of the original query term in translation_from() Returns : string Args : none Note : only applicable for esearch get_retstart Title : get_retstart Usage : $start = $qd->get_retstart(); Function : retstart setting for the query (either set or NCBI default) Returns : Integer Args : none Notes : esearch : retstart esummary : retstart all others : from parameter_base->retstart or undef get_retmax Title : get_retmax Usage : $max = $qd->get_retmax(); Function : retmax setting for the query (either set or NCBI default) Returns : Integer Args : none Notes : esearch : retmax esummary : retmax all others : from parameter_base->retmax or undef get_query_translation Title : get_query_translation Usage : $string = $qd->get_query_translation(); Function: returns the translated query used for the search (if any) Returns : string Args : none Notes : only applicable for esearch. This is the actual term used for esearch. get_corrected_query Title : get_corrected_query Usage : my $cor = $eutil->get_corrected_query; Function : retrieves the corrected query when using espell Returns : string Args : none Notes : only applicable for espell. get_replaced_terms Title : get_replaced_terms Usage : my $term = $eutil->get_replaced_terms Function : returns array of strings replaced in the query Returns : string Args : none Notes : only applicable for espell next_GlobalQuery Title : next_GlobalQuery Usage : while (my $query = $eutil->next_GlobalQuery) {...} Function : iterates through the queries returned from an egquery search Returns : GlobalQuery object Args : none Notes : only applicable for egquery get_GlobalQueries Title : get_GlobalQueries Usage : @queries = $eutil->get_GlobalQueries Function : returns list of GlobalQuery objects Returns : array of GlobalQuery objects Args : none Notes : only applicable for egquery print_GlobalQueries Title : print_GlobalQueries Usage : $docsum->print_GlobalQueries(); $docsum->print_GlobalQueries(-fh => $fh, -callback => $coderef); Function : prints item data for all global queries. The default printing method is each item per DocSum is printed with relevant values if present in a simple table using Text::Wrap. Returns : none Args : [optional] -file : file to print to -fh : filehandle to print to (cannot be used concurrently with file) -cb : coderef to use in place of default print method. This is passed in a GlobalQuery object; -wrap : number of columns to wrap default text output to (def = 80) Notes : only applicable for esummary. If -file or -fh are not defined, prints to STDOUT Summary-related methods next_DocSum Title : next_DocSum Usage : while (my $ds = $esum->next_DocSum) {...} Function : iterate through DocSum instances Returns : single Bio::Tools::EUtilities::Summary::DocSum Args : none yet Notes : only applicable for esummary get_DocSums Title : get_DocSums Usage : my @docsums = $esum->get_DocSums Function : retrieve a list of DocSum instances Returns : array of Bio::Tools::EUtilities::Summary::DocSum Args : none Notes : only applicable for esummary print_DocSums Title : print_DocSums Usage : $docsum->print_DocSums(); $docsum->print_DocSums(-fh => $fh, -cb => $coderef); Function : prints item data for all docsums. The default data is generated via DocSum::to_string Returns : none Args : [optional] -file : file to print to -fh : filehandle to print to (cannot be used concurrently with file) -cb : coderef to use in place of default print method. This is passed in a DocSum object -wrap : number of columns to wrap default text output to (def = 80) Notes : only applicable for esummary. If -file or -fh are not defined, prints to STDOUT Info-related methods get_available_databases Title : get_available_databases Usage : my @dbs = $info->get_available_databases Function : returns list of available eutil-compatible database names Returns : Array of strings Args : none Notes : only applicable for einfo. get_record_count Title : get_record_count Usage : my $ct = $eutil->get_record_count; Function : returns database record count Returns : integer Args : none Notes : only applicable for einfo. get_last_update Title : get_last_update Usage : my $time = $info->get_last_update; Function : returns string containing time/date stamp for last database update Returns : integer Args : none Notes : only applicable for einfo. get_menu_name Title : get_menu_name Usage : my $nm = $info->get_menu_name; Function : returns string of database menu name Returns : string Args : none Notes : only applicable for einfo. get_description Title : get_description Usage : my $desc = $info->get_description; Function : returns database description Returns : string Args : none Notes : only applicable for einfo. next_FieldInfo Title : next_FieldInfo Usage : while (my $field = $info->next_FieldInfo) {...} Function : iterate through FieldInfo objects Returns : Field object Args : none Notes : only applicable for einfo. Uses callback() for filtering if defined for 'fields' get_FieldInfo Title : get_FieldInfo Usage : my @fields = $info->get_FieldInfo; Function : returns list of FieldInfo objects Returns : array (FieldInfo objects) Args : none Notes : only applicable for einfo. next_LinkInfo Title : next_LinkInfo Usage : while (my $link = $info->next_LinkInfo) {...} Function : iterate through LinkInfo objects Returns : LinkInfo object Args : none Notes : only applicable for einfo. Uses callback() for filtering if defined for 'linkinfo' get_LinkInfo Title : get_LinkInfo Usage : my @links = $info->get_LinkInfo; Function : returns list of LinkInfo objects Returns : array (LinkInfo objects) Args : none Notes : only applicable for einfo. print_FieldInfo Title : print_FieldInfo Usage : $info->print_FieldInfo(); $info->print_FieldInfo(-fh => $fh, -cb => $coderef); Function : prints link data for each FieldInfo object. The default is generated via FieldInfo::to_string Returns : none Args : [optional] -file : file to print to -fh : filehandle to print to (cannot be used concurrently with file) -cb : coderef to use in place of default print method. This is passed in a FieldInfo object -wrap : number of columns to wrap default text output to (def = 80) Notes : only applicable for einfo. If -file or -fh are not defined, prints to STDOUT print_LinkInfo Title : print_LinkInfo Usage : $info->print_LinkInfo(); $info->print_LinkInfo(-fh => $fh, -cb => $coderef); Function : prints link data for each LinkInfo object. The default is generated via LinkInfo::to_string Returns : none Args : [optional] -file : file to print to -fh : filehandle to print to (cannot be used concurrently with file) -cb : coderef to use in place of default print method. This is passed in a LinkInfo object -wrap : number of columns to wrap default text output to (def = 80) Notes : only applicable for einfo. If -file or -fh are not defined, prints to STDOUT Bio::Tools::EUtilities::Link-related methods next_LinkSet Title : next_LinkSet Usage : while (my $ls = $eutil->next_LinkSet {...} Function : iterate through LinkSet objects Returns : LinkSet object Args : none Notes : only applicable for elink. Uses callback() for filtering if defined for 'linksets' get_LinkSets Title : get_LinkSets Usage : my @links = $info->get_LinkSets; Function : returns list of LinkSets objects Returns : array (LinkSet objects) Args : none Notes : only applicable for elink. print_LinkSets Title : print_LinkSets Usage : $info->print_LinkSets(); $info->print_LinkSets(-fh => $fh, -cb => $coderef); Function : prints link data for each LinkSet object. The default is generated via LinkSet::to_string Returns : none Args : [optional] -file : file to print to -fh : filehandle to print to (cannot be used concurrently with file) -cb : coderef to use in place of default print method. This is passed in a LinkSet object -wrap : number of columns to wrap default text output to (def = 80) Notes : only applicable for einfo. If -file or -fh are not defined, prints to STDOUT get_linked_databases Title : get_linked_databases Usage : my @dbs = $eutil->get_linked_databases Function : returns list of databases linked to in linksets Returns : array of databases Args : none Notes : only applicable for elink. Now defers to get_databases. Iterator- and callback-related methods rewind Title : rewind Usage : $esum->rewind() $esum->rewind('recursive') Function : retrieve a list of DocSum instances Returns : array of Bio::Tools::EUtilities::Summary::DocSum Args : [optional] Scalar; string ('all') to reset all iterators, or string describing the specific main object iterator to reset. The following are recognized (case-insensitive): 'all' - rewind all objects and also recursively resets nested object interators (such as LinkSets and DocSums). 'globalqueries' - GlobalQuery objects 'fieldinfo' or 'fieldinfos' - FieldInfo objects 'linkinfo' or 'linkinfos' - LinkInfo objects in this layer 'linksets' - LinkSet objects 'docsums' - DocSum objects 'histories' - HistoryI objects (Cookies, LinkSets) generate_iterator Title : generate_iterator Usage : my $coderef = $esum->generate_iterator('linkinfo') Function : generates an iterator (code reference) which iterates through the relevant object indicated by the args Returns : code reference Args : [REQUIRED] Scalar; string describing the specific object to iterate. The following are currently recognized (case-insensitive): 'globalqueries' 'fieldinfo' or 'fieldinfos' (the latter sounds clumsy, but I alias it JIC) 'linkinfo' or 'linkinfos' (the latter sounds clumsy, but I alias it JIC) 'linksets' 'docsums' 'histories' Note : This function generates a simple coderef that one can use independently of the various next_* functions (in fact, the next_* functions use lazily created iterators generated via this method, while rewind() merely deletes them so they can be regenerated on the next call). A callback specified using callback() will be used to filter objects for any generated iterator. This behaviour is implemented for both normal and lazy iterator types and is the default. If you don't want this, make sure to reset any previously set callbacks via reset_callback() (which just deletes the code ref). Note that setting callback() also changes the behavior of the next_* functions as the iterators are generated here (as described above); this is a feature and not a bug. 'Lazy' iterators are considered an experimental feature and may be modified in the future. A 'lazy' iterator, which loops through and returns objects as they are created (instead of creating all data instances up front, then iterating through) is returned if the parser is set to 'lazy' mode. This mode is only present for elink and esummary output as they are the two formats parsed which can generate potentially thousands of individual objects (note efetch isn't parsed, so isn't counted). Use of rewind() with these iterators is not supported for the time being as we can't guarantee you can rewind(), as this depends on whether the data source is seek()able and thus 'rewindable'. We will add rewind() support at a later time which will work for 'seekable' data or possibly cached objects via Storable or BDB. callback Title : callback Usage : $parser->callback(sub {$_[0]->get_database eq 'protein'}); Function : Get/set callback code ref used to filter returned data objects Returns : code ref if previously set Args : single argument: code ref - evaluates a passed object and returns true or false value (used in iterators) 'reset' - string, resets the iterator. returns upon any other args perl v5.14.2 2012-03-02 Bio::Tools::EUtilities(3pm)
All times are GMT -4. The time now is 10:35 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy