05-10-2011
What Operating System and version do you have? -- Linux .....2.6.9-42.ELsmp #1 SMP Wed Jul 12 23:27:17 EDT 2006 i686 i686 i386 GNU/Linux
What Shell do you prefer? /bin/ksh
How big is the keywords file? 50k keywords (each keyword upto 30-40 char)
How big is the total of the 3k data files? Each file will have about 300-400 lines
Are these all normal unix text files with a reasonable record size? All text files
You appear to be attempting 150,000,000 serial file passes (15,000 x 3,000) -- Yes :-(
Is this a one-off or something which will be run again and again? -- Not one off. This will done regularly
Do you have a full-works database engine such as Oracle? -- if found to be more efficient we can get a database engine such as Oracle
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Because I am not creative, I did this:
find . -type f -name '*.GIF'|cut -d'/' -f2|awk -F. '{print "mv "$1".GIF "$1".gif --reply=yes"}' > case.sh
Then ran the case.sh - I was wondering if you guys could come up with something more efficient? Or even limit CPU useage? It is killing my poor ext3... (3 Replies)
Discussion started by: r0sc0
3 Replies
2. Shell Programming and Scripting
Hi,
I have a challenging task,in which i have to find the duplicate files by its name and size,then i need to take anyone of the file.Then i need to open the file and find for more than one pattern and count of that pattern.
Note:These are the samples of two files,but i can have more... (2 Replies)
Discussion started by: jerome Sukumar
2 Replies
3. Shell Programming and Scripting
How can we find "latest files which have been recently updated/changed/created" in solaris 10??? (3 Replies)
Discussion started by: asadlone
3 Replies
4. UNIX for Dummies Questions & Answers
Hi to all
Sorry for the confusion because I did not explain the task clearly.
There are many .hhr files in a folder
There are so many lines in these .hhr files but I want only the following 2 lines to be transferred to the output file.
The keyword No 1 and all the words in the next line
They... (5 Replies)
Discussion started by: raghulrajan
5 Replies
5. Shell Programming and Scripting
Hi guys can you please help me with a script to find files with one row/1 line of content then move the file to another directory my script below runs but nothing happens to the files....Alternatively Ca I get a script to find the *.csv files with "wc -1" results = 1 then create a list of those... (5 Replies)
Discussion started by: Dj Moi
5 Replies
6. UNIX for Advanced & Expert Users
I have a huge list of files in an Unix directory (around 10000 files).
I need to be able to search for a certain keyword only within files that are modified between certain date and time, say for e.g 2012-08-20 12:30 to 2012-08-20 12:40
Can someone let me know what would be the fastest way... (10 Replies)
Discussion started by: virtual123
10 Replies
7. Shell Programming and Scripting
I have ~100 text files in a directory that I am trying to parse and output to a new file. I am looking for the words chr,start,stop,ref,alt in each of the files. Those fields should appear somewhere in those files. The first two fields of each new set of rows is also printed. Since this is on a... (7 Replies)
Discussion started by: cmccabe
7 Replies
8. UNIX for Dummies Questions & Answers
The Problem that I am having is when the code ran and populated the progflag.csv file, columns MEMSIZE, SECOND and SASEXE were blank. The next problems are the IF else statement isn't working and the email function isn't sending the progflag.csv attachment.
a. What I want the program to do is to... (2 Replies)
Discussion started by: dellanicholson
2 Replies
9. Shell Programming and Scripting
I have several problems with my program: I hope you can help me.
1) the If else statement isn't working . The IF Else syntax is:
If MEMSIZE OR sasfoundation (SASEXE) OR Real Time(second) >1.0 and Filename, output column name and value to csv or else nothing
Example progflag,cvs:... (13 Replies)
Discussion started by: dellanicholson
13 Replies
10. UNIX for Beginners Questions & Answers
I have two files to be compared to get the output of the differences.
File1 has a lot more lists than File2.
After searching a lot on this thread I'am unable to find the exact code that im willing to get.
This will be used as 'pre-check'/post-check utility (health check Tool) to compare... (1 Reply)
Discussion started by: GeekyJimmy
1 Replies
LEARN ABOUT DEBIAN
bp_load_gff
BP_LOAD_GFF(1p) User Contributed Perl Documentation BP_LOAD_GFF(1p)
NAME
bp_load_gff.pl - Load a Bio::DB::GFF database from GFF files.
SYNOPSIS
% bp_load_gff.pl -d testdb -u user -p pw
--dsn 'dbi:mysql:database=dmel_r5_1;host=myhost;port=myport'
dna1.fa dna2.fa features1.gff features2.gff ...
DESCRIPTION
This script loads a Bio::DB::GFF database with the features contained in a list of GFF files and/or FASTA sequence files. You must use the
exact variant of GFF described in Bio::DB::GFF. Various command-line options allow you to control which database to load and whether to
allow an existing database to be overwritten.
This script uses the Bio::DB::GFF interface, and so works with all database adaptors currently supported by that module (MySQL, Oracle,
PostgreSQL soon). However, it is slow. For faster loading, see the MySQL-specific bp_bulk_load_gff.pl and bp_fast_load_gff.pl scripts.
NOTES
If the filename is given as "-" then the input is taken from standard input. Compressed files (.gz, .Z, .bz2) are automatically
uncompressed.
FASTA format files are distinguished from GFF files by their filename extensions. Files ending in .fa, .fasta, .fast, .seq, .dna and their
uppercase variants are treated as FASTA files. Everything else is treated as a GFF file. If you wish to load -fasta files from STDIN,
then use the -f command-line swith with an argument of '-', as in
gunzip my_data.fa.gz | bp_fast_load_gff.pl -d test -f -
On the first load of a database, you will see a number of "unknown table" errors. This is normal.
About maxfeature: the default value is 100,000,000 bases. If you have features that are close to or greater that 100Mb in length, then the
value of maxfeature should be increased to 1,000,000,000, or another power of 10.
COMMAND-LINE OPTIONS
Command-line options can be abbreviated to single-letter options. e.g. -d instead of --database.
--dsn <dsn> Data source (default dbi:mysql:test)
--adaptor <adaptor> Schema adaptor (default dbi::mysqlopt)
--user <user> Username for mysql authentication
--pass <password> Password for mysql authentication
--fasta <path> Fasta file or directory containing fasta files for the DNA
--create Force creation and initialization of database
--maxfeature Set the value of the maximum feature size (default 100 Mb; must be a power of 10)
--group A list of one or more tag names (comma or space separated)
to be used for grouping in the 9th column.
--upgrade Upgrade existing database to current schema
--gff3_munge Activate GFF3 name munging (see Bio::DB::GFF)
--quiet No progress reports
--summary Generate summary statistics for drawing coverage histograms.
This can be run on a previously loaded database or during
the load.
SEE ALSO
Bio::DB::GFF, bulk_load_gff.pl, load_gff.pl
AUTHOR
Lincoln Stein, lstein@cshl.org
Copyright (c) 2002 Cold Spring Harbor Laboratory
This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself. See DISCLAIMER.txt for
disclaimers of warranty.
perl v5.14.2 2012-03-02 BP_LOAD_GFF(1p)