Sponsored Content
Top Forums Shell Programming and Scripting Bash script search, improve performance with large files Post 303034149 by SDohmen on Friday 19th of April 2019 11:42:16 AM
Old 04-19-2019
Quote:
Originally Posted by samitn
This works almost perfectly except that it takes ages to run on larger files.

I am not sure what you mean with a large file but i am running this for several weeks now with files around 150k ish. The complete script with 2x this one in it (1 to remove products that are in the filter and 1 to readd when it gets filtered (i use parts of names to filter)) and several mysql queries takes less then a min to complete that that is about max 30 sec for this 2 combined as 1 of the sql queries takes 30sec to run.


All in all i would not say that it is slow but super fast for my needs.
 

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Improve Performance

hi someone tell me which ways i can improve disk I/O and system process performance.kindly refer some commands so i can do it on my test machine.thanks, Mazhar (2 Replies)
Discussion started by: mazhar99
2 Replies

2. Shell Programming and Scripting

Any way to improve performance of this script

I have a data file of 2 gig I need to do all these, but its taking hours, any where i can improve performance, thanks a lot #!/usr/bin/ksh echo TIMESTAMP="$(date +'_%y-%m-%d.%H-%M-%S')" function showHelp { cat << EOF >&2 syntax extreme.sh FILENAME Specify filename to parse EOF... (3 Replies)
Discussion started by: sirababu
3 Replies

3. Shell Programming and Scripting

Improve the performance of a shell script

Hi Friends, I wrote the below shell script to generate a report on alert messages recieved on a day. But i for processing around 4500 lines (alerts) the script is taking aorund 30 minutes to process. Please help me to make it faster and improve the performace of the script. i would be very... (10 Replies)
Discussion started by: apsprabhu
10 Replies

4. Shell Programming and Scripting

Want to improve the performance of script

Hi All, I have written a script as follows which is taking lot of time in executing/searching only 3500 records taken as input from one file in log file of 12 GB Approximately. Working of script is read the csv file as an input having 2 arguments which are transaction_id,mobile_number and search... (6 Replies)
Discussion started by: poweroflinux
6 Replies

5. Programming

Help with improve the performance of grep

Input file: #content_1 12314345345 242467 #content_14 436677645 576577657 #content_100 3425546 56 #content_12 243254546 1232454 . . Reference file: content_100 (1 Reply)
Discussion started by: cpp_beginner
1 Replies

6. Shell Programming and Scripting

Performance issue in Grepping large files

I have around 300 files(*.rdf,*.fmb,*.pll,*.ctl,*.sh,*.sql,*.prog) which are of large size. Around 8000 keywords(which will be in the file $keywordfile) needed to be searched inside those files. If a keyword is found in a file..I have to insert the filename,extension,catagoery,keyword,occurrence... (8 Replies)
Discussion started by: millan
8 Replies

7. UNIX for Dummies Questions & Answers

How to improve the performance of this script?

Hi , i wrote a script to convert dates to the formate i want .it works fine but the conversion is tkaing lot of time . Can some one help me tweek this script #!/bin/bash file=$1 ofile=$2 cp $file $ofile mydates=$(grep -Po '+/+/+' $ofile) # gets 8/1/13 mydates=$(echo "$mydates" | sort |... (5 Replies)
Discussion started by: vikatakavi
5 Replies

8. Shell Programming and Scripting

Copying large files in a bash script stops execution

Hello, I'm new to this forum and like to first of all say hello to everyone. I've got a really annoying problem at the moment. I'm trying to rsync some files (about 200MB with one file of 120MB) from a Raspberry PI with raspbian to a debian server via rsync. This procedure is stored in a... (3 Replies)
Discussion started by: wex_storm
3 Replies

9. Programming

Improve the performance of my C++ code

Hello, Attached is my very simple C++ code to remove any substrings (DNA sequence) of each other, i.e. any redundant sequence is removed to get unique sequences. Similar to sort | uniq command except there is reverse-complementary for DNA sequence. The program runs well with small dataset, but... (11 Replies)
Discussion started by: yifangt
11 Replies
MYSQLDUMPSLOW(1)					       MySQL Database System						  MYSQLDUMPSLOW(1)

NAME
mysqldumpslow - Summarize slow query log files SYNOPSIS
mysqldumpslow [options] [log_file ...] DESCRIPTION
The MySQL slow query log contains information about queries that take a long time to execute (see Section 5.2.5, "The Slow Query Log"). mysqldumpslow parses MySQL slow query log files and prints a summary of their contents. Normally, mysqldumpslow groups queries that are similar except for the particular values of number and string data values. It "abstracts" these values to N and 'S' when displaying summary output. The -a and -n options can be used to modify value abstracting behavior. Invoke mysqldumpslow like this: shell> mysqldumpslow [options] [log_file ...] mysqldumpslow supports the following options. o --help Display a help message and exit. o -a Do not abstract all numbers to N and strings to 'S'. o --debug, -d Run in debug mode. o -g pattern Consider only queries that match the (grep-style) pattern. o -h host_name Host name of MySQL server for *-slow.log file name. The value can contain a wildcare. The default is * (match all). o -i name Name of server instance (if using mysql.server startup script). o -l Do not subtract lock time from total time. o -n N Abstract numbers with at least N digits within names. o -r Reverse the sort order. o -s sort_type How to sort the output. The value of sort_type should be chosen from the following list: o t, at: Sort by query time or average query time o l, al: Sort by lock time or average lock time o s, as: Sort by rows sent or average rows sent o c: Sort by count o -t N Display only the first N queries in the output. o --verbose, -v Verbose mode. Print more information about what the program does. Example of usage: shell> mysqldumpslow Reading mysql slow query log from /usr/local/mysql/data/mysqld51-apple-slow.log Count: 1 Time=4.32s (4s) Lock=0.00s (0s) Rows=0.0 (0), root[root]@localhost insert into t2 select * from t1 Count: 3 Time=2.53s (7s) Lock=0.00s (0s) Rows=0.0 (0), root[root]@localhost insert into t2 select * from t1 limit N Count: 3 Time=2.13s (6s) Lock=0.00s (0s) Rows=0.0 (0), root[root]@localhost insert into t1 select * from t1 COPYRIGHT
Copyright 2007-2008 MySQL AB, 2008-2010 Sun Microsystems, Inc. This documentation is free software; you can redistribute it and/or modify it only under the terms of the GNU General Public License as published by the Free Software Foundation; version 2 of the License. This documentation is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details. You should have received a copy of the GNU General Public License along with the program; if not, write to the Free Software Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA or see http://www.gnu.org/licenses/. SEE ALSO
For more information, please refer to the MySQL Reference Manual, which may already be installed locally and which is also available online at http://dev.mysql.com/doc/. AUTHOR
Sun Microsystems, Inc. (http://www.mysql.com/). MySQL 5.1 04/06/2010 MYSQLDUMPSLOW(1)
All times are GMT -4. The time now is 10:14 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy