Bash script search, improve performance with large files


Login or Register to Reply

 
Thread Tools Search this Thread
# 15  
Quote:
Originally Posted by RudiC
Congrats, that would be a factor ~21 of performance gain!


I'd be surprised if the script needs the -F";" as it doesn't handle single fields but just the entire line, $0

I have no idea how this is possible. When i tried without, it did not work.


Anyhow its a mayor improvement over the old version.
# 16  
Quote:
Originally Posted by samitn
This works almost perfectly except that it takes ages to run on larger files.

I am not sure what you mean with a large file but i am running this for several weeks now with files around 150k ish. The complete script with 2x this one in it (1 to remove products that are in the filter and 1 to readd when it gets filtered (i use parts of names to filter)) and several mysql queries takes less then a min to complete that that is about max 30 sec for this 2 combined as 1 of the sql queries takes 30sec to run.


All in all i would not say that it is slow but super fast for my needs.
Login or Register to Reply

|
Thread Tools Search this Thread
Search this Thread:
Advanced Search

More UNIX and Linux Forum Topics You Might Find Helpful
How to improve the performance of this script?
vikatakavi
Hi , i wrote a script to convert dates to the formate i want .it works fine but the conversion is tkaing lot of time . Can some one help me tweek this script #!/bin/bash file=$1 ofile=$2 cp $file $ofile mydates=$(grep -Po '+/+/+' $ofile) # gets 8/1/13 mydates=$(echo "$mydates" | sort |...... UNIX for Dummies Questions & Answers
5
UNIX for Dummies Questions & Answers
Want to improve the performance of script
poweroflinux
Hi All, I have written a script as follows which is taking lot of time in executing/searching only 3500 records taken as input from one file in log file of 12 GB Approximately. Working of script is read the csv file as an input having 2 arguments which are transaction_id,mobile_number and search...... Shell Programming and Scripting
6
Shell Programming and Scripting
Improve the performance of a shell script
apsprabhu
Hi Friends, I wrote the below shell script to generate a report on alert messages recieved on a day. But i for processing around 4500 lines (alerts) the script is taking aorund 30 minutes to process. Please help me to make it faster and improve the performace of the script. i would be very...... Shell Programming and Scripting
10
Shell Programming and Scripting
Any way to improve performance of this script
sirababu
I have a data file of 2 gig I need to do all these, but its taking hours, any where i can improve performance, thanks a lot #!/usr/bin/ksh echo TIMESTAMP="$(date +'_%y-%m-%d.%H-%M-%S')" function showHelp { cat << EOF >&2 syntax extreme.sh FILENAME Specify filename to parse EOF...... Shell Programming and Scripting
3
Shell Programming and Scripting
UNIX and Linux

Unix (trademarked as the UNIX certification mark) is a very mature family of multitasking, multiuser computer operating systems that evolved from the original AT&T Unix. Unix development starting in the 1970s by legendary Bell Labs programmers Ken Thompson, Dennis Ritchie, and others. Unix was first targeted for the Bell System and AT&T licensed Unix to outside parties. In the 1970s time period, this lead to a variety of for-profit as well as not-for-profit Unix variants.
In the early days, this included the University of California, Berkeley (BSD), Microsoft (Xenix), IBM (AIX), and Sun Microsystems (Solaris). In the early 1990s, AT&T sold their Unix rights to Novell. In 1995 Novell sold their Unix business to the Santa Cruz Operation (SCO). The UNIX trademark was passed to The Open Group, a "neutral" industry consortium. The Open Group promoted the use of the UNIX trademark for certified operating systems that comply with the Single UNIX Specification (SUS). In 2014 Apple's macOS became the Unix version with the largest global install base and macOS remains the largest Unix-user base today.
Linux is a family of open source Unix-like operating systems based on the Linux kernel. The Linux kernel is an operating system kernel first released by Linus Torvalds on 17 September 1991. Linux distributions include the Linux kernel, system software and libraries. Popular free open source Linux distributions include Debian, Fedora, and Ubuntu. Commercial Linux distributions include Red Hat Enterprise Linux and SUSE Linux Enterprise Server. Linux may be freely modified and redistributed. Anyone on the planet may create a Linux distribution for any purpose.
Please enjoy and help our forum community by "showing your work" and posting your code, even when your code is not working as expected. To help others help you, please wrap your code blocks, sample input, sample output, error messages, and other data in CODE tags and wrap your short commands and short data objects in ICODE tags. We were all beginners in the beginning. If you have any questions about how to register or how to post, please contact us in Live Chat. Thank you and enjoy this "forever free" technical support community for UNIX, Linux and computer information technology in general.