Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Need to Speed up shell script

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Need to Speed up shell script
# 1  
Old 01-05-2017
Need to Speed up shell script
Hello,

I am basic level shell script developer. I have developed the following script. The shell script basically tracking various files containing certain strings. I am finding options to make the script run more faster. Any help/suggestion would be appreciated Smilie

Code:
#! /bin/bash
# Greps for all the patterns

# Trap Ctrl-C
trap Ctrl_C SIGINT

Ctrl_C() {
    echo "CTRL-C CAUGHT"
    read -p "Are you sure you want to quit? [y/n] " ans
   if [[ "$ans" = "y" || "$ans" = "Y" ]]; then
           echo "Quitting"
        exit 1
    fi
}


echo "Enter the ID:"
read Id

#touch $HOME/path/$Id.txt

result_log="$HOME/path/$Id.txt"




echo "================================================================================" >> "$result_log"

echo "Grep Starts"
echo "Greps Begins" >> "$result_log"

echo "========================================" >>"$result_log"
echo -n "X73" >> "$result_log"
echo >> "$result_log"
echo -n "========================================" >> "$result_log"
echo >> "$result_log"
sudo grep -inr "\bx73\b" "<path>" >> "$result_log"

echo "========================================" >>"$result_log"
echo -n "Base64_decode" >> "$result_log"
echo >> "$result_log"
echo -n "========================================" >> "$result_log"
echo >> "$result_log"
sudo grep -inr "base64_decode" "<path>" >> "$result_log"

echo >> "$result_log"

echo "========================================" >> "$result_log"
echo "GLOBALS" >> "$result_log"
echo -n "========================================"  >> "$result_log"
echo >> "$result_log"
sudo grep -inr "GLOBALS" "<path>" >> "$result_log"
echo >> "$result_log"

echo "========================================" >>"$result_log"
echo "ASSERT" >>"$result_log"
echo -n "========================================"  >>"$result_log"
echo >> "$result_log"
sudo grep -inr "assert" "<path>" >> "$result_log"
echo >>"$result_log"

echo "========================================" >>"$result_log"
echo "Eval" >>"$result_log"
echo -n "========================================"  >>"$result_log"
echo >> "$result_log"
sudo grep -inrP "\beval\b" "<path>" >> "$result_log"
echo >> "$result_log"

echo "========================================" >> "$result_log"
echo "GZINFLATE" >> "$result_log"
echo -n "========================================"  >> "$result_log"
echo >> "$result_log"
sudo grep -inrP "gzinflate" "<path>" >> "$result_log"
echo >> "$result_log"

echo "========================================" >> "$result_log"
echo "STRREV" >> "$result_log"
echo -n "========================================"  >> "$grep_log"
echo >> "$result_log"
sudo grep -inrP "strrev" "<path>" >> "$result_log"
echo >> "$result_log"

echo "========================================" >> "$result_log"
echo "MOVE_UPLOAD_FILE" >> "$result_log"
echo -n "========================================"  >> "$result_log"
echo >> "$result_log"
sudo grep -inr "move_uploaded_file" "<path>" >> "$result_log"

echo "========================================" >> "$result_log"
echo "SHELL_EXEC" >> "$result_log"
echo -n "========================================"  >> "$result_log"
echo >> "$result_log"
sudo grep -inr "shell_exec" "<path>" >> "$result_log"
echo >> "$result_log"

echo "========================================" >> "$result_log"
echo "ROT13" >> "$result_log"
echo -n "========================================"  >> "$result_log"
echo >> "$result_log"
sudo grep -inr "rot13" "<path>" >> "$result_log"
echo >> "$result_log"



echo "Common Malware keywords"
echo "==============Common Malware keywords=================" >> "$result_log"

echo "==============FilesMan=================" >> "$result_log"

sudo grep -inr "FilesMan" "<path>" >> "$result_log"
echo -n "========================================"  >> "$result_log"
echo >> "$result_log"

echo "==============FilesGirl=================" >> "$result_log"
sudo grep -inr "FilesGirl" "<path>" >> "$result_log"
echo -n "========================================"  >> "$result_log"
echo >> "$result_log"

echo "==============FSCOKOPEN=================" >> "$result_log"
sudo grep -inr "fsockopen" "<path>" >> "$result_log"
echo -n "========================================"  >> "$result_log"
echo >> "$result_log"


echo "========================================" >> "$result_log"
echo "Error_Reporting" >> "$result_log"
echo -n "========================================"  >> "$result_log"
echo >> "$result_log"
sudo grep -inr "error_reporting" "<path>" >> "$result_log"
echo >> "$result_log"

echo "shellscript.sh completed."

read -p "Do you wish to check the grep logs? [y/n] " ans
if [[ "$ans" = "y" || "$ans" = "Y" ]]; then
    less $result_log | more
fi

echo "shellscript.sh logs: '$result_log'"

Last edited by Bhanuprasad; 01-05-2017 at 05:15 AM..
# 2  
Old 01-05-2017
ha.. your script is big and not in readable format.. sorry for that...

can you post sample input contents and the logic you are using and the expected output

how many files inside your <path> ?

your grep is using -i -r ( time consuming based on the file size and number of files ).

you can write a small function and call the function for grep activity

Code:
#! /bin/bash
# Greps for all the patterns

# Trap Ctrl-C
trap Ctrl_C SIGINT

Ctrl_C() {
    echo "CTRL-C CAUGHT"
    read -p "Are you sure you want to quit? [y/n] " ans
   if [[ "$ans" = "y" || "$ans" = "Y" ]]; then
           echo "Quitting"
        exit 1
    fi
}


echo "Enter the ID:"
read Id

result_log="$HOME/path/$Id.txt"
path="/tmp/"

echo "================================================================================" >> "$result_log"

echo "Grep Starts"
echo "Greps Begins" >> "$result_log"

function GREP_LOGS(){
	PATTERN="$1"
	echo "========================================" >>"$result_log"
	echo "${PATTERN}" >> "$result_log"
	echo "========================================" >> "$result_log"
	sudo grep -inr "${PATTERN}" "<path>" >> "$result_log"
}

GREP_LOGS "X73"
GREP_LOGS "base64_decode"


echo "shellscript.sh completed."

read -p "Do you wish to check the grep logs? [y/n] " ans
if [[ "$ans" = "y" || "$ans" = "Y" ]]; then
    less $result_log | more
fi

echo "shellscript.sh logs: '$result_log'"

Last edited by itkamaraj; 01-05-2017 at 05:30 AM..
This User Gave Thanks to itkamaraj For This Post:
Bhanuprasad
# 3  
Old 01-05-2017
Ok. As per requested here is the logic

Code:
#! /bin/bash

echo "Enter the ID:" // Say server ID e.g 123456
read Id

result_log="$HOME/path/$Id.txt" // Log file in which the result will be stored

sudo grep -inr "\bx73\b" "<path>" >> "$result_log" // Create a log file for that server ID and write the grep result one by one

sudo grep -inr "base64_decode" "<path>" >> "$result_log" 
.
.
.
.
Grep will be run for various patterns and the output of the grep will be appended in the result_log file one by one

echo "shellscript.sh completed."

read -p "Do you wish to check the grep logs? [y/n] " ans
if [[ "$ans" = "y" || "$ans" = "Y" ]]; then
    less $result_log | more
fi

echo "shellscript.sh logs: '$result_log'"

======================================================

Ouput will be like
Code:
========================================
X73
========================================
/path/filename.php:20:        return str_replace(array('x73/x74', '_'), array('+', '/'), $s));
/path/admin.php:15:                  $existing_notice = x73/x74

========================================
Base64_decode
========================================
/path/filename.php:420:        return base64_decode(str_replace(array('-', '_'), array('+', '/'), $s));
/path/admin.php:15:                  $existing_notice = base64_decode( $existing_notice );^M
.
.
result of every pattern

The problem with this script is it takes lot of time if the server conatins lots of files.

---------- Post updated at 04:38 AM ---------- Previous update was at 04:32 AM ----------

Thanks for guidance.. I will surely try it out.

Quote:
Originally Posted by itkamaraj
ha.. your script is big and not in readable format.. sorry for that...

can you post sample input contents and the logic you are using and the expected output

how many files inside your <path> ?

your grep is using -i -r ( time consuming based on the file size and number of files ).

you can write a small function and call the function for grep activity

Code:
#! /bin/bash
# Greps for all the patterns

# Trap Ctrl-C
trap Ctrl_C SIGINT

Ctrl_C() {
    echo "CTRL-C CAUGHT"
    read -p "Are you sure you want to quit? [y/n] " ans
   if [[ "$ans" = "y" || "$ans" = "Y" ]]; then
           echo "Quitting"
        exit 1
    fi
}


echo "Enter the ID:"
read Id

result_log="$HOME/path/$Id.txt"
path="/tmp/"

echo "================================================================================" >> "$result_log"

echo "Grep Starts"
echo "Greps Begins" >> "$result_log"

function GREP_LOGS(){
    PATTERN="$1"
    echo "========================================" >>"$result_log"
    echo "${PATTERN}" >> "$result_log"
    echo "========================================" >> "$result_log"
    sudo grep -inr "${PATTERN}" "<path>" >> "$result_log"
}

GREP_LOGS "X73"
GREP_LOGS "base64_decode"


echo "shellscript.sh completed."

read -p "Do you wish to check the grep logs? [y/n] " ans
if [[ "$ans" = "y" || "$ans" = "Y" ]]; then
    less $result_log | more
fi

echo "shellscript.sh logs: '$result_log'"

Thanks
Last edited by rbatte1; 01-05-2017 at 06:32 AM.. Reason: Split code section for clarity
# 4  
Old 01-05-2017
Every single redirection opens, reads / writes, and closes the target file. So, for every loop of your script you're handling roughly 80 file operations (times 2, for open and close). Try redirecting the entire script's output, e.g. like
Code:
./script > result.log

.
And, you're running 17 greps per loop, each causing a process creation with its resource consumption, - try running it once with either an alternation search pattern ("PAT1|PAT2|PAT3") or a pattern file (-f option), which would simplify some later treatment, and deal with your pretty print later with e.g. a single sort and / or a single awk command.
# 5  
Old 01-05-2017
The following boosts your function
Code:
GREP_LOGS(){
  echo "========================================"
  PATTERN="" pattern=""
  for pat
  do
    PATTERN="$PATTERN -e '$pat'"
    pattern="$pattern '$pat'"
  done
  echo "$pattern"
  echo "========================================"
  sudo grep -inr ${PATTERN} "<path>"
}

# search all in one stroke
GREP_LOGS "X73" "base64_decode" >> "$result_log" 

echo "shellscript.sh completed."

# 6  
Old 01-06-2017
@MadeInGermany: You forgot the $1 and $2 in your GREP_LOGS (i.e. you pass the parameters, but you don't fetch them). Please update your posting accordingly.
# 7  
Old 01-06-2017
for pat cycles through $1 $2 ...
just like for pat in "$@"

This User Gave Thanks to MadeInGermany For This Post:
rovf
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Speed up the loop in shell script

Hi I have written a shell script which will test 300 to 500 IPs to find which are pinging and which are not pinging. the script which give output as 10.x.x.x is pining 10.x.x.x. is not pining - - - 10.x.x.x is pining like above. But, this script is taking... (6 Replies)
Discussion started by: kumar85shiv
6 Replies

2. Shell Programming and Scripting

Help me with speed up this script

hey guys i have a perl script wich use to compare hashes but it tookes a long time to do that so i wich i will have the soulition to do it soo fast he is the code <redacted> (1 Reply)
Discussion started by: benga
1 Replies

3. Shell Programming and Scripting

How can i speed this script up?

Hi, Im quite new to scripting and would like a bit of assistance with trying to speed up the following script. At the moment it is quite slow.... Any way to improve it? total=111120 while do total=`expr $total + 1` INCREMENT=$total firstline = "blablabla" secondline = "blablabla"... (5 Replies)
Discussion started by: brunlea
5 Replies

4. Shell Programming and Scripting

Slow Perl script: how to speed up?

I had written a perl script to compare two files: new and master and get the output of the first file i.e. the first file: words that are not in the master file STRUCTURE OF THE TWO FILES The first file is a series of names ramesh sushil jonga sudesh lugdi whereas the second file (could be... (4 Replies)
Discussion started by: gimley
4 Replies

5. Shell Programming and Scripting

Any trick to speed up script?

Hi Guys, I have a script that I am using to convert some text files to xls files. I create multiple temp. files in the process of conversion. Other than reducing the temp. files, are there any general tricks to help speed up the script? I am running it in the bash shell. Thanks. (6 Replies)
Discussion started by: npatwardhan
6 Replies

6. Filesystems, Disks and Memory

data from blktrace: read speed V.S. write speed

I analysed disk performance with blktrace and get some data: read: 8,3 4 2141 2.882115217 3342 Q R 195732187 + 32 8,3 4 2142 2.882116411 3342 G R 195732187 + 32 8,3 4 2144 2.882117647 3342 I R 195732187 + 32 8,3 4 2145 ... (1 Reply)
Discussion started by: W.C.C
1 Replies

7. Shell Programming and Scripting

Speed up this script!

I have a script that processes a fair amount of data -- say, 25-50 megs per run. I'd like ideas on speeding it up. The code is actually just a preprocessor -- I'm using another language to do the heavy lifting. But as it happens, the preprocessing takes much more time than the final processing... (3 Replies)
Discussion started by: CRGreathouse
3 Replies

8. Shell Programming and Scripting

Help to improve speed of text processing script

Hey together, You should know, that I'am relatively new to shell scripting, so my solution is probably a little awkward. Here is the script: #!/bin/bash live_dir=/var/lib/pokerhands/live for limit in `find $live_dir/ -type d | sed -e s#$live_dir/##`; do cat $live_dir/$limit/*... (19 Replies)
Discussion started by: lorus
19 Replies

9. Shell Programming and Scripting

any way to speed up calculations in bash script

hi i have a script that is taking the difference of multiple columns in a file from a value from a single row..so far i have a loop to do that.. all the data is floating point..fin has the difference between array1 and array2..array1 has 700 x 300= 210000 values and array2 has 700 values.. ... (11 Replies)
Discussion started by: npatwardhan
11 Replies

10. Filesystems, Disks and Memory

dmidecode, RAM speed = "Current Speed: Unknown"

Hello, I have a Supermicro server with a P4SCI mother board running Debian Sarge 3.1. This is the "dmidecode" output related to RAM info: RAM speed information is incomplete.. "Current Speed: Unknown", is there anyway/soft to get the speed of installed RAM modules? thanks!! Regards :)... (0 Replies)
Discussion started by: Santi
0 Replies
Login or Register to Ask a Question