Script works, but I think it could be better and faster


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Script works, but I think it could be better and faster
# 1  
Old 11-11-2018
Script works, but I think it could be better and faster

Hi All,

I'm new to the forum and to bash scripting. I did some stuff with VB.net, Batch, and VBScripting in the past, but because I shifted over to Linux, I am learning to script in Bash at this moment. So bear with me if I seem to script like a newbie, that's just because I am ;-)

OK, I wrote this script. It does the job, but it's rather slow. I get the server names, files, and password from separate text files. I place the servers and files into a list and loop them, but I configured only 2 local VM's at the moment and it runs rather slow already. I think I know why... It is logging into the server every time and then gets the file(s) I need. How do I overcome this? So, in my opinion, it just needs to login once and then gets the files off and after that move on to the other server.
Let me know what your thoughts are

Here it is:

Code:
#!/usr/bin/env bash

############################################ Global Declarations #################################################

home_dir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
dest_dir="/home/cornelis/config_files_properties"
ssh_user="root"
ssh_pass=$home_dir"/password_file" # Not needed on the script server
logfile="/tmp/config_files_properties.log"


############################################### Do the job #####################################################

## first thing to check if we are root, if not close
#
#if [ $(whoami) != 'root' ]; then
#        echo "Must be root to run $0"
#        exit 1;
#fi
# redirect all output to log file

exec &> >(while read -r line; do printf '%s %s\n' "$(date)" "$line"; done >> $logfile)

## check if destination folder exists and if not make it

if [[ ! -d "$dest_dir" ]]; then
    mkdir -p $dest_dir
fi

# Deleting existing files in the destination folder (if empty send error to /dev/null)

# extra error handling just in case, before removing anything

cd $dest_dir
if [[ "$?" = "0" ]]; then


# empty folder

shopt -s dotglob && rm -r $dest_dir/* > /dev/null 2>&1

# get the files off the servers and place them in the destination folder in its own folder

index=0
while read line; do
  server_array[index]="$line"
    for server in ${server_array[*]}
    do
    # create the folders named per server
        mkdir -p $dest_dir/$server
          #find the correct files
          index2=0
         while read line2; do
           file_array[index2]="$line2"
           for file in ${file_array[*]}
            do
            #copy the files to the folders
           sshpass -f $ssh_pass scp -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -rpqv $ssh_user@$server:$file $dest_dir/$server
           done
          done < $home_dir/files
    done

done < $home_dir/test_server_names
# change the directory and run SED on the files and folders
cd $dest_dir || exit 1

find ./ -type f -print0 | xargs -0 sed -Ei 's/([^&]password=)([^\t "]*)/\1xxxxxx/gI; s/(password.*>).*(<)/\1xxxxxx\2/gI; s/(password" value=").[^"]*/\1xxxxxx/gI; s/=admin:.[^\t ]*/=admin:xxxxxx/g; /"Password">/ {n;N;s/([\t ]*).*(\n.*<\/)/\1xxxxxx\2/gI}'

#zip the folders into one zip file

zip -rq9 properties.zip *


else
	echo "Cannot change directory to delete old files! Aborting! " 1>&2 >> $logfile
	exit 1
fi



# open the folder with contents

xdg-open $dest_dir

# 2  
Old 11-11-2018
I don't see an immediate reason why your script should run slow. Try using the -x (xtrace option to isolate where / which line it has its difficulties; mayhap the remote login?
Please post all files' contents so people get a chance to follow what you are doing.


Some remarks:
- you use arrays, but don't increment / modify the respective indices?
- you're using FOUR nested loops, to what avail?
- how do you expect to use ssh_pass variable / file on various servers?
# 3  
Old 11-12-2018
Quote:
I don't see an immediate reason why your script should run slow. Try using the -x (xtrace option to isolate where / which line it has its difficulties; mayhap the remote login?
Please post all files' contents so people get a chance to follow what you are doing.
Can you explain how to implement that xtrace? Is this an option on SCP? I can't find it in the manpage

And here is my reason that I think it can be better. It logs in each time it needs a file, instead of log in once and then gets ALL the files off and after that, it should log off and log on to the next server. in my noob opinion, this would save time. Correct me if I am wrong. I just don't know how to do that.
For two servers now it takes about 10 seconds, but in production, it will be a lot more and some are fast responding and some are not, so I want to have the script as fast as possible.

there is not much to it.
I have three plain text files
1 with two servers in it, each server on its own line like:
192.168.122.7
192.168.122.74

1 with the path to the files needed like this:
/path/to/file
/path/to/file2

also each on its own line.

the last file is the password file and has only one password in it. I don't need the password in production, because the script server already has root access to all the other servers and doesn't need to log in.

Some remarks:
[QOUTE]- you use arrays, but don't increment / modify the respective indices?[/QUOTE]
do you have a link to an explanation what you mean, so I check that and learn?
Quote:
- you're using FOUR nested loops, to what avail?
one for error handling (show a message and exit the script when the folder that needs to be empty, doesn't exist.)
one to loop through the servers
one to create the folders per server name
one to loop through the file paths

Quote:
- how do you expect to use ssh_pass variable / file on various servers?
As I explained above I don't need that in production but have that in place for my test VM's, they both have the same password, so I don't need to run through that file.

I included my log file to this message

Thanks for the quick response btw!

With kind regards,

Cornelis

Last edited by cornelvis; 11-12-2018 at 07:10 AM.. Reason: added log file
# 4  
Old 11-12-2018
btw I forgot to mention that one VM has the files and the other doesn't, just for testing purposes. SO you will see that in the log.
# 5  
Old 11-12-2018
Your shell - which you failed to mention, BTW - should provide the -x (xtrace) option to print to stderr the commands / lines it is about to execute.
One option to accelerate the script would be to optimise the order in which the files are accessed / evaluated, and to drop the for loops. On top, there's no need to read the files' file again and again, if the files don't change with the respective servers. Read once and build the scp into a variable.Try something along

Code:
readarray FNARR < filesfile
while read SVNM
  do CHKDIR/MKDIR serverdir
     scp -options $SVNM/${FNARR[@]} serverdir
  done < serverfile

copying all target files from a single server in one go... This is an untested frame / pseudo code proposal, so YMMV.

Last edited by RudiC; 11-12-2018 at 11:21 AM..
# 6  
Old 11-13-2018
Thanks for the reply, I am working with GNU bash, version 4.4.23(1)-release (x86_64-redhat-linux-gnu)
on Fedora 29 at the moment. I use PyCharm as an IDE.
I will give your proposal a try.



Hi,

I try to get my head around it and have narrowed it down to a small test script
Code:
############################################## Declarations #######################################################

home_dir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
dest_dir="/home/cornelis/config_files_properties"
ssh_user="root"
ssh_pass=$home_dir"/password_file" # Not needed on the script server
logfile="/tmp/config_files_properties.log"


################################################ Do the job #######################################################
#
index=0
readarray FNARR < $home_dir/files
while read SVNM
  do if [[ ! -d "$dest_dir$SVNM" ]]; then
    mkdir -p $dest_dir/$SVNM
fi
     sshpass -f $ssh_pass scp -o GSSAPIAuthentication=no -o GSSAPIKeyExchange=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -rpqv $ssh_user@$SVNM:${FNARR[@]} $FNARR

     echo $ssh_user@$SVNM${FNARR[@]} $FNARR
echo $SVNM
  done < $home_dir/test_server_names

xdg-open $dest_dir

What I get now is "No such file or directory" and when I echo the path and server they are correct, plus the file exists on the server.
What am I not getting here? I must be doing something wrong somewhere.

Thanks in advance.

Cornelis

------ Post updated at 03:05 PM ------

Got it working like this now. Already reduced to 1 for loop:

Code:
index=0
readarray FNARR < $home_dir/files
while read SVNM
  do if [[ ! -d "$dest_dir/$SVNM" ]]; then
    mkdir -p $dest_dir/$SVNM
     fi
    for file in ${FNARR[*]}
                    do
   sshpass -f $ssh_pass scp -o GSSAPIAuthentication=no -o GSSAPIKeyExchange=no -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -rpqv $ssh_user@$SVNM:$file $dest_dir/$SVNM
    done

   done < $home_dir/test_server_names

xdg-open $dest_dir


Last edited by cornelvis; 11-14-2018 at 08:22 AM..
This User Gave Thanks to cornelvis For This Post:
# 7  
Old 11-13-2018
Thanks for trying - that's exactly the way to take. Reduce complexity, here: to one single file, and proceed step by step.


scp - as can cp - can copy multiple files into a target directory in one go. Your bash version offers "Parameter Expansion / Pattern substitution", so try like
Code:
scp  ${FNARR[@]/#/$ssh_user@$SVNM:} $SVNM

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Shell script works fine as a standalone script but not as part of a bigger script

Hello all, I am facing a weird issue while executing a code below - #!/bin/bash cd /wload/baot/home/baotasa0/sandboxes_finance/ext_ukba_bde/pset sh UKBA_publish.sh UKBA 28082015 3 if then echo "Param file conversion for all the areas are completed, please check in your home directory"... (2 Replies)
Discussion started by: ektubbe
2 Replies

2. Shell Programming and Scripting

Optimize shell script to run faster

data.file: contact { contact_name=royce-rolls modified_attributes=0 modified_host_attributes=0 modified_service_attributes=0 host_notification_period=24x7 service_notification_period=24x7 last_host_notification=0 last_service_notification=0 host_notifications_enabled=1... (8 Replies)
Discussion started by: SkySmart
8 Replies

3. Shell Programming and Scripting

Making script run faster

Can someone help me edit the below script to make it run faster? Shell: bash OS: Linux Red Hat The point of the script is to grab entire chunks of information that concerns the service "MEMORY_CHECK". For each chunk, the beginning starts with "service {", and ends with "}". I should... (15 Replies)
Discussion started by: SkySmart
15 Replies

4. Shell Programming and Scripting

Make script faster

Hi all, In bash scripting, I use to read files: cat $file | while read line; do ... doneHowever, it's a very slow way to read file line by line. E.g. In a file that has 3 columns, and less than 400 rows, like this: I run next script: cat $line | while read line; do ## Reads each... (10 Replies)
Discussion started by: AlbertGM
10 Replies

5. Shell Programming and Scripting

Script to parse a file faster

My example file is as given below: conn=1 uid=oracle conn=2 uid=db2 conn=3 uid=oracle conn=4 uid=hash conn=5 uid=skher conn=6 uid=oracle conn=7 uid=mpalkar conn=8 uid=anarke conn=1 op=-1 msgId=-1 - fd=104 slot=104 LDAPS connection from 10.10.5.6 to 10.18.6.5 conn=2 op=-1 msgId=-1 -... (7 Replies)
Discussion started by: sags007_99
7 Replies

6. Shell Programming and Scripting

Script works with bash 3.0 but not 3.2.

Hello, So my knowledge of bash scripting is not that great and I have been trying to solve this problem on my own for awhile to no avail. Here's the error I get when running it with an OS that uses bash 3.2.x: testagain.sh: line 10: *-1: syntax error: operand expected (error token is... (2 Replies)
Discussion started by: forkandspoon
2 Replies

7. Shell Programming and Scripting

Can anyone make this script run faster?

One of our servers runs Solaris 8 and does not have "ls -lh" as a valid command. I wrote the following script to make the ls output easier to read and emulate "ls -lh" functionality. The script works, but it is slow when executed on a directory that contains a large number of files. Can anyone make... (10 Replies)
Discussion started by: shew01
10 Replies

8. UNIX for Advanced & Expert Users

Country Codes script faster response ;please help

Dear all I have group of input lines which look like this These input lines is placed in a file named phonelines.txt and there is a script which match $4 and $5 with country codes placed in another file named country-codes.txt and its contents is : Italy 39 Libyana 21892 Thuraya... (12 Replies)
Discussion started by: zanetti321
12 Replies

9. Shell Programming and Scripting

perl - how come this script works?

#!/usr/bin/perl open (DATA, file.txt); @array = <DATA>; close (DATA); open (DATA, ">$file.txt"); for (@array) { s/text/replace text/; push(@contents,$_); } seek(DATA, 0, 0); print DATA (@contents); close(DATA); could someone please explain how this works. i've been... (3 Replies)
Discussion started by: mjays
3 Replies

10. Shell Programming and Scripting

Script works fine until I | more

Hello all, This beats me. I have a script that executes some commands and redirects their output to some text files that I will parse. The commands are along the lines of: dsmadmc -id=admin -pa=admin -outfile=/home/tools/qlog.txt q log f=d If I just run the script it works. If I execute... (2 Replies)
Discussion started by: Skovian
2 Replies
Login or Register to Ask a Question