Script works, but I think it could be better and faster Post: 303025744

Sponsored Content

Top Forums Shell Programming and Scripting Script works, but I think it could be better and faster Post 303025744 by cornelvis on Sunday 11th of November 2018 07:22:20 AM

11-11-2018

Registered User

Script works, but I think it could be better and faster

Hi All,

I'm new to the forum and to bash scripting. I did some stuff with VB.net, Batch, and VBScripting in the past, but because I shifted over to Linux, I am learning to script in Bash at this moment. So bear with me if I seem to script like a newbie, that's just because I am ;-)

OK, I wrote this script. It does the job, but it's rather slow. I get the server names, files, and password from separate text files. I place the servers and files into a list and loop them, but I configured only 2 local VM's at the moment and it runs rather slow already. I think I know why... It is logging into the server every time and then gets the file(s) I need. How do I overcome this? So, in my opinion, it just needs to login once and then gets the files off and after that move on to the other server.
Let me know what your thoughts are

Here it is:

Code:

#!/usr/bin/env bash

############################################ Global Declarations #################################################

home_dir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" >/dev/null && pwd )"
dest_dir="/home/cornelis/config_files_properties"
ssh_user="root"
ssh_pass=$home_dir"/password_file" # Not needed on the script server
logfile="/tmp/config_files_properties.log"


############################################### Do the job #####################################################

## first thing to check if we are root, if not close
#
#if [ $(whoami) != 'root' ]; then
#        echo "Must be root to run $0"
#        exit 1;
#fi
# redirect all output to log file

exec &> >(while read -r line; do printf '%s %s\n' "$(date)" "$line"; done >> $logfile)

## check if destination folder exists and if not make it

if [[ ! -d "$dest_dir" ]]; then
    mkdir -p $dest_dir
fi

# Deleting existing files in the destination folder (if empty send error to /dev/null)

# extra error handling just in case, before removing anything

cd $dest_dir
if [[ "$?" = "0" ]]; then


# empty folder

shopt -s dotglob && rm -r $dest_dir/* > /dev/null 2>&1

# get the files off the servers and place them in the destination folder in its own folder

index=0
while read line; do
  server_array[index]="$line"
    for server in ${server_array[*]}
    do
    # create the folders named per server
        mkdir -p $dest_dir/$server
          #find the correct files
          index2=0
         while read line2; do
           file_array[index2]="$line2"
           for file in ${file_array[*]}
            do
            #copy the files to the folders
           sshpass -f $ssh_pass scp -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -rpqv $ssh_user@$server:$file $dest_dir/$server
           done
          done < $home_dir/files
    done

done < $home_dir/test_server_names
# change the directory and run SED on the files and folders
cd $dest_dir || exit 1

find ./ -type f -print0 | xargs -0 sed -Ei 's/([^&]password=)([^\t "]*)/\1xxxxxx/gI; s/(password.*>).*(<)/\1xxxxxx\2/gI; s/(password" value=").[^"]*/\1xxxxxx/gI; s/=admin:.[^\t ]*/=admin:xxxxxx/g; /"Password">/ {n;N;s/([\t ]*).*(\n.*<\/)/\1xxxxxx\2/gI}'

#zip the folders into one zip file

zip -rq9 properties.zip *


else
	echo "Cannot change directory to delete old files! Aborting! " 1>&2 >> $logfile
	exit 1
fi



# open the folder with contents

xdg-open $dest_dir

cornelvis

View Public Profile for cornelvis

Visit cornelvis's homepage!

Find all posts by cornelvis

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Script works fine until I | more

Hello all, This beats me. I have a script that executes some commands and redirects their output to some text files that I will parse. The commands are along the lines of: dsmadmc -id=admin -pa=admin -outfile=/home/tools/qlog.txt q log f=d If I just run the script it works. If I execute...

2. Shell Programming and Scripting

perl - how come this script works?

#!/usr/bin/perl open (DATA, file.txt); @array = <DATA>; close (DATA); open (DATA, ">$file.txt"); for (@array) { s/text/replace text/; push(@contents,$_); } seek(DATA, 0, 0); print DATA (@contents); close(DATA); could someone please explain how this works. i've been...

3. UNIX for Advanced & Expert Users

Country Codes script faster response ;please help

Dear all I have group of input lines which look like this These input lines is placed in a file named phonelines.txt and there is a script which match $4 and $5 with country codes placed in another file named country-codes.txt and its contents is : Italy 39 Libyana 21892 Thuraya...

4. Shell Programming and Scripting

Can anyone make this script run faster?

One of our servers runs Solaris 8 and does not have "ls -lh" as a valid command. I wrote the following script to make the ls output easier to read and emulate "ls -lh" functionality. The script works, but it is slow when executed on a directory that contains a large number of files. Can anyone make...

5. Shell Programming and Scripting

Script works with bash 3.0 but not 3.2.

Hello, So my knowledge of bash scripting is not that great and I have been trying to solve this problem on my own for awhile to no avail. Here's the error I get when running it with an OS that uses bash 3.2.x: testagain.sh: line 10: *-1: syntax error: operand expected (error token is...

6. Shell Programming and Scripting

Script to parse a file faster

My example file is as given below: conn=1 uid=oracle conn=2 uid=db2 conn=3 uid=oracle conn=4 uid=hash conn=5 uid=skher conn=6 uid=oracle conn=7 uid=mpalkar conn=8 uid=anarke conn=1 op=-1 msgId=-1 - fd=104 slot=104 LDAPS connection from 10.10.5.6 to 10.18.6.5 conn=2 op=-1 msgId=-1 -...

7. Shell Programming and Scripting

Make script faster

Hi all, In bash scripting, I use to read files: cat $file | while read line; do ... doneHowever, it's a very slow way to read file line by line. E.g. In a file that has 3 columns, and less than 400 rows, like this: I run next script: cat $line | while read line; do ## Reads each...

8. Shell Programming and Scripting

Making script run faster

Can someone help me edit the below script to make it run faster? Shell: bash OS: Linux Red Hat The point of the script is to grab entire chunks of information that concerns the service "MEMORY_CHECK". For each chunk, the beginning starts with "service {", and ends with "}". I should...

9. Shell Programming and Scripting

Optimize shell script to run faster

data.file: contact { contact_name=royce-rolls modified_attributes=0 modified_host_attributes=0 modified_service_attributes=0 host_notification_period=24x7 service_notification_period=24x7 last_host_notification=0 last_service_notification=0 host_notifications_enabled=1...

10. Shell Programming and Scripting

Shell script works fine as a standalone script but not as part of a bigger script

Hello all, I am facing a weird issue while executing a code below - #!/bin/bash cd /wload/baot/home/baotasa0/sandboxes_finance/ext_ukba_bde/pset sh UKBA_publish.sh UKBA 28082015 3 if then echo "Param file conversion for all the areas are completed, please check in your home directory"...

LEARN ABOUT OPENSOLARIS

refile

REFILE(1)                                                            [nmh-1.5]                                                           REFILE(1)

NAME

       refile - file message in other folders

SYNOPSIS

       refile [msgs] [-draft] [-link | -nolink] [-preserve | -nopreserve] [-unlink | -nounlink] [-src +folder] [-file file] [-rmmproc program]
            [-normmproc] +folder1 ...  [-version] [-help]

DESCRIPTION

       Refile moves (see mv(1)) or links (see ln(1)) messages from a source folder into one or more destination folders.

       If you think of a message as a sheet of paper, this operation is not unlike filing the sheet of paper (or copies) in file cabinet  folders.
       When  a  message  is  filed,  it is linked into the destination folder(s) if possible, and is copied otherwise.  As long as the destination
       folders are all on the same file system, multiple filing causes little storage overhead.  This facility provides a good way  to  cross-file
       or multiply-index messages.  For example, if a message is received from Jones about the ARPA Map Project, the command

            refile cur +jones +Map

       would allow the message to be found in either of the two folders `jones' or `Map'.

       You may specify the source folder using -src +folder.  If this is not given, the current folder is used by default.  If no message is spec-
       ified, then `cur' is used by default.

       The option -file file directs refile to use the specified file as the source message to be filed, rather than  a  message  from  a  folder.
       Note  that  the file should be a validly formatted message, just like any other nmh message.  It should NOT be in mail drop format (to con-
       vert a file in mail drop format to a folder of nmh messages, see inc(1)).

       If a destination folder doesn't exist, refile will ask if you want to create it.  A negative response will abort the  file  operation.   If
       the  standard  input  for refile is not a tty, then refile will not ask any questions and will proceed as if the user answered "yes" to all
       questions.

       The option -link preserves the source folder copy of the message (i.e., it does a  ln(1)  rather  than  a  mv(1)),  whereas,  -nolink  (the
       default) deletes the filed messages from the source folder.

       Normally  when  a  message is refiled, for each destination folder it is assigned the number which is one above the current highest message
       number in that folder.  Use of the -preserv switch will override this message renaming, and try to preserve the number of the message.   If
       a  conflict for a particular folder occurs when using the -preserve switch, then refile will use the next available message number which is
       above the message number you wish to preserve.

       If -link is not specified (or -nolink is specified), the filed messages will be removed from the source folder.  The default is  to  remove
       these  messages  by  renaming  them with a site-dependent prefix (usually a comma).  Such files will then need to be removed in some manner
       after a certain amount of time.  Many sites arrange for cron to remove these files once a day, so check with your system administrator.

       Alternately, if you wish for refile to really remove the files representing these messages from the source folder, you can use the  -unlink
       switch (not to be confused with the -link switch).  But messages removed by this method cannot be later recovered.

       If  you prefer a more sophisticated method of `removing' the messages from the source folder, you can define the rmmproc profile component.
       For example, you can add a profile component such as

            rmmproc:  /home/coleman/bin/rmm_msgs

       then refile will instead call the named program or script to handle the message files.

       The user may specify -rmmproc program on the command line to override this profile specification.  The -normmproc option forces the message
       files to be deleted by renaming or unlinking them as described above.

       The -draft switch tells refile to file the <mh-dir>/draft.

FILES

       $HOME/.mh_profile          The user profile

PROFILE COMPONENTS

       Path:                To determine the user's nmh directory
       Current-Folder:      To find the default current folder
       Folder-Protect:      To set mode when creating a new folder
       rmmproc:             Program to delete the message

SEE ALSO

       folder(1), rmf(1), rmm(1)

DEFAULTS

       `-src +folder' defaults to the current folder
       `msgs' defaults to cur
       `-nolink'
       `-nounlink'
       `-nopreserve'

CONTEXT

       If  -src  +folder  is given, it will become the current folder.  If neither -link nor `all' is specified, the current message in the source
       folder will be set to the last message specified; otherwise, the current message won't be changed.

       If the "Previous-Sequence" profile entry is set, in addition to defining the named sequences from  the  source  folder,  refile  will  also
       define those sequences for the destination folders.  See mh-sequence (5) for information concerning the previous sequence.

BUGS

       Since refile uses your rmmproc to delete the message, the rmmproc must NOT call refile without specifying -normmproc, or you will create an
       infinite loop.

MH.6.8                                                             11 June 2012                                                          REFILE(1)