Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Associative array index question

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Associative array index question
# 1  
Old 09-27-2017
Associative array index question
I am trying to assign indexes to an associative array in a for loop but I have to use an eval command to make it work, this doesn't seem correct I don't have to do this with regular arrays

For example, the following assignment fails without the eval command:
Code:
#! /bin/bash

   read -d "\0" -a DN_ARRAY < <(ls -rlt /localbackup/ldap/ | awk '{ print $9 }' )
   declare -A DN_COUNT

   for NAME in "${DN_ARRAY[@]}" ; do
        eval DN_COUNT["$NAME"]=$(grep -E "dn: " /localbackup/ldap/${NAME} | wc -l)
   done

Why is that?
# 2  
Old 09-27-2017
I just encountered this behavior yesterday. ls on Linux now injects single quotes around filenames with spaces in them, for your convenience! Ugh! Eval or xargs would swallow those, anything else would take them literally.

Fortunately, you don't need ls here and never did... You can just do for FILE in /localbackup/ldap/* to cut out the middleman. This is completely safe even for file names with spaces and special characters.

Also, you don't need wc's help to count matching lines with grep.

Code:
for FILE in /localbackup/ldap/*
do
        NAME=$(basename "$FILE")
        DN_COUNT["$NAME"]=$(grep -c "dn: " "$FILE")
done

This User Gave Thanks to Corona688 For This Post:
Riker1204
# 3  
Old 09-27-2017
I'm afraid I don't understand the problem, except that your code is utmost cumbersome. Why do you use the read -a at all? Why the complicated "command substitution"? The -rlt options to ls?
Replicating above (but not using eval) with some of my data yields
Code:
declare -A DN_COUNT
for NAME in fi* ; do DN_COUNT["$NAME"]=$(grep -Ec "ron|ime|ava" $NAME ); done
echo ${!DN_COUNT[@]} ${DN_COUNT[@]}
file3~ file2~ file file1~ file4 file2 file3 file1 file4~ 0 0 0 0 9 0 2 0 9

This User Gave Thanks to RudiC For This Post:
Riker1204
# 4  
Old 09-27-2017
Actually, how about this?

Code:
grep -Hc "dn: " /localbackup/ldap/*

Will output
Code:
file1:count
file2:count
fle3:count
...

All in one efficient command.
Last edited by Corona688; 09-29-2017 at 04:40 PM.. Reason: -H, not -h
These 2 Users Gave Thanks to Corona688 For This Post:
rbatte1 Riker1204
# 5  
Old 09-27-2017
Guys this forum never ceases to amaze me. Great suggestions all around. @RudiC, I used that method to hopefully extract just the file names based on the age. ( Looking at my code now and it indeed looks slightly retarded. I thought it was brilliant 6 hours ago....)

Corona688 I will actually be using your suggestion. I just started working as a Linux admin two months ago, and I'm using Bash scripting now on a daily basis to automate things.

This was a snippet from an ldap recovery script. Anyways I will get better. Thanks. Marking as solved. But one thing still bothers me, I thought I understood the order of evaluation and it still doesn't make sense about the eval part. I will think about what you wrote Corona some more.

Thanks again!
# 6  
Old 09-28-2017
Quote:
Originally Posted by Riker1204
But one thing still bothers me, I thought I understood the order of evaluation and it still doesn't make sense about the eval part.
It's not about order of evaluation, it's about your filenames being mangled from passing them through a pointless pipe chain then cramming them into read -d "\0". The -d "\0" is especially pointless in this context, I suspect it was grabbed from some other script without understanding it.

I'm no longer certain of my original guess about how exactly they were mangled, but they must have been, in a way which eval would swallow. Nulls injected between records from -d "\0" maybe.

My question: How did it occur to you to use eval?
This User Gave Thanks to Corona688 For This Post:
Riker1204
# 7  
Old 09-28-2017
Eval occured to me because when I trace the output ( bash -x ) I get results like this: DN_COUNT[$NAME]=98 or DN_COUNT[$NAME]=1007 and so on. So of course eval then gives me the desired result. I'm not saying your wrong, but I just don't see how the mangling is occuring.

Also I did write the whole thing from scratch, which try not to laugh, I will post here. I didn't copy / paste anything. I tend to use the named pipes to feed with the null delimiter because I know C strings are terminated that way which allows for crazy characters (?Maybe I'm wrong there too!) and the named pipe always avoids subshells which tend to trip me up. It works fine now, but I'm sure you guys could reduce this to like 25% of its size! Not expecting you too to that, just complimenting. I just got done reading "classical shell scripting" and "pro bash programming", but I have no practical experience. Anyways here it is. I will be re-writing it with your guys suggestions. This is my VERY FIRST production script, so again, mea culpa. I will get better at this...

Code:
#! /bin/bash

    systemctl stop crond


    make_dir () {
        for i in "/var/lib/ldap" "/var/lib/ldap/accesslog" "/var/tmp/bdb-log" "/etc/openldap/slapd.d" ; do
            if [ -d "$i" ] ; then
                :
            else
                mkdir "$i"
            fi
        done

        if [ $? -eq 0 ] ; then
            printf "%b\n" "Checking directories:                         [ \e[92mOK\e[0m ]"
        else
            printf "%b\n" "Missing directories, unable to create them:   [ \e[91mFAILED\e[0m ]"
        fi
        cp /usr/share/openldap-servers/DB_CONFIG.example /var/lib/ldap/DB_CONFIG
    }


    set_permissions () {
        for i in "/var/lib/ldap" "/var/tmp/bdb-log" "/etc/openldap/slapd.d" ; do
            chown -R ldap:ldap "$i"
        done

        if [ $? -eq 0 ] ; then
            printf "%b\n" "Setting permissions:                          [ \e[92mOK\e[0m ]"
            return 0
        else
            printf "%b\n" "Unable to set permissions:                    [ \e[91mFAILED\e[0m ]"
            return 1
        fi
    }


    stop_ldap () {
        systemctl stop slapd
        pid_number=$(pgrep slapd)
        if [ $? -eq 0 ] ; then
            kill -9 $pid_number
        else
            :
        fi
    }


    start_ldap () {
        systemctl start slapd
        if [ $? -eq 0 ] && pgrep slapd ; then
            printf "%b\n" "slapd started:                                [ \e[92mOK\e[0m ]"
            return 0
        else
            printf "%b\n" "Unable to start slapd:                        [ \e[91mFAILED\e[0m ]"
            return 1
        fi
    }


    newest_ldif () {
       local_ldif_count="$(find /localbackup/ldap -name "*.ldif" | wc -l)"
       remote_ldif_count="$(find /backup/ldap -name "*.ldif" | wc -l)"
       if [ -d "/localbackup/ldap" ] && [ "$local_ldif_count" -ge 2 ] ; then
           readarray ldif_array < <(ls -lrt /localbackup/ldap/ )
           total=${#ldif_array[@]}
           NEWEST_LDIF="$(printf "%s" "${ldif_array[total-1]}" | awk '{ print $9 }' )"
           printf "%b%b%b\n" "Found local ldif: " "${NEWEST_LDIF}"  "       [ \e[92mOK\e[0m ]"
           PATH_TO_FILE="/localbackup/ldap/"
           LOCAL_NUMBER_OF_DN=$(grep -E "dn: " /localbackup/ldap/${NEWEST_LDIF} | wc -l )
           printf "%s\n" "Selected LDIF has ${LOCAL_NUMBER_OF_DN} DNs"
       elif [ -d "/backup/ldap" ] && [ "$remote_ldif_count" -ge 1 ] ; then
           printf "%b\n" "Local ldif not found checking NFS server:     [ \e[93mWARNING\e[0m ]"
           readarray ldif_array < <(ls -lrt /backup/ldap/ )
           total=${#ldif_array[@]}
           NEWEST_LDIF="$(printf "%s" "${ldif_array[total-1]}" | awk '{ print $9 }' )"
           printf "%b%b%b\n" "Found remote ldif: " "${NEWEST_LDIF}"  "          [ \e[92mOK\e[0m ]"
           PATH_TO_FILE="/backup/ldap/"
           REMOTE_NUMBER_OF_DN=$(grep -E "dn: " /backup/ldap/${NEWEST_LDIF} | wc -l )
           printf "%s\n" "Selected LDIF has ${REMOTE_NUMBER_OF_DN} DNs"
       else
          printf "%b\n" "Can not find an ldif, exiting...                [ \e[91mFAILED\e[0m ]"
          sleep 5s
          exit 1
       fi
    }


    dn_number () {
       LDIF_LOCAL_NUMBER=$(find /localbackup/ldap -name "*.ldif" 2> /dev/null | wc -l)
       LDIF_REMOTE_NUMBER=$(find /backup/ldap -name "*.ldif" 2> /dev/null | wc -l )

       read -d "\0" -a DN_ARRAY < <(ls -rlt /localbackup/ldap/ | awk '{ print $9 }' )

       if [ -d /localbackup/ldap ] && [ $LDIF_LOCAL_NUMBER -ge 1 ]
       then
           DN_PATH="/localbackup/ldap/"
       elif [ -d /backup/ldap ] && [ $LDIF_REMOTE_NUMBER -ge 1 ]
       then
           DN_PATH="/backup/ldap/"
       else
           printf "%s\n" "Can't find an LDIF file, have to exit...."
           exit 1
       fi

       printf "%s\n" "The following LDIFs have the highest DN counts:"


       for k in "${DN_ARRAY[@]}"
       do
           NUMBER_OF_DN=$(grep -E "dn: " ${DN_PATH}${k} | wc -l)
           printf "%s%s\n" "${k}:  " "$NUMBER_OF_DN"
       done |
       sort -rn -k 2 | head -n 5
       sleep 10s
    }

    case $1 in
    repair)    repair=true
               ;;
    testdb)    testdb=true
               ;;
    restore)   restore=true
               ;;
    rebuild)   rebuild=true
               ;;
    *)         printf "%s\n\n" "Usage ldap.sh <repair|restore|rebuild|test>"
               printf "%s\n"   "repair     ---> try to repair existing db via db_recover"
               printf "%s\n"   "restore    ---> try to restore from latest backup via slapadd"
               printf "%s\n"   "rebuild    ---> rebuild everything and restore latest backup"
               printf "%s\n\n" "testdb     ---> test service health; return a search"
               exit 1
               ;;
    esac

    if [ "$repair" = true ] ; then
        stop_ldap
        db_recover -v -f -h /var/lib/ldap
        set_permissions
        if [ $? -eq 0 ] ; then
            printf "%b\n" "db recovery complete:                         [ \e[92mOK\e[0m ]"
        else
            printf "%b\n" "db recovery could not complete:               [ \e[91mFAILED\e[0m ]"
        fi
        start_ldap

    fi

    if [ "$restore" = true ] ; then
        stop_ldap
        printf "%s\n" "backing up db directory"
        tar czf /localbackup/ldap-db-backup.tar /var/lib/ldap
        rm -rf /var/lib/ldap/*
        rm -rf /etc/openldap/slapd.d/*
        rm -rf /var/tmp/bdb-log/*
        make_dir
        set_permissions
        slaptest -f /localbackup/slapd.conf -F /etc/openldap/slapd.d
        set_permissions
        start_ldap
        sleep 5s
        stop_ldap
        newest_ldif
        dn_number
        slapadd -n 1 -l ${PATH_TO_FILE}${NEWEST_LDIF}
        if [ $? -eq 0 ] ; then
            printf "%b\n" "ldif import successful:                       [ \e[92mOK\e[0m ]"
        else
            printf "%b\n" "ldif import was not successful:               [ \e[91mFAILED\e[0m ]"
        fi
        set_permissions
        start_ldap
    fi

    if [ "$rebuild" = true ] ; then
        stop_ldap
        printf "%s" "backing up database"
        tar czf /localbackup/ldap-db-backup.tar /var/lib/ldap
        yum -y remove openldap-servers
        yum -y install openldap-servers
        rm -rf /var/lib/ldap/*
        rm -rf /etc/openldap/slapd.d/*
        rm -rf /var/tmp/bdb-log/*
        make_dir
        set_permissions
        slaptest -f /localbackup/slapd.conf -F /etc/openldap/slapd.d
        set_permissions
        start_ldap
        sleep 5s
        stop_ldap
        newest_ldif
        dn_number
        slapadd -n 1 -l ${PATH_TO_FILE}${NEWEST_LDIF}
        if [ $? -eq 0 ] ; then
            printf "%b\n" "ldif import successful:                       [ \e[92mOK\e[0m ]"
        else
            printf "%b\n" "ldif import was not successful:               [ \e[91mFAILED\e[0m ]"
        fi
        set_permissions
        start_ldap
      fi

    if [ "$testdb" = true ] ; then
        for i in 'cn=Standard' ; do
           if ldapsearch -h localhost -p 389 -D "cn=root,dc=removed,dc=com" -w 'removed' -b "dc=removed" \
               "(${i})" 2> /dev/null | grep -E "numEntries: 1"
           then
               printf "%b\n" "found ${i}:                                    [ \e[92mOK\e[0m ]"
               ldapsearch -h localhost -p 389 -D "cn=root,dc=removeddc=com" -w 'removed -b "dc=celeritas,dc=com" \
               "(${i})" | grep -E "(dn: )|(cn: )|(uid: )"
           elif ldapsearch -h localhost -p 389 -D "cn=root,dc=removed,dc=com" -w 'removed -b "dc=removed" \
                "(${i})" 2>&1 | grep -E "Can't contact LDAP server"
           then
               printf "%b\n" "Can not find ${i}:                             [ \e[91mFAILED\e[0m ]"
           fi
        done
    fi

    systemctl start crond

And I wan't to say thanks again for your help and patience.
Last edited by Riker1204; 09-28-2017 at 10:06 PM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Using associative array for comparison

Hello together, i make something wrong... I want an array that contains information to associate it for further processing. Here is something from my bash... You will know, what I'm trying to do. I have to point out in advance, that the variable $SYSOS is changing and not as static as in my... (2 Replies)
Discussion started by: Decstasy
2 Replies

2. Shell Programming and Scripting

Morse Code with Associative Array

Continuing my quest to learn BASH, Bourne, Awk, Grep, etc. on my own through the use of a few books. I've come to an exercise that has me absolutely stumped. The specifics: 1. Using ONLY BASH scripting commands (not sed, awk, etc.), write a script to convert a string on the command line to... (22 Replies)
Discussion started by: ksmarine1980
22 Replies

3. Shell Programming and Scripting

Associative Array with more than one item per entry

Hi all I have a problem where i have a large list ( up to 1000 of items) and need to have 2 items pulled from it into variables in a bash script my list is like the following and I could have it as an array or possibly an external text file maintained separately. Every line is different and... (6 Replies)
Discussion started by: kcpoole
6 Replies

4. Shell Programming and Scripting

Associative array

I have an associative array named table declare -A table table="fruit" table="veggie" table="GT" table="eminem" Now say I have a variable returning the value highway How do I find corresponding value GT ?? (this value that I find (GT in this case) is supposed to be the name of a mysql... (1 Reply)
Discussion started by: leghorn
1 Replies

5. Shell Programming and Scripting

Split string into map (Associative Array)

Hi Input: { committed = 782958592; init = 805306368; max = 1051394048; used = 63456712; } Result: A map (maybe Associative Array) where I can iterate through the key/value. Something like this: for key in $map do echo key=$key value=$map done Sample output from the map: ... (2 Replies)
Discussion started by: chitech
2 Replies

6. Shell Programming and Scripting

Help needed on Associative array in awk

Hi All, I got stuck up with shell script where i use awk. The scenario which i am working on is as below. I have a file text.txt with contents COL1 COL2 COL3 COL4 1 A 500 400 1 B 500 400 1 A 500 200 2 A 290 300 2 B 290 280 3 C 100 100 I could able to sum col 3 and col4 based on... (3 Replies)
Discussion started by: imsularif
3 Replies

7. UNIX for Dummies Questions & Answers

wh inode index starts from 1 unlike array index (0)

brothers why inode index starts from 1 unlike array inex which starts from 0 its a question from the design of unix operating system of maurice j.bach i need to know the answer urgently...someone help please (1 Reply)
Discussion started by: sairamdevotee
1 Replies

8. Filesystems, Disks and Memory

why the inode index of file system starts from 1 unlike array index(0)

why do inode indices starts from 1 unlike array indexes which starts from 0 its a question from "the design of unix operating system" of maurice j bach id be glad if i get to know the answer quickly :) (0 Replies)
Discussion started by: sairamdevotee
0 Replies

9. Shell Programming and Scripting

Perl: Sorting an associative array

Hi, When using sort on an associative array: foreach $key (sort(keys(%opalfabet))){ $value = $opalfabet{$key}; $result .= $value; } How does it handle double values? It seems to me that it removes them, is that true? If so, is there a way to get... (2 Replies)
Discussion started by: tine
2 Replies

10. Shell Programming and Scripting

Associative Array

Hi, I am trying to make an associative array to use in a popup_menu on a website. Here is what i have: foreach $entr ( @entries ) { $temp_uid = $entr->get_value(uid); $temp_naam = $entr->get_value(sn); $s++; } This is the popup_menu i want to use it in. popup_menu(-name=>'modcon',... (4 Replies)
Discussion started by: tine
4 Replies
Login or Register to Ask a Question