Shell script to apply functions to multiple columns dynamically
Hello,
I have a requirement to apply hashing algorithm on flat file on one or more columns dynamically based on header
sample input file
Code:
ID|NAME|AGE|GENDER
10|ABC|30|M
20|DEF|20|F
say if i want multiple columns based on the header example id,name or id,age or name,gender and hash and store the output value in a new column for each row
I am having hard time visualizing this
I have written something but i dont know if its going to work or not really lost here.
Code:
#!/usr/bin/sh
target="id" # here i m passing the header column to be md5 applied ideally this is parameterized and can send more than one column name seperated by comma "
target_idx= # the column number of that field
{
# reading header.
IFS=, read -r -a header
for idx in "${!header[@]}"; do # and look for the target field in it
[[ ${header[$idx]} = $target ]] && { target_idx=$idx; break; }
done
# then, iterate over lines
while IFS=, read -r -a line; do
a=echo-n ${line[$target_idx]}|md5sum # if my input in target is "id"
done
} <testfile.txt
echo$line|"$a"
say if i want multiple columns based on the header example id,name or id,age or name,gender and hash and store the output value in a new column for each row
If you want multiple columns you need to make the variable target_id and target_idx both arrays, no?
Then there is this:
Code:
# then, iterate over lines
while IFS=, read -r -a line; do
a=echo-n ${line[$target_idx]}|md5sum # if my input in target is "id"
done
} <testfile.txt
echo$line|"$a"
You iterate over the input lines, yes, but you produce output (the echo-line) only once: after you have read all the input. This must be wrong, no? Apart from the fact that after "echo" there needs to be a space (like after any command), but i suppose that was simply a typing error.
Also this line is not correct:
Code:
a=echo-n ${line[$target_idx]}|md5sum # if my input in target is "id"
It should perhaps read:
Code:
a=$(echo -n ${line[$target_idx]}|md5sum) # if my input in target is "id"
For parametrisation via the commandline i suggest to read the man page for getopts. I have just written an example in another thread you might want to use as a starting point.
I have to pass the input as arrays yes and i did make a typo near the echo
yes the echo at the end is outside the loop so it will only print out the last row i think
issue is even if i pass the target as variable the code is only picking the first value passed generating the md5 and outputting the md5 for that line. I am trying to figure out how can i loop through all the columns passed inside the array and concatenate them using, and then calculate their md5 and then write it back to the file as a new column.
I know its a strange requirement Please point me to the write direction
I m trying something like this with no avail
Code:
#!/usr/bin/sh
target=("id,name")
target_idx=() # the column number of that field
{
# reading header.
for i in $(echo $target | sed "s/,/ /g");do # trying to pass the comma seperated values inside target variable
IFS=, read -r -a header
for idx in "${!header[@]}"; do # and look for the target field in it
[[ ${header[$idx]} = $target ]] && { target_idx=$idx; break; }
done
done
# then, iterate over lines
while IFS=, read -r -a line; do
a=$(echo -n ${line[$target_idx]}|md5sum)
echo "$line"
done
}
Moderator's Comments:
edit by bakunin: i think you posted this to the wrong thread. Pushing it to the right one.
I have to pass the input as arrays yes and i did make a typo near the echo
yes the echo at the end is outside the loop so it will only print out the last row i think
issue is even if i pass the target as variable the code is only picking the first value passed generating the md5 and outputting the md5 for that line. I am trying to figure out how can i loop through all the columns passed inside the array and concatenate them using, and then calculate their md5 and then write it back to the file as a new column.
I know its a strange requirement Please point me to the write direction
I m trying something like this with no avail
Code:
#!/usr/bin/sh
target=("id,name")
target_idx=() # the column number of that field
{
# reading header.
for i in $(echo $target | sed "s/,/ /g");do # trying to pass the comma seperated values inside target variable
IFS=, read -r -a header
for idx in "${!header[@]}"; do # and look for the target field in it
[[ ${header[$idx]} = $target ]] && { target_idx=$idx; break; }
done
done
# then, iterate over lines
while IFS=, read -r -a line; do
a=$(echo -n ${line[$target_idx]}|md5sum)
echo "$line"
done
}
------ Post updated at 01:45 AM ------
Just a thought may be i should change my thinking. Just throwing it out i still did not figure out how to write the code but i may have a starting point
step one is if i can concatenate all the column names i send as a parameter to the script and then apply md5 function on it and then store the result of the output in a column in the end does this make any sense.
Please be aware that the md5sum of '10,abc' will NEVER be 73aca49763216fb96bbc2acef7b60afb as it is case sensitive.
Looks like you want a comma included. Try
Hi All,
i am trying to print required multiple columns dynamically from a fie.
But i am able to print only one column at a time.
i am new to shell script, please help me on this issue.
i am using below script
awk -v COLT=$1 '
NR==1 {
for (i=1; i<=NF; i++) {
... (2 Replies)
Hi ,
Can any please help the below requirement on all multiple servers and multiple dbs.
update configuration set value='yes' ;1)
the above statement apply on 31 Databases at a time on different Ip address
eg : 10.104.1.12 (unix ip address )
the above ip box contains 4 db's eg : db... (2 Replies)
Hi everyone,
I have a situation in which I have multiple (3 at last count) date columns in a CSV file (, delim), which need to be changed from:
January 1 2017 (note, no comma after day)
to:
YYYY-MM-DD
So far, I am able to convert a date using:
date --date="January 12, 1990" +%Y-%m-%d
... (7 Replies)
Hi all,
I have a PLSQL script which has a IN list where it takes some ids as input. For example
SELECT * FROM EMPLOYEE WHERE EMPLOYEE_ID IN (comma separated list )
I want to run this quest inside a shell script but I would like to prepare the IN list dynamically where the employee ids... (1 Reply)
Hi,
I am using ksh , i have requirement to run 4 functions in background , 4 functions call are available in a case that case is also in function, i need to execute 1st function it should run in background and return to case and next i will call 2nd function it should run in background and... (8 Replies)
Hallo,
I am working on a kdialog. This shall be able to load the required commands from a .conf file.
First step runs good by loading the entries (selectabel entries) in a variable:
MIRRORSELECT=$(kdialog --radiolist "Select your nearest mirror" $VAR1)
The kdialog is accordingly correct... (2 Replies)
I want to launch some shell scripts.
I would have the possibility to change the number of shell scripts launched dynamically by modifying a variable, or a configuration file.
For example, I start to launch 4 scripts at the same time, and after that, by modifying a variable, 6 scripts are... (0 Replies)
I've recently started using structures, but I am having problems in allocating the structure dynamically. In the code below if i allocate the structure in the main program it works fine, and i get the expected output. However if i use the function rper below to increase the size of the structure i... (0 Replies)
11-10-2018
