Bash to append array value to file before copying


 
Thread Tools Search this Thread
Top Forums UNIX for Beginners Questions & Answers Bash to append array value to file before copying
# 1  
Old 10-15-2019
Bash to append array value to file before copying

The bash stores each uniqueid in an array and then passes them to %q to get the unique path. That seems to work what I am having trouble with is renaming each .png with the unique value in %q. I thought it was working but upon closer inspection, a .png file is being sent to scp.... but only 1 and with the wrong uniqueid. It seems like the first .png is being used by scp, but with the last uniqueid. In this example there are 2, but there may be more or less. Thank you Smilie.


Code:
printf -v cmd_q '(cd /path/to/%q*/*/*/folder && exec sshpass -f file.txt scp -- *.png* xxx@xxx.xx.xx.xxx:'"$dir"'/folder/%q*-data.png)\n' "${array[@]}"
sshpass -f out.txt ssh -o strictHostKeyChecking=no -t xxx@xxx.xx.xx.xx "$cmd_q"

array
Code:
uniqueid1   ---- this is stored in array ----
data.png    ---- this is the file in the path ----

uniqueid2   ---- this is stored in array ----
data.png    ---- this is the file in the path ---

current
Code:
uniqueid1-data.png (but this is really the first png just with the wrong name)

file transfered by scp
Code:
uniqueid1-data.png
uniqueid2-data.png


Last edited by cmccabe; 10-15-2019 at 11:40 AM.. Reason: fixed format
# 2  
Old 10-15-2019
This is a rather roundabout way of sending files! sshpass executing sshpass executing scp. Isn't there a better way?

Anyway, this doesn't do what you think. Each ID only gets used once. The first statement will contain the first and second ID; the second contains the third and fourth. Et cetera.

Quick fix:
Code:
for ID in "${array[@]}"
do
        echo "( cd /path/to/${ID}*/*/*/folder && exec sshpass -f file.txt scp -- *.png* xxx@xxx.xx.xx.xxx:'$dir'/folder/${ID}*-data.png )"
done 
# Test to make sure the output is what you want.  Then replace the above line with the below one.
# done | sshpass -f out.txt ssh -o strictHostKeyChecking=no -t xxx@xxx.xx.xx.xx

This User Gave Thanks to Corona688 For This Post:
# 3  
Old 10-16-2019
With the echo the output is correct, but with sshpass added I get the file not foung though I can manually finnd the directory but the .png that is there is cn_results and when it is transferred by scp it is renamed or has $ID append to it. Thank you Smilie.

echo --- looks good ---
Code:
( cd /path/to/ID1*/*/folder && exec sshpass -f file.txt scp -- *.png* xxx@xxx.xx.xx.xxx:path/to/destination/ID1-cn_results.png )
( cd /path/to/ID2*/*/folder && exec sshpass -f file.txt scp -- *.png* xxx@xxx.xx.xx.xxx:path/to/destination/ID2-cn_results.png )

echo removed and done |.... added
Code:
script.sh: line 39: ( cd /path/to/ID1*/*/folder && exec sshpass -f file.txt scp -- *.png* xxx@xxx.xx.xx.xxx:path/to/destination/ID1-cn_results.png ): No such file or directory
script.sh: line 39: ( cd /path/to/ID2*/*/folder && exec sshpass -f file.txt scp -- *.png* xxx@xxx.xx.xx.xxx:path/to/destination/ID2-cn_results.png ): No such file or directory

I tried to add the loop to scp and rename as such:

Code:
scp "$ID" xxx@xxx.xx.xx.xxx:path/to/destination/"${ID%.png}"-cn_results.png )

but I get the same result.

Last edited by cmccabe; 10-16-2019 at 10:04 AM..
# 4  
Old 10-16-2019
Show your code.

Always. Show your code.
This User Gave Thanks to Corona688 For This Post:
# 5  
Old 10-16-2019
I tried it two ways both resulting in file or directory not found, but I can manually ssh to them. This is not the full code but I hope it's enough as I don't want to post code that is not needed. Thank you Smilie.

attempt 1
Code:
# match unique run in list and extract samples
run=$(awk -F '\n' -v RS="" -v ref="$s5" '$0 ~ ref {d=split($0, val, " "); for(i=2;i<d;i+=2) printf "%s ",val[i]; printf "\n"}' list)

# create array of samples from $run
readarray -t array <<< "$(printf '%s\n' $run)"
for ((i=0; i<${#array[@]}; i++ )) ; do
    echo "${array[$i]}"
done

# loop through array and append with ID
for ID in "${array[@]}" ; do
      "( cd /path/to/${ID}*/*/folder && exec sshpass -f file.txt scp xxx@xxx.xx.xx.xxx:path/to/destination/${ID}-cn_results.png )"
done | sshpass -f out.txt ssh -o strictHostKeyChecking=no -t xxxx@xxx.xx.xxx.xx

attempt 2
Code:
# match unique run in list and extract samples
run=$(awk -F '\n' -v RS="" -v ref="$s5" '$0 ~ ref {d=split($0, val, " "); for(i=2;i<d;i+=2) printf "%s ",val[i]; printf "\n"}' list)

# create array of samples from $run
readarray -t array <<< "$(printf '%s\n' $run)"
for ((i=0; i<${#array[@]}; i++ )) ; do
    echo "${array[$i]}"
done

# loop through array and append with ID
for ID in "${array[@]}" ; do
      "( cd /path/to/${ID}*/*/folder && exec sshpass -f file.txt scp "$ID" xxx@xxx.xx.xx.xxx:path/to/destination/"${ID%}"-cn_results.png )"
done | sshpass -f out.txt ssh -o strictHostKeyChecking=no -t xxxx@xxx.xx.xxx.xx

attempt 1 error:
Code:
script.sh: line 39: ( cd /path/to/ID1*/*/folder && exec sshpass -f file.txt scp -- *.png* xxx@xxx.xx.xx.xxx:path/to/destination/ID1-cn_results.png ): No such file or directory
script.sh: line 39: ( cd /path/to/ID2*/*/folder && exec sshpass -f file.txt scp -- *.png* xxx@xxx.xx.xx.xxx:path/to/destination/ID2-cn_results.png ): No such file or directory

attempt 2 error: --- in the server these exsist as ID1-xxxx-xxxxxxx-xxxxxx so I added * but thats not expanding it seems ----
Code:
ID1: No such file or directory
ID2: No such file or directory


Last edited by cmccabe; 10-16-2019 at 01:40 PM.. Reason: fixed format
# 6  
Old 10-16-2019
The echo you removed was necessary. Without it, you are attempting to run a file named "( cd /path/to/${ID}*/*/folder && exec sshpass -f file.txt scp xxx@xxx.xx.xx.xxx:path/to/destinatio...cn_results.png )" which quite probably does not exist.

Code:
rm -f /tmp/testoutput
for ID in "${array[@]}" ; do
      echo "( cd /path/to/${ID}*/*/folder && exec sshpass -f file.txt scp xxx@xxx.xx.xx.xxx:path/to/destination/${ID}-cn_results.png )"
done | tee /tmp/testoutput | sshpass -f out.txt ssh -o strictHostKeyChecking=no -t xxxx@xxx.xx.xxx.xx

The tee is optional but may help you by duplicating the output in /tmp/testoutput for you to check. * won't expand in /tmp/testoutput but ought to do so on the remote host.

Last edited by Corona688; 10-16-2019 at 02:49 PM..
This User Gave Thanks to Corona688 For This Post:
# 7  
Old 10-17-2019
So if I understand and am reading the error correctly the * is not expanding on the remote machine? I think this because tee looks good but the terminal doesn't like the *. I am testing this on ubuntu but will use it on centos. Thank you Smilie.


Code:
for ID in "${array[@]}" ; do
      echo "( cd /path/to/${ID}*/*/folder && exec sshpass -f file.txt scp xxx@xxx.xx.xx.xxx:path/to/destination/${ID}-cn_results.png )"
done | tee /tmp/tmp.txt | sshpass -f out.txt ssh -o strictHostKeyChecking=no -t xxxx@xxx.xx.xxx.xx

terminal output
Code:
-bash: line 1: cd: /path/to/ID1*/*/folder: No such file or directory
-bash: line 2: cd: /path/to/ID2*/*/folder: No such file or directory

tee ouput in tmp.txt
Code:
( cd /path/to/ID1*/*/folder && exec sshpass -f file.txt scp xxx@xxx.xx.xx.xxx:path/to/destination/ID1-cn_results.png )
( cd /path/to/ID2*/*/folder && exec sshpass -f file.txt scp xxx@xxx.xx.xx.xxx:path/to/destination/ID2-cn_results.png )

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. OS X (Apple)

Create a bash array from a flat file of whitespaces only.

Hi guys and gals... MacBook Pro. OSX 10.13.2, default bash terminal. I have a flat file 1920 bytes in size of whitespaces only. I need to put every single whitespace character into a bash array cell. Below are two methods that work, but both are seriously ugly. The first one requires that I... (7 Replies)
Discussion started by: wisecracker
7 Replies

2. Shell Programming and Scripting

Bash append values if keywords are present in the file

Hi Team, i have a web ui where user will be passing values and the output will be saved to a file say test with the following contents . These below mentioned values will change according to the user_input Just gave here one example Contents of file test is given below Gateway... (7 Replies)
Discussion started by: venkitesh
7 Replies

3. Shell Programming and Scripting

Ftp with bash, append file where left off

I'm working on a bash script to finish uploading a file. I need a way to get $filesize so that "restart $filesize" will work. Here is my script: ftp -n -v <<END_SCRIPT open ftp.$domain user $user@$domain $password size $file restart $filesize put $file quit END_SCRIPTWayne Sallee... (9 Replies)
Discussion started by: WayneSallee
9 Replies

4. Shell Programming and Scripting

Append awk results into file or array

for a in {1..100} do awk '{ sum+=$a} END {print sum}' a=$a file1 > file2 done I know I will get only one number if following the code above, how can I get 100 sum numbers in file2? (2 Replies)
Discussion started by: wanliushao
2 Replies

5. UNIX for Dummies Questions & Answers

Passing values from file into array in Bash

Hi, I'm trying to write a bash script that takes a file and passes each line from the file into an array with elements separated by column. For example: Sample file "file1.txt": 1 name1 a first 2 name2 b second 3 name3 c third and have arrays such as: line1 = ( "1" "name1" "a"... (3 Replies)
Discussion started by: ShiGua
3 Replies

6. Shell Programming and Scripting

Bash Script Issues (If statement for file copying)

Writing a bash script for use with Geektool, pulls the battery info, and shuffles images around so that an Image geeklet can display the correct expression as the desktop background. (Eventually I intend to make it more intricate, based on more variables, and add more expressions) I'm extremely... (1 Reply)
Discussion started by: The_Ardly374
1 Replies

7. Shell Programming and Scripting

Bash Scripting - How to grep a file into an array

I have figured out how to grep the file like this: echo `grep $(date +'%Y-%m-%d') Cos-01.csv | cut -d',' -f1` The above line does echo the correct information from the lines in which my search criteria is found. Now I am trying to get that information (Yes, just one column of every line) into... (6 Replies)
Discussion started by: TwelveDays
6 Replies

8. Shell Programming and Scripting

Noob's 1st...bash-script for copying one file into many

I have one file "file.a.b.c-d.r" that I would like to use to spawn 4 other files: "file.a.b.1-A.r" "file.a.b.1-B.r" "file.a.b.1-C.r" "file.a.b.1-D.r" where the field "c-d" changes into my 1 and A-D. I was doing this manually at the prompt with > cp "file.a.b.c-d.r" "file.a.b.1-A.r" >... (13 Replies)
Discussion started by: WSUToad
13 Replies

9. Shell Programming and Scripting

Most reliable way to store file contents in an array in bash

Hi Guys, I have a file which has numbers in it separated by newlines as follows: 1.113 1.456 0.556 0.021 -0.541 -0.444 I am using the following code to store these in an array in bash: FILE14=data.txt ARRAY14=(`awk '{print}' $FILE14`) (6 Replies)
Discussion started by: npatwardhan
6 Replies

10. Shell Programming and Scripting

[bash] redirect (save) array to a file

Hi, I'm trying to write a function that redirects the contents of an array to a file. The array contains the lines of a data file with white space. The function seems to preserve all white space when redirected except that it seems to ignore newlines. As a consequence, the elements of the... (7 Replies)
Discussion started by: ASGR
7 Replies
Login or Register to Ask a Question