awk to create variables to pass into a bash loop to create a download link


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting awk to create variables to pass into a bash loop to create a download link
# 1  
Old 09-26-2016
awk to create variables to pass into a bash loop to create a download link

I have created one file that contains all the necessary info in it to create a download link. In each of the lines /results/analysis/output/Home/Auto_user_S5-00580-6-Medexome_67_032/plugin_out/FileExporter_out.67
/results/analysis/output/Home/Auto_user_S5-00580-4-Medexome_65_028/plugin_out/FileExporter_out.52
the _user_S5-00580-6-Medexome, the digit between the - before Medexome is variable, but that line will match a line in the file that begins with R_2016_09_20_10_12_41_user_S5-00580-6-Medexome. I need to read each of those two strings in variables and then use them to create a download link.

download link
Code:
http://xxx.xx.xxx.xxx  ---- hardcode
/output/Home/Auto_user_S5-00580-4-Medexome_65_028/plugin_out/FileExporter_out.52  -- from line1 with the /, /results/analysis is removed
/   --- harcode
R_2016_09_01_10_24_52_user_S5-00580-4-Medexome  -- from line R_
.tar.bz  --- hardcode

the --- are not part of the link just there for clarification i hope. Also, the

file
Code:
/results/analysis/output/Home/Auto_user_S5-00580-6-Medexome_67_032/plugin_out/FileExporter_out.67
/results/analysis/output/Home/Auto_user_S5-00580-4-Medexome_65_028/plugin_out/FileExporter_out.52
IonXpress_007 MEV21
IonXpress_008 MEV22
IonXpress_009 MEV23
R_2016_09_21_14_01_15_user_S5-00580-9-Medexome
IonXpress_001 MEC1
IonXpress_002 MEC32
IonXpress_003 MEC33
R_2016_09_21_11_26_19_user_S5-00580-8-Medexome
IonXpress_007 MEV37
IonXpress_008 MEV38
IonXpress_009 MEV39
R_2016_09_20_12_47_36_user_S5-00580-7-Medexome
IonXpress_004 MEV34
IonXpress_005 MEV35
IonXpress_006 MEV36
R_2016_09_20_10_12_41_user_S5-00580-6-Medexome
IonXpress_007 MEV45
IonXpress_008 MEV46
IonXpress_009 MEV47
R_2016_09_01_13_20_02_user_S5-00580-5-Medexome
IonXpress_004 MEV42
IonXpress_005 MEV43
IonXpress_006 MEV44
R_2016_09_01_10_24_52_user_S5-00580-4-Medexome
IonXpress_001 MEC1
IonXpress_002 MEV40
IonXpress_003 MEV41
R_2016_08_03_10_42_57_user_S5-00580-2-Medical_Exome

awk
Code:
awk {
function pA(arg, string) {
    string = arg
    sub(/_R_.*/, "", string)
    return string
}
print (u = $(i+1))
          sub(/.*_user_/, "", u)
          sub(/_.*/, "", u)
          sub(/^/, "user_", u)
          i += 2
            continue
}' file

The above was an attempt to create the two variable to read in a bash loop, but that had many errors. Is this possible or is there a better way? Thank you Smilie.
# 2  
Old 09-26-2016
From what you have shown us, I have absolutely no idea what two variables you are trying to create, what values you want to assign to either of those variables, nor how many pairs of values you expect to extract from your sample input file. (Note that the use of underlined text in your specification of the download link you want to create when there are other underscores in that specification is really confusing.)

Please clearly describe what you are trying to do and show us the output you want your script to produce.
This User Gave Thanks to Don Cragun For This Post:
# 3  
Old 09-26-2016
The two values are:

varA= /results/analysis/output/Home/Auto_user_S5-00580-6-Medexome_67_032/plugin_out/FileExporter_out.67

the digit between the - before Medexome is variable, but that line will match a line in the file that begins with

varB= R_2016_09_20_10_12_41_user_S5-00580-6-Medexome

Since _user_S5-00580-6-Medexome matches between those two lines

desired output
Code:
http://xxx.xx.xxx.xxx/output/Home/Auto_user_S5-00580-6-Medexome_67_032/plugin_out/FileExporter_out.67/R_2016_09_20_10_12_41_user_S5-00580-6-Medexome.tar.bz

Code:
 the (...) is not part of the download link only there to help see where the data comes from
http://xxx.xx.xxx.xxx   (harcoded)
varA or /output/Home/Auto_user_S5-00580-6-Medexome_67_032/plugin_out/FileExporter_out.67  [/results/analysis  is removed]
/    (harcoded)
varB or R_2016_09_20_10_12_41_user_S5-00580-6-Medexome
.tar.bz  (harcoded)

each download link is in the format above and consists of varA and varB

If each download link can be written to the same file, after the files are downloaded and moved, maybe I can create a check to see if the download exits in the directory. I hope this helps and thank you very much Smilie.

Last edited by cmccabe; 09-26-2016 at 11:04 PM.. Reason: fixed typo, added details
# 4  
Old 09-27-2016
Hello cmccabe,

Not completly sure about your requirements but could you please try following and let me know if this helps you.
Code:
awk -vvar1="/results/analysis/output/Home/Auto_user_S5-00580-6-Medexome_67_032/plugin_out/FileExporter_out.67" -vvar2="R_2016_09_20_10_12_41_user_S5-00580-6-Medexome" '{match(var1,/user_.*Medexome/);Q=substr(var1,RSTART,RLENGTH);match(var2,/user_.*Medexome/);W=substr(var2,RSTART,RLENGTH);if(Q==W){match(var1,/output.*/);print "http://xxx.xx.xxx.xxx/" substr(var1,RSTART,RLENGTH) "/" var2".tar.bz"}}'

Output will be as follows.
Code:
http://xxx.xx.xxx.xxx/output/Home/Auto_user_S5-00580-6-Medexome_67_032/plugin_out/FileExporter_out.67/R_2016_09_20_10_12_41_user_S5-00580-6-Medexome.tar.bz
http://xxx.xx.xxx.xxx/output/Home/Auto_user_S5-00580-6-Medexome_67_032/plugin_out/FileExporter_out.67/R_2016_09_20_10_12_41_user_S5-00580-6-Medexome.tar.bz
http://xxx.xx.xxx.xxx/output/Home/Auto_user_S5-00580-6-Medexome_67_032/plugin_out/FileExporter_out.67/R_2016_09_20_10_12_41_user_S5-00580-6-Medexome.tar.bz
http://xxx.xx.xxx.xxx/output/Home/Auto_user_S5-00580-6-Medexome_67_032/plugin_out/FileExporter_out.67/R_2016_09_20_10_12_41_user_S5-00580-6-Medexome.tar.bz
http://xxx.xx.xxx.xxx/output/Home/Auto_user_S5-00580-6-Medexome_67_032/plugin_out/FileExporter_out.67/R_2016_09_20_10_12_41_user_S5-00580-6-Medexome.tar.bz
http://xxx.xx.xxx.xxx/output/Home/Auto_user_S5-00580-6-Medexome_67_032/plugin_out/FileExporter_out.67/R_2016_09_20_10_12_41_user_S5-00580-6-Medexome.tar.bz

If above is not meeting your expectations, kindly provide more sample Input_file with expected sample output, with your all conditions too.
EDIT: Adding a non-one liner form of solution on same too.
Code:
awk -vvar1="/results/analysis/output/Home/Auto_user_S5-00580-6-Medexome_67_032/plugin_out/FileExporter_out.67" -vvar2="R_2016_09_20_10_12_41_user_S5-00580-6-Medexome" '{
	match(var1,/user_.*Medexome/);
	Q=substr(var1,RSTART,RLENGTH);
	match(var2,/user_.*Medexome/);
	W=substr(var2,RSTART,RLENGTH);
	if(Q==W){
			match(var1,/output.*/);
			print "http://xxx.xx.xxx.xxx/" substr(var1,RSTART,RLENGTH) "/" var2".tar.bz"
                }
 }
'   Input_file

Thanks,
R. Singh

Last edited by RavinderSingh13; 09-27-2016 at 01:12 AM.. Reason: Adding a non-one liner form of solution on same too.
This User Gave Thanks to RavinderSingh13 For This Post:
# 5  
Old 09-27-2016
The output.txt attached is from running the awk on the combine.txt file attached, which will be the input. It is very close but the two matching lines only need to be outputted. I hope this helps and thank you very much Smilie.

desired output (just the two matching lines from combine.txt in the same order)

Code:
http://172.24.188.111/output/Home/Auto_user_S5-00580-6-Medexome_67_032/plugin_out/FileExporter_out.67/R_2016_09_20_10_12_41_user_S5-00580-6-Medexome.tar.bz
http://172.24.188.111/output/Home/Auto_user_S5-00580-4-Medexome_65_028/plugin_out/FileExporter_out.52R_2016_09_01_10_24_52_user_S5-00580-4-Medexome.tar.bz


Last edited by cmccabe; 09-27-2016 at 09:53 AM.. Reason: added details
# 6  
Old 09-27-2016
Hello cmccabe,

It is not at all clear from attachments too. Please rephrase your requirements and come up with correctly phrased problem. I would like to request you to please spend sometime in your question and review it before posting it, once you have enough information into your post with proper sample Input_file and expected output_file with all conditions(most importantly) then you could post it.

Thanks,
R. Singh
This User Gave Thanks to RavinderSingh13 For This Post:
# 7  
Old 09-27-2016
I apologize for the confusion and hope I explain this better.

In combine.txt the first two lines are
Code:
 
 /results/analysis/output/Home/Auto_user_S5-00580-6-Medexome_67_032/plugin_out/FileExporter_out.67
/results/analysis/output/Home/Auto_user_S5-00580-4-Medexome_65_028/plugin_out/FileExporter_out.52

In each of those lines (in my real data there are several more but all follow the same format), the section in bold will match a line below in the file that looks like R_2016_09_20_10_12_41_user_S5-00580-6-Medexome

When a match is found then a download link is created from these variables along with some additional data.

Using the first line as an example:

varA= /results/analysis/output/Home/Auto_user_S5-00580-6-Medexome_67_032/plugin_out/FileExporter_out.67

and _user_S5-00580-6-Medexome from VarA matches varB= R_2016_09_20_10_12_41_user_S5-00580-6-Medexome

using these two variables VarA and varB a download link is made in the below format:
where:
Code:
 the (...) is not part of the download link only there to help see where the data comes from 
http://xxx.xx.xxx.xxx   (harcoded)
varA or /output/Home/Auto_user_S5-00580-6-Medexome_67_032/plugin_out/FileExporter_out.67  [/results/analysis  is removed]
/    (harcoded)
varB or R_2016_09_20_10_12_41_user_S5-00580-6-Medexome
.tar.bz  (harcoded)

The
Code:
awk

outputs one line multiple times, but only needs to output each match once.

Since the two lines in combine.txt with the/ have matches in the file their links are below:

Code:
http://172.24.188.111/output/Home/Auto_user_S5-00580-6-Medexome_67_032/plugin_out/FileExporter_out.67/R_2016_09_20_10_12_41_user_S5-00580-6-Medexome.tar.bz
http://172.24.188.111/output/Home/Auto_user_S5-00580-4-Medexome_65_028/plugin_out/FileExporter_out.52R_2016_09_01_10_24_52_user_S5-00580-4-Medexome.tar.bz

the digit between the- before Medexome is variable, but that line should match a line in combine.txt, if it does not then it is skipped. There could and often times will be more than one line with a / like at the top of combine.txt. Does this help? Thank you very much Smilie.

Last edited by cmccabe; 09-27-2016 at 12:19 PM.. Reason: fixed typo
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

awk to create link, download, and extract in sub-directory

The awk below will create sub-directories in a directory (which is always the last line of file1, each block separated by an empty line), if the number in line 2 (always the first 6 digits in the format xx-xxxx) of file2 is found in $2 of file1. This is the current awk output. If there is a... (0 Replies)
Discussion started by: cmccabe
0 Replies

2. Shell Programming and Scripting

awk to extract digit in line of text and create link

I am trying to extract the number in bold (leading zero removed) after Medexome_xx_numbertoextractin file and create an output using that extracted number. In the output the on thing that will change is the number the other test is static and will be the same each time. Thank you :). file ... (8 Replies)
Discussion started by: cmccabe
8 Replies

3. Shell Programming and Scripting

Bash/awk and for loop to create a template

Source File: google.cz http://czechrepublic.google.com/ http://czechrepublic.google.cz http://czechrepublic.google.com/ http://brno.google.cz http://brno.google.com/ Fail Code root@arisvm ~/g] $ cat trya rm -f ss for i in a.txt do #b=`cat $i|awk '{print $1}'` #c=`cat $i|awk '{print... (4 Replies)
Discussion started by: invinzin21
4 Replies

4. Shell Programming and Scripting

Bash for loop with awk and variables

I'm relatively new to bash scripting and am trying to use awk inside a bash for loop but am having a problem with the variables. for i in $(seq 30 5 60) do # Define variables up and down in AWK eval $(awk 'BEGIN{ print "up=$((i+1))"}' < /dev/null) eval $(awk 'BEGIN{ print... (2 Replies)
Discussion started by: lily-anne
2 Replies

5. Shell Programming and Scripting

Create, validate and using dynamic variables in Bash scripting

Hi All, I am really struggling to solve this problem, this might be small but I am not able to, can somebody help me? I have few directories and these directories receives text files in large amount with in fraction of seconds. So I just want to send all the files in current directory to... (2 Replies)
Discussion started by: VasuKukkapalli
2 Replies

6. UNIX for Advanced & Expert Users

Cannot create a link with the same name

Hi - Our system admin cannot create a link to NAS with one particular name but he can create with other names. What might be the cause? Regards ---------- Post updated at 11:30 AM ---------- Previous update was at 11:29 AM ---------- It says permission denied. (2 Replies)
Discussion started by: w020637
2 Replies

7. Solaris

Create a Link for a directory

Hi, I need to create a link as stagein01 for the /p11/prod/stagein01/. Please let me know the procedure for the same. Regards, VN (5 Replies)
Discussion started by: narayanv
5 Replies

8. Shell Programming and Scripting

how to create variables in loop and assign filename after set command?

Hi, does anybody knows how to manage, that the filenames are assigned to a variable in a loop afer getting them with set command in a ksh, like: set B*.txt i=1 c=$# x=$((c+1)) echo "$x" while ] ; do _ftpfile$i="$"$i echo "$_ftpfile$i" i=$((i+1)) done The first echo returns,... (2 Replies)
Discussion started by: spidermike
2 Replies

9. Shell Programming and Scripting

AWK create loop for fields

Hi everybody: I just create awk script. This script calculate the average of a field, but i want to do it for every fields. The original field has 40 fields of numbers like that: 38.00 51.00 10.00 -99.90 75.00 47.00 4.00 -99.90 69.00 121.00 62.00 6.00 70.00 43.00 36.00 49.00 8.00 36.00 50.00... (1 Reply)
Discussion started by: tonet
1 Replies

10. UNIX for Dummies Questions & Answers

Create Symbolic Link

I am Solaris korn shell. I want to create a symbolic link. I have a directory /u01/ftp01/db I want to reference it as /u05/swe/my (this is not a real directory) I tried a symbolic link but it does not work ln -s /u01/ftp01/db /u05/swe/my ln: cannot create //u05/swe/my: No such... (2 Replies)
Discussion started by: lesstjm
2 Replies
Login or Register to Ask a Question