Today (Saturday) We will make some minor tuning adjustments to MySQL.

You may experience 2 up to 10 seconds "glitch time" when we restart MySQL. We expect to make these adjustments around 1AM Eastern Daylight Saving Time (EDT) US.


awk to create link, download, and extract in sub-directory


Login or Register to Reply

 
Thread Tools Search this Thread
# 1  
awk to create link, download, and extract in sub-directory

The awk below will create sub-directories in a directory (which is always the last line of file1, each block separated by an empty line), if the number in line 2 (always the first 6 digits in the format xx-xxxx) of file2 is found in $2 of file1. This is the current awk output.

If there is a match and a sub-directory is created in a directory then the corresponding line1 https in file2 will always be a link to a zip file for download. I can not seem to create that link in the sub-folder, download and extract the .zip. Thank you Smilie

I updated the awk with the lines in bold to grab the download link and put it in the each sub-directory. If I manually enter the download in the terminal it does work. Thank you Smilie.


file1
Code:
xxx_006 19-0000_xxx-yyy-aaa
xxx_007 19-0001_zzz-bbb-ccc
R_2019_02_28_00_xx_yy_user_S5-0271-00-Medexome

yyyy_0287 19-0v02-xxx
yyyy_0289 19-0v31-xxxx
yyyy_0293 19-0v05-xxxx
R_2019_02_15_11_56_40_user_S5-0271-00-v5.6_Oncomine_Childhood_Cancer_Research_DNA_and_Fusions

file2
Code:
https://xx.yy.zz/path/to/file.zip
19-0v05-xxx_000_001
cc112233
https://xx.yy.zz/path/to/download/file.zip
19-0v31-xxx-001-000
bb4456784
https://xx.yy.zz/path/to/file.zip
19-0v02-xxx_000_001
aaa331232

awk
Code:
awk 'NR==FNR { for(i=2; i<NF; i+=2) a[substr($i,1,7)] = $NF; next }  ## start loop and iterate over each first 6 digits in $2 of line 2 in file1
     { k = substr($0, 1, 7) } ## store value extracted in k
             { for(i=1; i<NF; i+=1) a[substr($i,1,7)] = $NF; next } ## start loop and iterate over each previous matching line of file1
     { l = ($0) } ## store value extracted in l (grab the link in line 1)
     k in a { cmd = sprintf("mkdir -p %s/%s", a[k], $0); system(cmd); }  ## for each k in file2 make a directory with sub-directory as k
     l in a { cmd = sprintf("curl -O -v -k -X GET "https://xxx/path/to/download/.zip" -H "Content-Type:application/x-www-form-urlencoded" -H "Authorization:xxx"", a[k], $0); system(cmd); && unzip }  ## for each l in file2 make a directory with sub-directory as k and download l in it
' RS= file1 RS='\n' file2  ## files to use

current awk output
Code:
R_2019_02_15_11_56_40_user_S5-0271-00-v5.6_Oncomine_Childhood_Cancer_Research_DNA_and_Fusions --- directory
   19-0v02-xxx_000_001  --- sub folder
   19-0v05-xxx_000_001  --- sub-folder
   19-0v31-xxx-001-000  --- sub-folder

desired awk output
Code:
R_2019_02_15_11_56_40_user_S5-0271-00-v5.6_Oncomine_Childhood_Cancer_Research_DNA_and_Fusions --- directory
   19-0v02-xxx_000_001  --- sub folder
       https://xx.yy.zz/path/to/file.zip  --- zip and extracted downloaded to sub-folder
   19-0v05-xxx_000_001  --- sub-folder
       https://xx.yy.zz/path/to/file.zip  --- zip and extracted downloaded to sub-folder
   19-0v31-xxx-001-000  --- sub-folder
       https://xx.yy.zz/path/to/file.zip  --- zip and extracted downloaded to sub-folder


Last edited by cmccabe; 03-06-2019 at 08:41 PM.. Reason: updated awk
Login or Register to Reply

|
Thread Tools Search this Thread
Search this Thread:
Advanced Search

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Create directory and sub-directory with awk and bash

In the below I am trying to create a parent directory using the R_2019 line from f1 if what above it is not empty. I then create sub-directories under each parent if there is a match between $2 of f1 and $2. Inside each sub-folder the matching paths in $3 and $4 in f2are printed. If there is no... (2 Replies)
Discussion started by: cmccabe
2 Replies

2. Shell Programming and Scripting

Download on a link doing it indirectly

How do I download on a link doing it indirectly, with once redirection by using curl commands or Bash script of them (2 Replies)
Discussion started by: abdan
2 Replies

3. Shell Programming and Scripting

awk to extract digit in line of text and create link

I am trying to extract the number in bold (leading zero removed) after Medexome_xx_numbertoextractin file and create an output using that extracted number. In the output the on thing that will change is the number the other test is static and will be the same each time. Thank you :). file ... (8 Replies)
Discussion started by: cmccabe
8 Replies

4. Shell Programming and Scripting

awk to create variables to pass into a bash loop to create a download link

I have created one file that contains all the necessary info in it to create a download link. In each of the lines /results/analysis/output/Home/Auto_user_S5-00580-6-Medexome_67_032/plugin_out/FileExporter_out.67... (8 Replies)
Discussion started by: cmccabe
8 Replies

5. Shell Programming and Scripting

Download and extract to a specific directory

Trying to download and extract a file to a specific folder, but getting an error. What am I doing wrong? Is there a way to rename the download if desired? Thank you :). curl --url https://github.com/arq5x/bedtools2/releases/download/v2.26.0/bedtools-2.26.0.tar.gz | tar -xz --output... (4 Replies)
Discussion started by: cmccabe
4 Replies

6. Solaris

Create a Link for a directory

Hi, I need to create a link as stagein01 for the /p11/prod/stagein01/. Please let me know the procedure for the same. Regards, VN (5 Replies)
Discussion started by: narayanv
5 Replies

7. Post Here to Contact Site Administrators and Moderators

Download link

I know this is not the correct area to post this but I am not able to post anywhere. Where do I find the download of Unix, or where have the link to buy it if available? :o (4 Replies)
Discussion started by: CaroLPunk
4 Replies

8. AIX

Multi Link Interface Runtime - where to download ?

Hello, I need "devices.common.IBM.ml 1.4.0.0 C F Multi Link Interface Runtime" to be installed on my machine. I need it for two SAN cards to work correctly. Where do I get it ? thanks Vilius (1 Reply)
Discussion started by: vilius
1 Replies

9. Shell Programming and Scripting

Use awk to create new folder in current directory

Alright, I am sure this is a laughable question, but I don't know so I am going to ask anyway. I have a little script I am writing to take information from one source, recode it in a certain way, and print to files for each subject I have data for. This all works perfectly. I just want to put... (6 Replies)
Discussion started by: ccox85
6 Replies

10. AIX

Link download Exceed 8.0

Please send link download Exceed 8.0 to me,thanks!!! (1 Reply)
Discussion started by: vietrung
1 Replies

Featured Tech Videos