awk to create link, download, and extract in sub-directory


Login or Register to Reply

 
Thread Tools Search this Thread
# 1  
Old 2 Weeks Ago
awk to create link, download, and extract in sub-directory

The awk below will create sub-directories in a directory (which is always the last line of file1, each block separated by an empty line), if the number in line 2 (always the first 6 digits in the format xx-xxxx) of file2 is found in $2 of file1. This is the current awk output.

If there is a match and a sub-directory is created in a directory then the corresponding line1 https in file2 will always be a link to a zip file for download. I can not seem to create that link in the sub-folder, download and extract the .zip. Thank you Smilie

I updated the awk with the lines in bold to grab the download link and put it in the each sub-directory. If I manually enter the download in the terminal it does work. Thank you Smilie.


file1
Code:
xxx_006 19-0000_xxx-yyy-aaa
xxx_007 19-0001_zzz-bbb-ccc
R_2019_02_28_00_xx_yy_user_S5-0271-00-Medexome

yyyy_0287 19-0v02-xxx
yyyy_0289 19-0v31-xxxx
yyyy_0293 19-0v05-xxxx
R_2019_02_15_11_56_40_user_S5-0271-00-v5.6_Oncomine_Childhood_Cancer_Research_DNA_and_Fusions

file2
Code:
https://xx.yy.zz/path/to/file.zip
19-0v05-xxx_000_001
cc112233
https://xx.yy.zz/path/to/download/file.zip
19-0v31-xxx-001-000
bb4456784
https://xx.yy.zz/path/to/file.zip
19-0v02-xxx_000_001
aaa331232

awk
Code:
awk 'NR==FNR { for(i=2; i<NF; i+=2) a[substr($i,1,7)] = $NF; next }  ## start loop and iterate over each first 6 digits in $2 of line 2 in file1
     { k = substr($0, 1, 7) } ## store value extracted in k
             { for(i=1; i<NF; i+=1) a[substr($i,1,7)] = $NF; next } ## start loop and iterate over each previous matching line of file1
     { l = ($0) } ## store value extracted in l (grab the link in line 1)
     k in a { cmd = sprintf("mkdir -p %s/%s", a[k], $0); system(cmd); }  ## for each k in file2 make a directory with sub-directory as k
     l in a { cmd = sprintf("curl -O -v -k -X GET "https://xxx/path/to/download/.zip" -H "Content-Type:application/x-www-form-urlencoded" -H "Authorization:xxx"", a[k], $0); system(cmd); && unzip }  ## for each l in file2 make a directory with sub-directory as k and download l in it
' RS= file1 RS='\n' file2  ## files to use

current awk output
Code:
R_2019_02_15_11_56_40_user_S5-0271-00-v5.6_Oncomine_Childhood_Cancer_Research_DNA_and_Fusions --- directory
   19-0v02-xxx_000_001  --- sub folder
   19-0v05-xxx_000_001  --- sub-folder
   19-0v31-xxx-001-000  --- sub-folder

desired awk output
Code:
R_2019_02_15_11_56_40_user_S5-0271-00-v5.6_Oncomine_Childhood_Cancer_Research_DNA_and_Fusions --- directory
   19-0v02-xxx_000_001  --- sub folder
       https://xx.yy.zz/path/to/file.zip  --- zip and extracted downloaded to sub-folder
   19-0v05-xxx_000_001  --- sub-folder
       https://xx.yy.zz/path/to/file.zip  --- zip and extracted downloaded to sub-folder
   19-0v31-xxx-001-000  --- sub-folder
       https://xx.yy.zz/path/to/file.zip  --- zip and extracted downloaded to sub-folder


Last edited by cmccabe; 2 Weeks Ago at 07:41 PM.. Reason: updated awk
Login or Register to Reply

|
Thread Tools Search this Thread
Search this Thread:
Advanced Search

More UNIX and Linux Forum Topics You Might Find Helpful
Download on a link doing it indirectly abdan Shell Programming and Scripting 2 08-22-2018 05:39 AM
awk to extract digit in line of text and create link cmccabe Shell Programming and Scripting 8 10-29-2016 02:21 PM
awk to create variables to pass into a bash loop to create a download link cmccabe Shell Programming and Scripting 8 09-27-2016 12:51 PM
Download and extract to a specific directory cmccabe Shell Programming and Scripting 4 08-01-2016 10:10 AM
AIX 4.3 ISO download link require Rajan Ghanshyam AIX 2 12-11-2014 12:17 PM
Cannot create a link with the same name w020637 UNIX for Advanced & Expert Users 2 09-12-2013 09:54 PM
Create a Link for a directory narayanv Solaris 5 08-06-2013 04:32 PM
Not able to create a link in /etc/rc3.d in Solaris 10 muraliinfy04 Solaris 5 03-19-2013 06:55 AM
SED - Create mailto: link Nigel_R Shell Programming and Scripting 5 07-21-2011 02:17 PM
Download link CaroLPunk Post Here to Contact Site Administrators and Moderators 4 04-23-2011 04:04 PM
Multi Link Interface Runtime - where to download ? vilius AIX 1 07-23-2009 03:04 PM
Use awk to create new folder in current directory ccox85 Shell Programming and Scripting 6 01-28-2008 02:59 AM
Link download Exceed 8.0 vietrung AIX 1 10-24-2007 12:45 AM
how to create symbolic link between folders..... manas_ranjan Shell Programming and Scripting 2 10-05-2007 04:31 AM
Create Symbolic Link lesstjm UNIX for Dummies Questions & Answers 2 03-21-2005 02:36 PM