Create directory and sub-directory with awk and bash


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Create directory and sub-directory with awk and bash
# 1  
Old 04-19-2019
Create directory and sub-directory with awk and bash

In the below I am trying to create a parent directory using the R_2019 line from f1 if what above it is not empty.
I then create sub-directories under each parent if there is a match between $2 of f1 and $2. Inside each sub-folder the matching paths in $3 and $4 in f2are printed. If there is no match in f2 then no parent directory is created. So eventhogh in f1 there is a R_2019 line, a directory is only created if a match in $2 between f1 and f2.

There may be multiple matches between each $2 but only 1 match for each parent directory. So there may be multiple sub-folders under each parent but the parent will always be unique. In the case where the R_2019 has an empty value or newline above it, nothing is done or it is skipped.

f1
Code:
xxxx_0000 190326-Control
xxxx_0004 19-0000-L,F
R_2019_xx_yy_xx_yy_xx_xxxx_x0-0000-101-x0.0_xxx_xxxx_xxxx_xx_x_xxx

R_2019_xx_yy_xx_yy_xx_xxxx_x0-0000-100-x0.0_xxx_xxxx_xxxx_xx_x_xxx

xxxx_0002 190326-Control
R_2019_xx_yy_xx_yy_xx_xxxx_x0-0000-99-x0.0_xxx_xxxx_xxxx_xx_x_xxx

xxxx_0008 190302-Control
R_2019_xx_yy_xx_yy_xx_xxxx_x0-0000-93-x0.0_xxx_xxxx_xxxx_xx_x_xxx


f2
Code:
xxxx_0000 190326-Control /path/to/file1 /path/to/file2 /path/to/file3
xxxx_0004 19-0000-L,F /path/to/file1 /path/to/file2 /path/to/file3

xxxx_0002 190302-Control /path/to/file1 /path/to/file2 /path/to/file3


desired
Code:
R_2019_xx_yy_xx_yy_xx_xxxx_x0-0000-101
    190326-Control
       /path/to/file1 /path/to/file2
    19-0000 L,F
       /path/to/file1 /path/to/file2

R_2019_xx_yy_xx_yy_xx_xxxx_x0-0000-99
    190326-Control
      /path/to/file1 /path/to/file2

awk
Code:
awk '
    # create an associative array (key/value pairs) based on the f1
    NR==FNR { for(i=2; i<NF; i+=2) a[substr($i,1,7)] = $NF; next } ## store the 7 digits in $2 in a

    # retrieve the first 7-char of each line in f2 as the key to test against the above hash
    { b = substr($2, 1, 7) } ## store the 7 digits in $2 in b

    # if find k, then print
    b in a { print a[b] "\t" $3 && $4 "\t" p }

    # save prev line to 'p'(these are the paths)
    { p = $0  } 

' RS= f1 RS='\n' f2 | while IFS=$'\t' read -r base_dir sub_dir path; do  ## loop through each key/pair and matching p 
    echo "adding path to data [$path] in each '$base_dir/$sub_dir'"  ## display message
       base_dir=${/R_2019/%%-x0.0*}  ## define parent directory
        sub_dir=${a[b]}  ## define sub-directory
         path=${p}  ## define path
           mkdir -p "$base_dir/$sub_dir/$path"  ## make folder and sub-folder


Last edited by cmccabe; 04-19-2019 at 12:25 PM..
# 2  
Old 04-19-2019
Quote:
Originally Posted by cmccabe
In the below I am trying to create a parent directory using the R_2019 line from f1 if what above it is not empty.
I then create sub-directories under each parent if there is a match between $2 of f1 and $2. Inside each sub-folder the matching paths in $3 and $4 in f2are printed. If there is no match in f2 then no parent directory is created. So eventhogh in f1 there is a R_2019 line, a directory is only created if a match in $2 between f1 and f2.

There may be multiple matches between each $2 but only 1 match for each parent directory. So there may be multiple sub-folders under each parent but the parent will always be unique. In the case where the R_2019 has an empty value or newline above it, nothing is done or it is skipped.

f1
Code:
xxxx_0000 190326-Control
xxxx_0004 19-0000-L,F
R_2019_xx_yy_xx_yy_xx_xxxx_x0-0000-101-x0.0_xxx_xxxx_xxxx_xx_x_xxx

R_2019_xx_yy_xx_yy_xx_xxxx_x0-0000-100-x0.0_xxx_xxxx_xxxx_xx_x_xxx

xxxx_0002 190326-Control
R_2019_xx_yy_xx_yy_xx_xxxx_x0-0000-99-x0.0_xxx_xxxx_xxxx_xx_x_xxx

xxxx_0008 190302-Control
R_2019_xx_yy_xx_yy_xx_xxxx_x0-0000-93-x0.0_xxx_xxxx_xxxx_xx_x_xxx


f2
Code:
xxxx_0000 190326-Control /path/to/file1 /path/to/file2 /path/to/file3
xxxx_0004 19-0000-L,F /path/to/file1 /path/to/file2 /path/to/file3

xxxx_0002 190302-Control /path/to/file1 /path/to/file2 /path/to/file3


desired
Code:
R_2019_xx_yy_xx_yy_xx_xxxx_x0-0000-101
    190326-Control
       /path/to/file1 /path/to/file2
    19-0000 L,F
       /path/to/file1 /path/to/file2

R_2019_xx_yy_xx_yy_xx_xxxx_x0-0000-99
    190326-Control
      /path/to/file1 /path/to/file2

awk
Code:
awk '
    # create an associative array (key/value pairs) based on the f1
    NR==FNR { for(i=2; i<NF; i+=2) a[substr($i,1,7)] = $NF; next } ## store the 7 digits in $2 in a

    # retrieve the first 7-char of each line in f2 as the key to test against the above hash
    { b = substr($2, 1, 7) } ## store the 7 digits in $2 in b

    # if find k, then print
    b in a { print a[b] "\t" $3 && $4 "\t" p }

    # save prev line to 'p'(these are the paths)
    { p = $0  } 

' RS= f1 RS='\n' f2 | while IFS=$'\t' read -r base_dir sub_dir path; do  ## loop through each key/pair and matching p 
    echo "adding path to data [$path] in each '$base_dir/$sub_dir'"  ## display message
       base_dir=${/R_2019/%%-x0.0*}  ## define parent directory
        sub_dir=${a[b]}  ## define sub-directory
         path=${p}  ## define path
           mkdir -p "$base_dir/$sub_dir/$path"  ## make folder and sub-folder

In the first section of code marked in red above you have an awk script that defines the array named a[] and the variable named b. That array and that variable do not exist outside of that awk script. So in the shell script, the third section of code marked in red cannot work because the array a[] and the subscript b have not been defined in your shell script.

In shell code, every parameter expansion (such as the second section of code marked in red above) requires the name of a variable immediately following the opening ${. That expansion cannot work because /R_2019/ is not a valid shell variable name. And no name even slightly resembling that string has been defined in your shell code.

If you want to create a set of directories from data gathered in an awk script you ether need to have the awk script print the names of those directories and have you shell code read those names and use mkdir to create them or you need to have the awk script create mkdir statements and pipe the output through your shell to have the commands created by the awk script be executed by your shell.
This User Gave Thanks to Don Cragun For This Post:
# 3  
Old 04-19-2019
I'm not sure if I completely follow, but I modified the script to utilize awk better. Is the below closer or what do you suggest? I included comments in the code as well. Thank you very much Smilie.

awk
Code:
DIR=/home/cmccabe/pre  ## define data directory
awk '
    # create an associative array (key/value pairs) based on the f1
    NR==FNR { for(i=2; i<NF; i+=2) a[substr($i,1,7)] = $NF; next } ## store the 7 digits in $2 in a

    # retrieve the first 7-char of each line in f2 as the key to test against the above hash
    { b = substr($2, 1, 7) } ## store the 7 digits in $2 in b

    # if find b, then print
    b in a { print a[b] "\t" $3 && $4 "\t" p }

    # save prev line to 'p'(these are the paths)
    { p = $0  } 

' RS= f1 RS='\n' f2 | for RDIR in "$DIR"/R_2019* ; do  ## # matching "R_2019*" to operate on desired directory and expand
                       TRIMSTR=${RDIR%%-x0.0*}  ## trim folder match in RDIR from -x0.0 and store in TRIMSTR
                         mv "$RDIR" "${TRIMSTR}"  ## trim folder
                           mkdir -p "$TRIMSTR/$a[b]/$p"  ## make folder/sub-folder/path
                       done ## end loop

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Bash to check directory and create missing folder from file

In the below bash I am trying to ensure that all folders (represented by $folders) in a given directory are created. In the file f1 the trimmed folder will be there somewhere (will be multiple trimmed folders). When that trimmed folder is found (represented by $S5) the the contents of $2 printed... (19 Replies)
Discussion started by: cmccabe
19 Replies

2. Shell Programming and Scripting

awk to create link, download, and extract in sub-directory

The awk below will create sub-directories in a directory (which is always the last line of file1, each block separated by an empty line), if the number in line 2 (always the first 6 digits in the format xx-xxxx) of file2 is found in $2 of file1. This is the current awk output. If there is a... (0 Replies)
Discussion started by: cmccabe
0 Replies

3. Shell Programming and Scripting

Shell script cannot create directory and move the file to that directory

I have a script, which is checking if file exists and move it to another directory if then mkdir -p ${LOCL_FILES_DIR}/cool_${Today}/monthly mv report_manual_alloc_rpt_A_I_ASSIGNMENT.${Today}*.csv ${LOCL_FILES_DIR}/cool_${Today}/monthly ... (9 Replies)
Discussion started by: digioleg54
9 Replies

4. Shell Programming and Scripting

Create automated scan of specific directory using bash

I am trying to use bash to automate the scan of a specific directory using clamav. Having this in place is a network requirement. The below is an attempt to: 1. count the extensions (.txt, .jpeg) in a directory and write them to a virus-scan.log (section in bold) 2. scan each folder in the... (6 Replies)
Discussion started by: cmccabe
6 Replies

5. Shell Programming and Scripting

How Create new directory and move files to that directory.?

Hi All, We have main directory called "head" under this we have several sub directories and under these directories we have sub directories. My requirement is I have to find the SQL files which are having the string "procedure" under "head" directory and sub directories as well. And create... (14 Replies)
Discussion started by: ROCK_PLSQL
14 Replies

6. Shell Programming and Scripting

Bash to create new directory by date followed by identifier and additional subdirectories

I have a bash that downloads a list and if that list has data in it then a new main directory is created (with the date) with several subdirectories (example1, example2, example3). My question is in that list there are portion of specific file types (.vcf.gz) - identifier towards the end that have... (0 Replies)
Discussion started by: cmccabe
0 Replies

7. Shell Programming and Scripting

List files with date, create directory, move to the created directory

Hi all, i have a folder, with tons of files containing as following, on /my/folder/jobs/ some_name_2016-01-17-22-38-58_some name_0_0.zip.done some_name_2016-01-17-22-40-30_some name_0_0.zip.done some_name_2016-01-17-22-48-50_some name_0_0.zip.done and these can be lots of similar files,... (6 Replies)
Discussion started by: charli1
6 Replies

8. Homework & Coursework Questions

Create script to add user and create directory

first off let me introduce myself. My name is Eric and I am new to linux, I am taking an advanced linux administration class and we are tasked with creating a script to add new users that anyone can run, has to check for the existence of a directory. if the directory does not exist then it has... (12 Replies)
Discussion started by: pbhound
12 Replies

9. Shell Programming and Scripting

Use awk to create new folder in current directory

Alright, I am sure this is a laughable question, but I don't know so I am going to ask anyway. I have a little script I am writing to take information from one source, recode it in a certain way, and print to files for each subject I have data for. This all works perfectly. I just want to put... (6 Replies)
Discussion started by: ccox85
6 Replies

10. UNIX for Advanced & Expert Users

unable to create any directory that uses numbers as the directory name

Version-Release number of selected component (if applicable): ================================================================== root@server # cat /etc/redhat-release Red Hat Enterprise Linux ES release 4 (Nahant Update 5) root@server # uname -a Linux server.integrityserver.net... (16 Replies)
Discussion started by: fed.linuxgossip
16 Replies
Login or Register to Ask a Question