Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

Bash to create sub directories from specific file extension

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Bash to create sub directories from specific file extension
# 1  
Old 03-20-2019
Bash to create sub directories from specific file extension
In the bash below I am trying to create sub-directories inside a directory from files with specific .bam extensions. There may be more then one $RDIR ing the directory and the .bam file(s) are trimmed (removing the extension and IonCode_0000_) and the result is the folder name that is saved in $RDIR or one level up from where the original files are found. There may be multiple .bam files but they are always the same format. I made comments as well and ran set -x and think the find is not working as expected but am not able to fix it . Thank you Smilie.

structure of /home/cmccabe/Desktop/folder --- this is $DIR ---

Code:
R_2019_00_00_00_00_00_xxxx_xx-0000-00  --- this is $RDIR ---
       BAM   ---subdirectory---
       IonCode_0241_19-0000-Last-First.bam.bai
       IonCode_0241_19-0000-Last-First.bam
       IonCode_0243_19-0001-Las-Fir.bam.bai
       IonCode_0243_19-0001-Las-Fir.bam
     QC    ---subdirectory---

after script structure of /home/cmccabe/Desktop/folder --- this is $DIR ---
Code:
R_2019_00_00_00_00_00_xxxx_xx-0000-00  --- this is $RDIR ---
     BAM                  ---subdirectory---
     19-0000-Last-First   ---subdirectory---
     19-0001-Las-Fir      ---subdirectory---
     QC                   ---subdirectory---

set -x
Code:
set -x
cmccabe@DTV-A5211QLM:~$ DIR=/home/cmccabe/Desktop/sub   ## define data directory path
+ DIR=/home/cmccabe/Desktop/sub
cmccabe@DTV-A5211QLM:~$ for RDIR in "$DIR"/R_2019* ; do  ## start processing matching "R_2019*" to operate on desired directory and expand
> echo "$RDIR"
>  cd "$RDIR"/BAM   ## change directory to subfolder inside $RDIR
>   bam=$(find . -type f -name "*.bam")   # extract .bam
>   sample="$(echo $bam|cut -d_ -f3-)" # remove before second underscore
>   mkdir -p "${sample%.*}" && mv "$sample" "RDIR"/"${sample%.*}"  ## make directory of sample id one level up
> cd ..
> done  ## close loop
+ for RDIR in '"$DIR"/R_2019*'
+ echo /home/cmccabe/Desktop/sub/R_2019_01_23_13_22_58_user_S5-0271-93
/home/cmccabe/Desktop/sub/R_2019_01_23_13_22_58_user_S5-0271-93
+ cd /home/cmccabe/Desktop/sub/R_2019_01_23_13_22_58_user_S5-0271-93/BAM
++ find . -type f -name '*.bam'
+ bam='./IonCode_0251_19-0003-La-Fi.bam
./IonCode_0243_19-0001-Las-Fir.bam
./IonCode_0255_190319-Control.bam
./IonCode_0241_19-0000-Last-First.bam'
++ echo ./IonCode_0251_19-0003-La-Fi.bam ./IonCode_0243_19-0001-Las-Fir.bam ./IonCode_0255_190319-Control.bam ./IonCode_0241_19-0000-Last-First.bam
++ cut -d_ -f3-
+ sample='19-0003-La-Fi.bam ./IonCode_0243_19-0001-Las-Fir.bam ./IonCode_0255_190319-Control.bam ./IonCode_0241_19-0000-Last-First.bam'
+ mkdir -p '19-0003-La-Fi.bam ./IonCode_0243_19-0001-Las-Fir.bam ./IonCode_0255_190319-Control.bam ./IonCode_0241_19-0000-Last-First'
+ mv '19-0003-La-Fi.bam ./IonCode_0243_19-0001-Las-Fir.bam ./IonCode_0255_190319-Control.bam ./IonCode_0241_19-0000-Last-First.bam' 'RDIR/19-0003-La-Fi.bam ./IonCode_0243_19-0001-Las-Fir.bam ./IonCode_0255_190319-Control.bam ./IonCode_0241_19-0000-Last-First'
mv: cannot stat ‘19-0003-La-Fi.bam ./IonCode_0243_19-0001-Las-Fir.bam ./IonCode_0255_190319-Control.bam ./IonCode_0241_19-0000-Last-First.bam': No such file or directory
+ cd ..
+ for RDIR in '"$DIR"/R_2019*'
+ echo /home/cmccabe/Desktop/sub/R_2019_01_23_13_22_58_user_S5-0271-96
/home/cmccabe/Desktop/sub/R_2019_01_23_13_22_58_user_S5-0271-96
+ cd /home/cmccabe/Desktop/sub/R_2019_01_23_13_22_58_user_S5-0271-96/BAM
++ find . -type f -name '*.bam'
+ bam='./IonCode_0251_19-0003-La-Fi.bam
./IonCode_0243_19-0001-Las-Fir.bam
./IonCode_0255_190319-Control.bam
./IonCode_0241_19-0000-Last-First.bam'
++ echo ./IonCode_0251_19-0003-La-Fi.bam ./IonCode_0243_19-0001-Las-Fir.bam ./IonCode_0255_190319-Control.bam ./IonCode_0241_19-0000-Last-First.bam
++ cut -d_ -f3-
+ sample='19-0003-La-Fi.bam ./IonCode_0243_19-0001-Las-Fir.bam ./IonCode_0255_190319-Control.bam ./IonCode_0241_19-0000-Last-First.bam'
+ mkdir -p '19-0003-La-Fi.bam ./IonCode_0243_19-0001-Las-Fir.bam ./IonCode_0255_190319-Control.bam ./IonCode_0241_19-0000-Last-First'
+ mv '19-0003-La-Fi.bam ./IonCode_0243_19-0001-Las-Fir.bam ./IonCode_0255_190319-Control.bam ./IonCode_0241_19-0000-Last-First.bam' 'RDIR/19-0003-La-Fi.bam ./IonCode_0243_19-0001-Las-Fir.bam ./IonCode_0255_190319-Control.bam ./IonCode_0241_19-0000-Last-First'
mv: cannot stat ‘19-0003-La-Fi.bam ./IonCode_0243_19-0001-Las-Fir.bam ./IonCode_0255_190319-Control.bam ./IonCode_0241_19-0000-Last-First.bam': No such file or directory
+ cd ..

Last edited by cmccabe; 03-20-2019 at 10:30 AM.. Reason: fixed format
# 2  
Old 03-20-2019
I modified the find and added a loop to mkdir and got the expected results. Just wanted to share in case it helps anyone else in the future. Thank you Smilie.

Code:
DIR=/home/cmccabe/Desktop/sub   ## define data directory path
for RDIR in "$DIR"/R_2019* ; do  ## start processing matching "R_2019*" to operate on desired directory and expand
 cd "$RDIR"/BAM   ## change directory to subfolder inside $RDIR
i=$(find . -type f -name "*.bam" -print | while read f;do echo "$f" | cut -d_ -f2-;done| cut -f 1 -d '.')  # extract .bam
for x in $i
        do mkdir -p $RDIR/$x
        done
done  ## close loop

This User Gave Thanks to cmccabe For This Post:
Neo
# 3  
Old 03-20-2019
Code:
$(find * -type f -name "*.bam" | cut -d_ -f2 | cut -d. -f1)

--- Post updated at 18:32 ---

endless running through directories means only one thing, you don't know where you will when will creating catalogs
Code:
find "$RDIR"/BAM -type f -name "*.bam" | cut -d_ -f2 | cut -d. -f1 |  xargs -I {} mkdir -p "$RDIR/"{}

--- Post updated at 18:44 ---

In any case, it is better not to run on items twice.
Code:
do
    cd "$RDIR"/BAM
    find * -type f -name "*.bam" | cut -d_ -f2 | cut -d. -f1 |  xargs -I {} mkdir -p "$RDIR/"{}
done

Last edited by nezabudka; 03-20-2019 at 12:37 PM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

Bash to remove find and remove specific extension

The bash below executes and does find all the .bam files in each R_2019 folder. However set -x shows that the .bam extension only gets removed from one .bam file in each folder (appears to be the last in each). Why is it not removing the extension from each (this is $SAMPLE)? Thank you :). set... (4 Replies)
Discussion started by: cmccabe
4 Replies

2. Shell Programming and Scripting

Create automated scan of specific directory using bash

I am trying to use bash to automate the scan of a specific directory using clamav. Having this in place is a network requirement. The below is an attempt to: 1. count the extensions (.txt, .jpeg) in a directory and write them to a virus-scan.log (section in bold) 2. scan each folder in the... (6 Replies)
Discussion started by: cmccabe
6 Replies

3. Shell Programming and Scripting

Rename specific file extension in directory with match to another file in bash

I have a specific set (all ending with .bam) of downloaded files in a directory /home/cmccabe/Desktop/NGS/API/2-15-2016. What I am trying to do is use a match to $2 in name to rename the downloaded files. To make things a more involved the date of the folder is unique and in the header of name... (1 Reply)
Discussion started by: cmccabe
1 Replies

4. Shell Programming and Scripting

Create a specific bash to increase the automatic update on my pi

hi everyone, I just began to be interested about the bash access. I buy a time ago a Raspberry pi, I installed raspbmc and now I would like build a bash to copy everyday all new files inside my server directly on the hard drive in my Pi. So my HDD is directly plug on my pi with usb connect,... (4 Replies)
Discussion started by: nagito34
4 Replies

5. Shell Programming and Scripting

Bash command line to strip tar.gz file extension?

What's the command syntax for stripping out the tar.gz file extension in a bash command line (not script file). Thanks! prompt/> ls *.tar.gz | <what comes here?> (3 Replies)
Discussion started by: ZillaG
3 Replies

6. Shell Programming and Scripting

Passinng specific file extension to while loop in ksh

hello, i have the below while loop wherein i am passig list of filenames to be scped. this is in unix ksh - filenamelist.txt has list of files names, including .dat and .txt files but i want to pass only the .txt filenames to the while loop so that only .txt files gets scped. how can... (4 Replies)
Discussion started by: billpeter3010
4 Replies

7. UNIX for Dummies Questions & Answers

creating separate directories according to file extension and keeping file in different directory as

unix program to which a directory name will be passed as parameter. This directory will contain files with various extensions. This script will create directories with the names of the extention of the files and then put the files in the corresponding folder. All files which do not have any... (2 Replies)
Discussion started by: Deekay.p
2 Replies

8. Shell Programming and Scripting

find directories that do NOT contain a certain file extension

for x in `find /vmfs/volumes/v01tstn01a01/ -type d`; do find $x -name '*.vmx' > /dev/null || echo $x; done; The goal of this is to find the subdirectories that do NOT contain a file with the extension of .vmx Any help would be great! (6 Replies)
Discussion started by: hodnov
6 Replies

9. Homework & Coursework Questions

Create file and then change the extension case.

Interpreter should be bash. 1. The problem statement, all variables and given/known data: I need to make a file (myText.txt or song.mp3 or cloud.tar.gz or whatever) and then change the extension to (myText.TXT , song.MP3, cloud.TAR.GZ). It would be good if I can add all information in... (4 Replies)
Discussion started by: Kdenmen
4 Replies

10. Shell Programming and Scripting

While file exists with specific extension

Hi, I need some help to see if I this is posible with a while loop. I need to run a script once a day that will take all of the *.TXT files in a folder and rename them to a specific file name structure with a .dat extension. I wrote this script, but I get an error that says " line 3: too many... (2 Replies)
Discussion started by: strpwr
2 Replies
Login or Register to Ask a Question