How to list and move files with spaces and wildcard?


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting How to list and move files with spaces and wildcard?
# 1  
Old 04-16-2018
How to list and move files with spaces and wildcard?

I am writing a code that can move and archve all the files in a directory except the latest file based on file pattern provided in a controlfile.
The filename is in the form of pattern. So basically we find the all the files of the pattern provided and archive all of them, leaving one latest file.
I am on Sun Solaris unix machine.

The control fileTestcontrol.csv is a .csv(comma delimited) file that contains three fields- directorypath,FilenamePattern,archivedir
Something like below

Code:
DirectoryPath,Filename Pattern,ArchiveDirectory
/temp,test*cmp*.txt,/archive/temp/cmp1
/data,cmp raw * carrier cleaned*.txt,/archive/temp/carrier
/data,cigna raw * client*.txt,/archive/temp/client

I have written the code below that reads the control file and perform the logic. It works fine for filenames or patterns that does not have any spaces in the control file.
But the code below does NOT work with pattern that has spaces.This is my issue. I would appreciate if you can help me out in modifying the code below to work for filename pattern with spaces.

Code:
 
 CNTRL_FILE=Testcontrol.csv
 while read rec
do 
     v_dirpath=`echo $rec | cut -d',' -f1`
     v_filename=`echo $rec | cut -d',' -f2`
     v_arch_dir=`echo $rec | cut -d',' -f3`
     
     echo "Processing Dir is: ${v_dirpath}"
     echo "Processing File is: ${v_filename}"
     echo "Archive Directory is: ${v_arch_dir}"
     cd ${v_dirpath}
                                
     ls -tp ${v_ProcessingFile} | tail +2  # this finds all the files that matches the file pattern except the latest file
      mv `ls -tp ${v_ProcessingFile} | tail +2` ${v_arch_dir};
cd -
 done < ${REF_DIR}/${CNTRL_FILE}

Moderator's Comments:
Mod Comment Please use CODE tags as required by forum rules!

Last edited by RudiC; 04-17-2018 at 03:45 AM.. Reason: Changed CODE tags.
# 2  
Old 04-17-2018
The mv command can't tell spaces in file names from spaces used as separators between files, so you have to escape the former by either prefixing each with a \ or by enclosing the entire file name in quotes. An alternative solution would be to mv every single file by itself, like
Code:
{
read
while IFS=, read v_dirpath v_filename v_arch_dir
   do    echo "Processing Dir is: ${v_dirpath}"
         echo "Processing File is: ${v_filename}"
         echo "Archive Directory is: ${v_arch_dir}"
         ls -tp ${v_filename} |
           while read FN
             do    echo mv $FN ${v_arch_dir}
             done
   done 
 } < ${REF_DIR}/${CNTRL_FILE}

As you can see, the unnecessary variable assignments can be avoided by reading the variables from the input file using the correct input field separator (IFS=,). The header line is skipped by the first read outside the loop. The echo in the second while loop is for control / debugging - remove it if happy with the output.

Notabene, the variable v_ProcessingFile is unset and thus empty in above sample code.

Give it a try and report back.

Last edited by RudiC; 04-17-2018 at 04:19 AM..
This User Gave Thanks to RudiC For This Post:
# 3  
Old 04-18-2018
Tried it but did not work

Quote:
Originally Posted by RudiC
The mv command can't tell spaces in file names from spaces used as separators between files, so you have to escape the former by either prefixing each with a \ or by enclosing the entire file name in quotes. An alternative solution would be to mv every single file by itself, like
Code:
{
read
while IFS=, read v_dirpath v_filename v_arch_dir
   do    echo "Processing Dir is: ${v_dirpath}"
         echo "Processing File is: ${v_filename}"
         echo "Archive Directory is: ${v_arch_dir}"
         ls -tp ${v_filename} |
           while read FN
             do    echo mv $FN ${v_arch_dir}
             done
   done 
 } < ${REF_DIR}/${CNTRL_FILE}

As you can see, the unnecessary variable assignments can be avoided by reading the variables from the input file using the correct input field separator (IFS=,). The header line is skipped by the first read outside the loop. The echo in the second while loop is for control / debugging - remove it if happy with the output.

Notabene, the variable v_ProcessingFile is unset and thus empty in above sample code.

Give it a try and report back.

I tried this as suggested in the code below. This works fine for filenames with no spaces but does not work for filenames that spaces. I have pasted the code I tried and output below:

Code:
 
 {
 read
 while IFS=, read v_dirpath v_filename v_arch_dir
 do    
          
          echo "Processing Dir is: ${v_dirpath}"
          echo "Processing File is: ${v_filename}"
          echo "Archive Directory is: ${v_arch_dir}"
          
         echo “The output file is:”          
          ls -tp ${v_dirpath}/${v_filename}
         
 done 
 } < ./Test.csv

Output from above code:
Code:
 
 Processing Dir is: /apps/Inb
 Processing File is: Agt_Test_Ready*.txt
 Archive Directory is: /apps/archive/Inb
 The output file is:
 /apps/Inb/Agt_Test_Ready_201508261311.txt
 /apps/Inb/Agt_Test_Ready_201407161159.txt
  
 Processing Dir is: /apps/Inb
 Processing File is: Health State Carrier up*.xls
 Archive Directory is: /apps/archive/Inb
 The output is:
 /apps/Inb/Health: No such file or directory
 State: No such file or directory
 Carrier: No such file or directory
 up*.xls: No such file or directory

Will appreciate any help on this.

Thank you so much
# 4  
Old 04-18-2018
You are right - it's the combination of spaces and wildcards in one pattern that is the killer. Spaces only you could deal with by enclosing in double quotes, which in turn prevent the expansion of the wildcards. Catch 22.
Solution would be escaping every single space with a \ , and using the (deprecated, as generally dangerous) eval command:
Code:
eval  ls -tp ${v_filename// /\\ }

Please make sure you know EXACTLY what is being evaled, as it would rigorously evaluate and execute e.g. rm / ...

And, enclose "$FN" in double quotes in my above script as well...

Last edited by RudiC; 04-18-2018 at 04:10 PM..
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Expanding a list of wildcard filenames with spaces

I think I must be missing something obvious but I have a file containing a list of files and paths, some with wildcard, others with spaces. e.g. /this/is/a/file /this/is/a/directory/ /this/is/a/collection/* /this/has spaces/in/it /this/had spaces/and/list/of/files*... (6 Replies)
Discussion started by: mij
6 Replies

2. UNIX for Dummies Questions & Answers

Find a list of files in directory, move to new, allow duplicates

Greetings. I know enough Unix to be dangerous (!) and know that there is a clever way to do the following and it will save me about a day of agony (this time) and I will use it forever after! (many days of agony saved in the future)! Basically I need to find any image files (JPGs, PSDs etc)... (5 Replies)
Discussion started by: Clyde Lovett
5 Replies

3. Shell Programming and Scripting

Move files in a list to another directory

I have a number of files in a directory that can be grouped with something like "ls | grep SH2". I would like to move each file in this list to another directory. Thanks (4 Replies)
Discussion started by: kg6iia
4 Replies

4. UNIX for Dummies Questions & Answers

List of files with spaces

Hi, Anyone can help me on how to list the file with spaces? Like I want to "ls" only the 2008 files. 2008 _overview102.jpg 2008 _overview103.jpg 2008 _overview106.jpg 2008 _overview677.jpg 2008 _overview680.jpg 2008 _overview110.jpg 2008 _overview682.jpg 2009 _overview4373.jpg 2009... (1 Reply)
Discussion started by: fspalero
1 Replies

5. Shell Programming and Scripting

List files with spaces in filename

I have a text file containing files in a directory structure i.e. /project/hr/raw/jcpdatav/datav_aug03 /project/hr/raw/jcpdatav/comb8121sep02n /project/hr/raw/jcpdatav/datav_feb04_ons /project/hr/raw/jcpdatav/corpsick_jun06 /project/hr/raw/jcpdatav/jcpjoiners200507... (3 Replies)
Discussion started by: mr_crosby
3 Replies

6. Shell Programming and Scripting

Need to move and rename a list of files

Hi, I need to do something easy but I can't seem to figure out how to do this. Let's say I have 6 files in the directory below: /ebsbeta_f/flash/EBSUATQB/onlinelog o1_mf_6_55klt7nr_.log o1_mf_3_55klskj4_.log o1_mf_4_55klsrl1_.log o1_mf_5_55klt09p_.log o1_mf_2_55klv1ts_.log... (10 Replies)
Discussion started by: exm
10 Replies

7. Shell Programming and Scripting

Fastest way to list a file in a folder containing 800,000 files using wildcard

Hi, I have a directory with possibly around 800,000 files in it. What is the fastest way to list file(s) in this directory with a wildcard. for example would ls -1 *.abcdefg.Z or find . -name "*.abcdefg.Z" be the fastest way to find all of the files that end with .abcdefg.Z... (6 Replies)
Discussion started by: jerardfjay
6 Replies

8. Shell Programming and Scripting

move files using wildcard

I have the following requirement. PATHA =/opr/itr/ PATHB=/etc/data/ FILENAME=*abc* file name is wild carded as there could be many files with name abc anywhere I tried mv $PATHA/$FILENAME $PATHB and I got a error that /etc/data/ can not be created Could any one please help. ... (1 Reply)
Discussion started by: onlyjayaram
1 Replies

9. UNIX for Dummies Questions & Answers

wildcard for spaces

I want to do a search and replace in a file but there may or may not be one space, two or none. i.e.: test&nbsp;3,1 test&nbsp;&nbsp; 3,1 test,3,1 test&nbsp;&nbsp;&nbsp;3,1 The output I will be looking for is this: test&nbsp;3,2 test&nbsp;&nbsp;3,2 test,3,2... (6 Replies)
Discussion started by: Shakey21
6 Replies

10. UNIX for Dummies Questions & Answers

Find wildcard .shtml files in wildcard directories and removing them- How's it done?

I'm trying to figure out how to build a small shell script that will find old .shtml files in every /tgp/ directory on the server and delete them if they are older than 10 days... The structure of the paths are like this: /home/domains/www.domain2.com/tgp/ /home/domains/www.domain3.com/tgp/... (1 Reply)
Discussion started by: Neko
1 Replies
Login or Register to Ask a Question