How to list and move files with spaces and wildcard?
I am writing a code that can move and archve all the files in a directory except the latest file based on file pattern provided in a controlfile.
The filename is in the form of pattern. So basically we find the all the files of the pattern provided and archive all of them, leaving one latest file.
I am on Sun Solaris unix machine.
The control fileTestcontrol.csv is a .csv(comma delimited) file that contains three fields- directorypath,FilenamePattern,archivedir Something like below
Code:
DirectoryPath,Filename Pattern,ArchiveDirectory
/temp,test*cmp*.txt,/archive/temp/cmp1
/data,cmp raw * carrier cleaned*.txt,/archive/temp/carrier
/data,cigna raw * client*.txt,/archive/temp/client
I have written the code below that reads the control file and perform the logic. It works fine for filenames or patterns that does not have any spaces in the control file. But the code below does NOT work with pattern that has spaces.This is my issue. I would appreciate if you can help me out in modifying the code below to work for filename pattern with spaces.
Code:
CNTRL_FILE=Testcontrol.csv
while read rec
do
v_dirpath=`echo $rec | cut -d',' -f1`
v_filename=`echo $rec | cut -d',' -f2`
v_arch_dir=`echo $rec | cut -d',' -f3`
echo "Processing Dir is: ${v_dirpath}"
echo "Processing File is: ${v_filename}"
echo "Archive Directory is: ${v_arch_dir}"
cd ${v_dirpath}
ls -tp ${v_ProcessingFile} | tail +2 # this finds all the files that matches the file pattern except the latest file
mv `ls -tp ${v_ProcessingFile} | tail +2` ${v_arch_dir};
cd -
done < ${REF_DIR}/${CNTRL_FILE}
Moderator's Comments:
Please use CODE tags as required by forum rules!
Last edited by RudiC; 04-17-2018 at 03:45 AM..
Reason: Changed CODE tags.
The mv command can't tell spaces in file names from spaces used as separators between files, so you have to escape the former by either prefixing each with a \ or by enclosing the entire file name in quotes. An alternative solution would be to mv every single file by itself, like
Code:
{
read
while IFS=, read v_dirpath v_filename v_arch_dir
do echo "Processing Dir is: ${v_dirpath}"
echo "Processing File is: ${v_filename}"
echo "Archive Directory is: ${v_arch_dir}"
ls -tp ${v_filename} |
while read FN
do echo mv $FN ${v_arch_dir}
done
done
} < ${REF_DIR}/${CNTRL_FILE}
As you can see, the unnecessary variable assignments can be avoided by reading the variables from the input file using the correct input field separator (IFS=,). The header line is skipped by the first read outside the loop. The echo in the second while loop is for control / debugging - remove it if happy with the output.
Notabene, the variable v_ProcessingFile is unset and thus empty in above sample code.
The mv command can't tell spaces in file names from spaces used as separators between files, so you have to escape the former by either prefixing each with a \ or by enclosing the entire file name in quotes. An alternative solution would be to mv every single file by itself, like
Code:
{
read
while IFS=, read v_dirpath v_filename v_arch_dir
do echo "Processing Dir is: ${v_dirpath}"
echo "Processing File is: ${v_filename}"
echo "Archive Directory is: ${v_arch_dir}"
ls -tp ${v_filename} |
while read FN
do echo mv $FN ${v_arch_dir}
done
done
} < ${REF_DIR}/${CNTRL_FILE}
As you can see, the unnecessary variable assignments can be avoided by reading the variables from the input file using the correct input field separator (IFS=,). The header line is skipped by the first read outside the loop. The echo in the second while loop is for control / debugging - remove it if happy with the output.
Notabene, the variable v_ProcessingFile is unset and thus empty in above sample code.
Give it a try and report back.
I tried this as suggested in the code below. This works fine for filenames with no spaces but does not work for filenames that spaces. I have pasted the code I tried and output below:
Code:
{
read
while IFS=, read v_dirpath v_filename v_arch_dir
do
echo "Processing Dir is: ${v_dirpath}"
echo "Processing File is: ${v_filename}"
echo "Archive Directory is: ${v_arch_dir}"
echo “The output file is:”
ls -tp ${v_dirpath}/${v_filename}
done
} < ./Test.csv
Output from above code:
Code:
Processing Dir is: /apps/Inb
Processing File is: Agt_Test_Ready*.txt
Archive Directory is: /apps/archive/Inb
The output file is:
/apps/Inb/Agt_Test_Ready_201508261311.txt
/apps/Inb/Agt_Test_Ready_201407161159.txt
Processing Dir is: /apps/Inb
Processing File is: Health State Carrier up*.xls
Archive Directory is: /apps/archive/Inb
The output is:
/apps/Inb/Health: No such file or directoryState: No such file or directoryCarrier: No such file or directoryup*.xls: No such file or directory
You are right - it's the combination of spaces and wildcards in one pattern that is the killer. Spaces only you could deal with by enclosing in double quotes, which in turn prevent the expansion of the wildcards. Catch 22.
Solution would be escaping every single space with a \ , and using the (deprecated, as generally dangerous) eval command:
Code:
eval ls -tp ${v_filename// /\\ }
Please make sure you know EXACTLY what is being evaled, as it would rigorously evaluate and execute e.g. rm / ...
And, enclose "$FN" in double quotes in my above script as well...
I think I must be missing something obvious but I have a file containing a list of files and paths, some with wildcard, others with spaces. e.g.
/this/is/a/file
/this/is/a/directory/
/this/is/a/collection/*
/this/has spaces/in/it
/this/had spaces/and/list/of/files*... (6 Replies)
Greetings. I know enough Unix to be dangerous (!) and know that there is a clever way to do the following and it will save me about a day of agony (this time) and I will use it forever after! (many days of agony saved in the future)!
Basically
I need to find any image files (JPGs, PSDs etc)... (5 Replies)
I have a number of files in a directory that can be grouped with something like "ls | grep SH2". I would like to move each file in this list to another directory.
Thanks (4 Replies)
Hi,
Anyone can help me on how to list the file with spaces? Like I want to "ls" only the 2008 files.
2008 _overview102.jpg
2008 _overview103.jpg
2008 _overview106.jpg
2008 _overview677.jpg
2008 _overview680.jpg
2008 _overview110.jpg
2008 _overview682.jpg
2009 _overview4373.jpg
2009... (1 Reply)
I have a text file containing files in a directory structure i.e.
/project/hr/raw/jcpdatav/datav_aug03
/project/hr/raw/jcpdatav/comb8121sep02n
/project/hr/raw/jcpdatav/datav_feb04_ons
/project/hr/raw/jcpdatav/corpsick_jun06
/project/hr/raw/jcpdatav/jcpjoiners200507... (3 Replies)
Hi,
I need to do something easy but I can't seem to figure out how to do this.
Let's say I have 6 files in the directory below:
/ebsbeta_f/flash/EBSUATQB/onlinelog
o1_mf_6_55klt7nr_.log
o1_mf_3_55klskj4_.log
o1_mf_4_55klsrl1_.log
o1_mf_5_55klt09p_.log
o1_mf_2_55klv1ts_.log... (10 Replies)
Hi,
I have a directory with possibly around 800,000 files in it.
What is the fastest way to list file(s) in this directory with a wildcard.
for example would
ls -1 *.abcdefg.Z
or
find . -name "*.abcdefg.Z"
be the fastest way to find all of the files that end with .abcdefg.Z... (6 Replies)
I have the following requirement.
PATHA =/opr/itr/
PATHB=/etc/data/
FILENAME=*abc*
file name is wild carded as there could be many files with name abc anywhere
I tried mv $PATHA/$FILENAME $PATHB
and I got a error that /etc/data/ can not be created
Could any one please help.
... (1 Reply)
I want to do a search and replace in a file but there may or may not be one space, two or none.
i.e.:
test 3,1
test 3,1
test,3,1
test 3,1
The output I will be looking for is this:
test 3,2
test 3,2
test,3,2... (6 Replies)
I'm trying to figure out how to build a small shell script that will find old .shtml files in every /tgp/ directory on the server and delete them if they are older than 10 days...
The structure of the paths are like this:
/home/domains/www.domain2.com/tgp/
/home/domains/www.domain3.com/tgp/... (1 Reply)