Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

put file in different directory according to the filename

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting put file in different directory according to the filename
# 1  
Old 02-01-2012
put file in different directory according to the filename
Dear all,

I would like to know how to move the downloaded files in differenent directories according to the name of the file?

i.e.
Code:
P10120111201_122013M.jpg
P10120120101_122013M.jpg
P10120120201_122013M.jpg

The first 4 charactors "P101" is the station name, 4-8 means year "2012", 9-10 means month "12""01""02", 11-12 means day"01", the 6 charactors after underline are hour, minutes and second in UTC time.

Now, I want to move the photos according to the year and day. Also, there are millions of photos like that.

The directories are :
Code:
/data/htdocs/field_photos/files/by_site/p101/webcam/2011/webcam/Dec

/data/htdocs/field_photos/files/by_site/p101/webcam/2012/webcam/Jan

/data/htdocs/field_photos/files/by_site/p101/webcam/2012/webcam/Feb

Thank you very much.

Chen


Moderator's Comments:
Mod Comment Use code tags please, see PM; thanks.
Last edited by zaxxon; 02-01-2012 at 12:47 PM.. Reason: code tags
# 2  
Old 02-01-2012
What's your system?

What's your shell?

What do you want the file names to end up as?
# 3  
Old 02-01-2012
Further to Corona688:
What type of filesystem is this? If "millions" is a real figure, the choice of filesystem is going to be important.
Is all the movement taking place within the original filesystem?
"I want to move the photos according to the year and day". Where is the "day" in your directory structure.
Is "P101" (data) and "p101" (directory) a typo?
Any reason to have "webcam" twice in the directory tree?
What directory are all these files in at the moment?

Unless you have any burning reason to use alphabetic months in directory names, I'd recommend that you use numerics (with leading zeros) because alphabetic months just don't come out in the right order in directory listings (in your example Feb would be before Jan).
# 4  
Old 02-01-2012
I hope my code can scale to 'million', please let me know.

Code:
function f2p()
{
  awk -v f="$1" '
  BEGIN {
    station=substr(f,1,4)
    year=substr(f,5,4)
    month=substr(f,9,2)
    m["01"]="Jan"; m["02"]="Feb";m["03"]="Mar";m["04"]="Apr";
    m["05"]="May"; m["06"]="Jun";m["07"]="Jul";m["08"]="Aug";
    m["09"]="Sep"; m["10"]="Oct";m["11"]="Nov";m["12"]="Dec";
    printf("/data/htdocs/field_photos/files/by_site/%s/webcam/%s/webcam/%s",station,year,m[month])
  }' /dev/null
}

cd to_the_directory
find . -name "P*.jpg" -type f | while read f 
do
  d=`f2p ${i#*/}`
  if [ ! -d $d ]; then
    mkdir -p $d
  fi
  mv $i $d
done

Last edited by chihung; 02-01-2012 at 10:56 PM..
# 5  
Old 02-02-2012
You're running awk 1 million separate times, to process 1 million separate lines. That's as wasteful as making 1 million phone calls to say 1 million words, slow and pointless. It is an external program that takes time to load and quit. You should run awk once to process 1 million lines.

Your awk code can be refactored easily enough fortunately. Perhaps:

Code:
find . -name "P*.jpg" -type f | awk 'BEGIN {
    m["01"]="Jan"; m["02"]="Feb";m["03"]="Mar";m["04"]="Apr";
    m["05"]="May"; m["06"]="Jun";m["07"]="Jul";m["08"]="Aug";
    m["09"]="Sep"; m["10"]="Oct";m["11"]="Nov";m["12"]="Dec"; }
{
    station=substr($0,1,4)
    year=substr($0,5,4)
    month=substr($0,9,2)
    printf("%s\t/data/htdocs/field_photos/files/by_site/%s/webcam/%s/webcam/%s",$0,station,year,m[month]) }' |
while read ORIG NEW
do
        echo mv "$ORIG" "$NEW"
done

For each file, awk prints the original location, a tab, then the new location. This is fed into a shell loop which reads both and moves the file. Remove the 'echo' once you've tested and seen that it does what you want.
Last edited by Corona688; 02-02-2012 at 02:36 PM..
This User Gave Thanks to Corona688 For This Post:
chihung
# 6  
Old 02-02-2012
Corona688, many thanks for the performance tips. I think we still need to test if the directory exists within the loop before the 'mv'
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Cat files listed in text file and redirect to new directory with same filename

I have a directory that is restricted and I cannot just copy the files need, but I can cat them and redirect them to a new directory. The files all have the date listed in them. If I perform a long listing and grep for the date (150620) I can redirect that output to a text file. Now I need to... (5 Replies)
Discussion started by: trigger467
5 Replies

2. Shell Programming and Scripting

How to put a date from last week into a filename?

I have written three database queries to extract data from yesterday, last week and last month. I need a way to name the output files something like this : "Daily 2014-05-21", "Weekly 2014-05-19" & "Monthly 2014-05-01" if I run them today ( 2014-05-22 ). I can specify the output file name from... (5 Replies)
Discussion started by: djohnson123
5 Replies

3. Shell Programming and Scripting

extract every filename containing certain string in a directory and do some commend per file

Hi, Here is my question: suppose I have files like 1990_8xdaily_atmos.nc 1991_8xdaily_atmos.nc 1992_8xdaily_atmos.nc 1993_8xdaily_atmos.nc 1990_daily_atmos.nc 1991_daily_atmos.nc 1992_daily_atmos.nc 1993_daily_atmos.nc 1990_month_atmos.nc 1991_month_atmos.nc 1992_month_atmos.nc... (1 Reply)
Discussion started by: 1988PF
1 Replies

4. Shell Programming and Scripting

Diff - filename and directory name are same

Hi, I have in the one folder file and directory that have same name. I need make diff from first directory where exists file in folder FOLDER/filename and second file where not exist folder, but FOLDER is filename. I use -N switch for create new file. Scripts report: Not a directory Sample:... (2 Replies)
Discussion started by: tomix
2 Replies

5. Shell Programming and Scripting

put working directory in file

how to put pwd in my file my working directory is /usr/my_dir below my file aaaaaa bbbbbb so output file become /usr/my_dir aaaaaa bbbbbb (2 Replies)
Discussion started by: zulabc
2 Replies

6. Shell Programming and Scripting

Filename from splitting files to have the same filename of the original file with counter value

Hi all, I have a list of xml file. I need to split the files to a different files when see the <ko> tag. The list of filename are B20090908.1100-20090908.1200_CDMA=1,NO=2,SITE=3.xml B20090908.1200-20090908.1300_CDMA=1,NO=2,SITE=3.xml B20090908.1300-20090908.1400_CDMA=1,NO=2,SITE=3.xml ... (3 Replies)
Discussion started by: natalie23
3 Replies

7. UNIX for Dummies Questions & Answers

Cant find stdint.h? What directory do i put it in?

Hello everyone, I've got a C program that I'm trying to run through my unix shell but whenever i compile it using gcc (using "cc test.c"), it cannot find stdint.h. Right now, i have it in the same directory as the program I'm running, test.c. Any suggestions? (3 Replies)
Discussion started by: TeamUSA
3 Replies

8. Shell Programming and Scripting

Put header if file exists in a directory

Hi all, I will have to check a directory which is full of files. If any file more than 5 MB of space exists, write on it's head "Checked". Please help me. Thanks in advance! Deepak (5 Replies)
Discussion started by: naw_deepak
5 Replies

9. Shell Programming and Scripting

how to get filename from directory

hi find /home -type f -atime +5 shows me files like : home/test/aaa.txt home/test/bbb.html How can I get the file name I mean only aaa.txt and bbb.html (19 Replies)
Discussion started by: tjay83
19 Replies

10. Shell Programming and Scripting

how to test filename is file or directory

hi all plz let me how to test output of "tail -1 filelist.lst" is file or directory. regards -Bali Reddy (1 Reply)
Discussion started by: balireddy_77
1 Replies
Login or Register to Ask a Question