Move file in to directory- script


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Move file in to directory- script
# 1  
Old 11-14-2014
Move file in to directory- script

Hi

In directory /mnt/upload I have about 100 000 files (*.png) that have been created during the last six months. Now I need to move them to right folders. eg:
Code:
file created on 2014-10-10 move to directory /mnt/upload/20141010
file created on 2014-11-11 move to directory /mnt/upload/20141111
file created on 2014-08-01 move to directory /mnt/upload/20140801

So the script first creates a directory with the correct name ( format +%Y%m%d) move all files in a day (but only files not directory) in to right directory.

any ideas ?

Last edited by jim mcnamara; 11-14-2014 at 04:52 PM..
# 2  
Old 11-14-2014
find them with the -mmin or -mtime options, and -exec mv {} them.
# 3  
Old 11-14-2014
Quote:
Originally Posted by primo102
any ideas ?
Yes, not a too efficient one though... Use it at your own risk. All commands are supposed to be run in the /mnt/upload directory.
A proper backup of those png files (with preserved modification timestamps) is highly recommended before attempting to perform the mass mv operation.

Code:
# get last modification date + filename
# sample output line: 2014-09-02 16:47:34.973268280 +0200 ./test.png
find . -maxdepth 1 -type f -name '*.png' -exec stat -c '%y %n' {} + >temp1.txt

Code:
# cut + reformat last modification date from YYYY-MM-DD to YYYYMMDD, cut filename
# sample output line: 20140902,test.png,
sed 's/^\(....\)-\(..\)-\(..\).*\/\(.*\)$/\1\2\3,\4,/' temp1.txt | sort >temp2.txt

Code:
# create neccessary directories
cut -d, -f1 temp2.txt | uniq | xargs mkdir

Code:
# assembling and executing mv commands
# sample output line: mv "test.png" 20140902
# try without "| sh" first to see if the output is correct!
awk -F, '{ printf "mv \"%s\" %s\n", $2, $1 }' temp2.txt | sh

Code:
# deleting temp files
rm temp[12].txt

# 4  
Old 11-14-2014
A rather more direct way -- have find print a list of files like

Code:
2014-01-02 /path/to/filename.png
2014-01-03 /path/to/anotherfile.png
...

...then read them into the shell to run with mv.

Code:
DEST=/path/to/

find /opt/camera/ -type f -printf '%TY%Tm%Td %p\n' |
while read -r DATE FILE
do
        [ -d "$DEST/$DATE" ] || mkdir -p "$DEST/$DATE"

        # Remove the 'echo' once you've tested and seen it does what you want.
        echo mv "$FILE" /path/to/$DATE/
done

I could make it more efficient, but the real inefficiency is going to be that one dir with 100,000 files, which there is absolutely no way to speed up... Each relink() has to lock and update that one huge folder.

Last edited by Corona688; 11-14-2014 at 05:22 PM..
# 5  
Old 11-14-2014
The big time cost is finding and removing them from the source dir. I suspect it might be fastest to move each file found in order, so the directory entry is the first one checked, or nearly so, each time a file is moved.

You might hard link them rather than move them, a sort of half move, no source directory rewriting but still a long search for file entries in there. Once they are all linked, you can delete all the old links in the source dir.

I'd use a find -ls and figure the destination from that in sed, not the time arguments.
# 6  
Old 11-14-2014
Thank's guys, but I am aleredy finished moving files (it took nearly six hours Smilie)

I made and run something like this:

Code:
#move signature
#!/bin/bash

cd /mnt/upload/
ls -pl1 --time-style='+%Y%m%d' | grep -v / | awk '/png/ {print $6,$7}' > name.txt
name=`cat name.txt | awk '{print $1}' | uniq`
while read line; do mkdir -p $line; done <<< "$name"
copy=`cat name.txt | awk '{print "/mnt/upload/"$2" /mnt/upload/"$1}'`
while read line; do mv $line; done <<< "$copy"
copy1=`echo $?`;
if [ $copy1 -eq 0 ]; then
echo OK > log.log
else 
echo ERROR > log.log
fi;

# 7  
Old 11-21-2014
Wow, needs some indentation and formatting for ease of maintenance! You, too, deserve pretty code. It's an investment in your future.

Moving all the files for one destination might have been an economy, using the mkdir -p line extended.
Code:
#!/bin/bash
 
cd /mnt/upload/
export d_last=''
 
ls -pl1 --time-style='+%Y%m%d' | awk '/-.*\.png$/{print $6,$7}' | ( sort ; echo x x) | while read d f
do
 if [ "$d" != "$d_last" ]
 then
 
  if [ $d_last != '' ]
  then
   mkdir -p /somewhere/$d_last
   mv $fs /somewhere/$d_last
  fi
 
  d_last=$d
  fs=$f
  continue
 fi 
 
 fs="$fs $f"
done

Note the need for a dummy last line to flush the last dir, and presetting the d_last to deal with the first line. Loops are neat except dealing with the ends.

Bash/ksh/awk/PERL have associative arrays that might be used in place of the sort with less loop ends stuff. All the file names would be stored under the date, and then at EOF you can go through the array key dates making dirs and moving files.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Shell script cannot create directory and move the file to that directory

I have a script, which is checking if file exists and move it to another directory if then mkdir -p ${LOCL_FILES_DIR}/cool_${Today}/monthly mv report_manual_alloc_rpt_A_I_ASSIGNMENT.${Today}*.csv ${LOCL_FILES_DIR}/cool_${Today}/monthly ... (9 Replies)
Discussion started by: digioleg54
9 Replies

2. UNIX for Dummies Questions & Answers

How to move file from one directory to other of only particular user?

I written unix script where a pdf file generates. But if the script is used by multiple people at a time it generates same pdf with two different owner names and creating a problem with permission while moving the file. is there a way where i can move the file filtering with the user? (4 Replies)
Discussion started by: lakers646
4 Replies

3. Shell Programming and Scripting

Script to move latest zip file to another directory

Hi folks, In my application there is a job running which create a .dat file along with it zip file also at unix box location /opt/app/cvf/temp1 so in temp1 directory I have one .dat file and its zip file also. Now since this job runs every day so if a job runs today there will be two files... (5 Replies)
Discussion started by: punpun66
5 Replies

4. UNIX for Dummies Questions & Answers

move a file content to another directory

my script is: /u/user/orginal/:#! /bin/ksh find . -name "aur_prog*" -exec grep -il "error" > test.out #awk command to destination directory exit test.out file contain: ./aur_prog1.log ./aur_prog2.log ... (5 Replies)
Discussion started by: roughwal
5 Replies

5. Programming

Script for creating a directory & move the .tif files in it.

Hi Team, I have thousands of TIF files which are converted from PDF. Below is a sample of it. LH9406_BLANCARAMOS_2012041812103210320001.tif LH9406_BLANCARAMOS_2012041812103210320002.tif LH9406_BLANCARAMOS_2012041812103210320003.tif LH9411_ANGENIAHUTCHINSON_2012041812102510250001.tif... (9 Replies)
Discussion started by: paragnehete
9 Replies

6. Shell Programming and Scripting

Script to move files to a directory according to date

hi all, here is the description to my problem. input parameter: $date1 based on the date i need to select three files starting with audit.log* based on its modified date, a date before, a date after(if its exists). We need to compare the input date to modified date of the file. And then... (3 Replies)
Discussion started by: ashrocks
3 Replies

7. Shell Programming and Scripting

Move the latest or older File from one directory to another Directory

I Need help for one requirement, I want to move the latest/Older file in the folder to another file. File have the datetimestamp in postfix. Example: Source Directory : \a destination Directory : \a\b File1 : xy_MMDDYYYYHHMM.txt (xy_032120101456.txt) File2: xy_MMDDYYYYHHMM.txt... (1 Reply)
Discussion started by: pp_ayyanar
1 Replies

8. Shell Programming and Scripting

A script that will move a file to a directory with the same name and then rename that file

Hello all. I am new to this forum (and somewhat new to UNIX / LINUX - I started using ubuntu 1 year ago).:b: I have the following problem that I have not been able to figure out how to take care of and I was wondering if anyone could help me out.:confused: I have all of my music stored in... (7 Replies)
Discussion started by: marcozd
7 Replies

9. Shell Programming and Scripting

Unix script to MOVE via FTP a directory

Good afternoon! I need to move a bunch of files from an FTP site to my server. However mget *, dele * will not suffice as I cannot guarantee that new files are not uploaded between commands. Roughly, this is what I would like to do: (using a .netrc) ftp somehost bin prompt cd... (1 Reply)
Discussion started by: abirdd
1 Replies

10. Shell Programming and Scripting

Move a file from windows directory to unix directory

Move a file from windows directory to unix directory, is this possible? if it is, can someone help me on this? Thanks! God bless! (1 Reply)
Discussion started by: kingpeejay
1 Replies
Login or Register to Ask a Question