Parallel move keeping folder structure along with files in it


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Parallel move keeping folder structure along with files in it
# 1  
Old 06-16-2017
Parallel move keeping folder structure along with files in it

The below will move all the files in the directory dir to the destination using parallel and create a log, however will not keep them in the directory. I have tried mkdir -p but that does not seem to work or at least I can not seem to get it (as it deletes others files when I use it). What is the correct way to move the 3 files into the same directory? Thank you Smilie.

Code:
dir="/home/cmccabe/Downloads/snp"
cd "$dir"
parallel -j10 --progress mv -v {} /home/cmccabe/Desktop/indel/ ::: * > /home/cmccabe/medex.logs/archive.log

Folder structure:
Code:
FolderName   --- dir
|
file1
file2
file3

Desired after move:
Code:
FolderName
|
3 files in it

# 2  
Old 06-16-2017
Your hard drive does not work in parallel. There is no point or benefit in running cp, mv, rsync, rm, rmdir, etc in parallel. They will still have to wait for each other. Trying to run them in parallel will actually slow it down and cause fragmentation. Don't do it.

Running tar in parallel - as in your other thread - only worked because bzip2 is so slow that the disk was waiting for the CPU, not vice versa. That's an unusual situation and not at all true of basic file utilities.

Last edited by Corona688; 06-16-2017 at 06:23 PM..
This User Gave Thanks to Corona688 For This Post:
# 3  
Old 06-16-2017
Thank you very much Smilie.
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Request for Shell script to move files from Subfolder to Parent folder and delete sub folder

Hi Team, I am new to shell script and there is a requirement where files should be moved from Subfolder to parent folder. Eg: parent folder --> /Interface/data/test/IN Sub folder -->/Interface/data/test/IN/Invoice20180607233338 Subfolder will be always with timestamp... (6 Replies)
Discussion started by: srivarun15
6 Replies

2. Shell Programming and Scripting

Script to move files and Creation of folder structure

We are receiving few zipped files in one location say : apple/oranges/incoming All .zip files are placed here in incoming folder. So few of the files are password encrypted. There are only 10 zipped files, so we are planning to create a script which will pick that zip file from incoming... (1 Reply)
Discussion started by: Sidhant
1 Replies

3. Shell Programming and Scripting

How to move the files older than x days with similar directory structure?

Hello, I need to move all the files inside /XYZ (has multi-depth sub directories) that are older than 14 days to/ABC directory but with retaining the SAME directory structure. for example: /XYZ/1/2/3/A/b.txt should be moved as /ABC/1/2/3/A/b.txt I know about find /XYZ -type f -mtime +14... (3 Replies)
Discussion started by: prvnrk
3 Replies

4. Shell Programming and Scripting

Move files from Space Folder to other folder

I want to move a folder with spaces from one folder to another. I have two folders like this, 1).RT_032-222 -4444-01/ 2). RT_032-555 -7777-01/ I want to move files from 2 to 1 through shell script.Here I want to assign this like a user defined variable like as Source branch... (2 Replies)
Discussion started by: kannansoft1985
2 Replies

5. Shell Programming and Scripting

Copying files from various folders to similar folder structure in another location

Hi, I need to write a script the has to copy the files from folders and subfolders to the same folder structure located in another location. Ex: mainfolder1 file1,file2,file3 subfolder1(file1,etc) subfolder2(file1,etc) to another folder location of same folder structure. rsync is not... (7 Replies)
Discussion started by: Raji Perumal
7 Replies

6. Shell Programming and Scripting

Archiving files keeping the same structure directory

Hello Team, We would like to backup a lot of files inside of a structure of directories, four, five or more levels in some Ubuntu, Mac and Solaris systems. For instance: /home/chuck/sales/virgin/rent-quote.pdf /home/chuck/sales/marriott/vacation-quote.pdf... (2 Replies)
Discussion started by: csierra
2 Replies

7. Shell Programming and Scripting

Need to Zip files three years old or longer but leave folder structure intact

Hi all, Hello everyone, my first post here :). I tried to search the forum but I didn't find exactly what I was looking for... I need to zip/tar files across entire filesystem which are more 3+ years old but leave folder structure intact. If the script locates tar/zip files they are more... (1 Reply)
Discussion started by: sashruby
1 Replies

8. Shell Programming and Scripting

Move files & folder structure

Hey, this might be a really basic question, but I'm new to Unix scripting. I'm trying to find a command to replicate a file structure from one location to another & move the actual files contained in the base directories, i.e. I have this structure - home/temp/test/dir1/ ... (3 Replies)
Discussion started by: SOCLA_CELT
3 Replies

9. Shell Programming and Scripting

File Management: How do I move all JPGS in a folder structure to a single folder?

This is the file structure: DESKTOP/Root of Photo Folders/Folder1qweqwasdfsd/*jpg DESKTOP/Root of Photo Folders/Folder2asdasdasd/*jpg DESKTOP/Root of Photo Folders/Folder3asdadfhgasdf/*jpg DESKTOP/Root of Photo Folders/Folder4qwetwdfsdfg/*jpg DESKTOP/Root of Photo... (4 Replies)
Discussion started by: guptaxpn
4 Replies

10. Shell Programming and Scripting

move files and retain subdir structure

hi: I have some files like this folder1/recording1.mp3 folder1/docs/budget.doc folder2/others/misc.mp3 folder3/others/notes.doc all this folders and files are under the mp3 folder. I would like to move just the mp3s to another folder but retain the subdir structure i have. So if... (4 Replies)
Discussion started by: jason7
4 Replies
Login or Register to Ask a Question