Move files with a certain suffix based on how many files are in another folder


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Move files with a certain suffix based on how many files are in another folder
# 1  
Old 08-31-2018
Move files with a certain suffix based on how many files are in another folder

Hello,


First time poster. I am looking for a way to script or program the process of moving files from one folder to another, automatically, based on the count of files in the destination folder.


I was thinking a shell script would work, but am open to the suggestions of the experts here.


Files I am trying to move:
UNIQUEID.rmdcall


Files start at:
/tmp


Files need to go to:
/tmp/test


Details:
1. If the count of .rmdcall files in /tmp is greater than 0, then count the .rmdcall files in /tmp/test, otherwise do nothing.


2. If the count of .rmdcalls in /tmp/test is less than 30, then move up to 100 of the oldest .rmdcall files from /tmp to /tmp/test, otherwise do nothing


3. Repeat the process every 5 seconds


Thanks for any help you can provide, it is greatly appreciated!
# 2  
Old 08-31-2018
Welcome to the UNIX & Linux Forums.

Homework and coursework questions can only be posted in this forum under special homework rules.

Please review the rules, which you agreed to when you registered, if you have not already done so.

If you did not post homework, please explain what the problem you have described is intended to do. Needing to move 1200 files per minute from one directory to another is not a common requirement. Why don't you just have whatever is creating these files in /tmp create them in /tmp/test instead?

Furthermore, always tell us what operating system and shell you're using when posting requests in the Shell Programming and Scripting forum and show us what you have tried to solve this problem on your own.

If you did post homework in the main forums, please review the guidelines for posting homework and repost.
# 3  
Old 09-06-2018
Hello,


This is not homework, it is an idea for a more automated workflow. We have a folder (/tmp/test) that an application listens to. When a file is moved into that folder, it processes and deletes the file. Basically I have work to do in the /tmp files and I need to throttle how much I send to the application /tmp/test so I do not crash the application. This is done manually today (someone moves the files to /tmp/test periodically). I am hoping we can automate this to save time.



  • It is not that 1,200 files per minute will move but up to 1,200 files may move (if they exist).


  • There is another process in /tmp/test that processes any file dropped in that folder. The problem is that I cannot move all the files at once or I risk overloading the workflow. The goal is to systematically throttle the files coming into /tmp/test to avoid this.
Linux Version: Sangoma Linux release 7.4.1710 (ID_LIKE="centos rhel fedora")
Shell Version: GNU bash, version 4.2.46(2)-release (x86_64-redhat-linux-gnu)


I don't have a lot of experience in scripting, but I know I need the mv command to move the files. I think I need the touch command to dictate when to move the files.


I also know I can get file counts by using the ls /tmp/*.rmdcall |wc -l command.


I am just not sure how to string these all together, and time is currently not on my side. Any insights? Thanks in advance for any help you can provide.
# 4  
Old 09-06-2018
Untested starting point?

Code:
#!/bin/bash                                                                              

src="/tmp"
dst="/tmp/test"
threshold=30
 
file_count=$(ls ${dst}/*.rmdcall 2>/dev/null | wc -l)
 
if [ $file_count -lt $threshold ]; then
    transfer=100
  
    for f in ${src}/*.rmdcall; do
        if [ $transfer -eq 0 ]; then
            break
        fi  
        [ -e "$f" ] && mv -v "$f" "${dst}/" || break
        transfer=$((transfer -1))
    done
fi


Last edited by Aia; 09-06-2018 at 01:22 PM..
# 5  
Old 09-06-2018
Thanks, I'll give it a go and see how it goes.
# 6  
Old 09-06-2018
How to identify the oldest files - by modification time stamp? By UniqueID (if in increasing order)?




EDIT: OK, let's postpone the file identification question for now. I'd prefer a different approach. Your file names seem not to contain any white spaces, so deploying xargs seems safe.
After having determined the file count in /tmp/test having dropped below 30 just like Aia proposed, how about



Code:
ls /tmp/*.rmdcall | head -100 | xargs echo mv -t /tmp/test

? Should your system's config parameter LINE_MAX be exceeded with the resulting command line, add the -n option to xargs like e.g. -n50. Remove echo when happy with the proposed result.

Last edited by RudiC; 09-06-2018 at 01:29 PM..
This User Gave Thanks to RudiC For This Post:
# 7  
Old 09-11-2018
Thanks all, that seemed to work!
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Request for Shell script to move files from Subfolder to Parent folder and delete sub folder

Hi Team, I am new to shell script and there is a requirement where files should be moved from Subfolder to parent folder. Eg: parent folder --> /Interface/data/test/IN Sub folder -->/Interface/data/test/IN/Invoice20180607233338 Subfolder will be always with timestamp... (6 Replies)
Discussion started by: srivarun15
6 Replies

2. Shell Programming and Scripting

Move files from Space Folder to other folder

I want to move a folder with spaces from one folder to another. I have two folders like this, 1).RT_032-222 -4444-01/ 2). RT_032-555 -7777-01/ I want to move files from 2 to 1 through shell script.Here I want to assign this like a user defined variable like as Source branch... (2 Replies)
Discussion started by: kannansoft1985
2 Replies

3. Solaris

Move files to another folder base on DU

Hi, I want to move files like *.txt to another filesystem on the same server only when the disk usage reaches 80% or more. But need to keep the latest 5 files. After that delete from the original. How to proceed? Please help Gav... (5 Replies)
Discussion started by: Gavisht
5 Replies

4. UNIX for Dummies Questions & Answers

deleting files based on the suffix

how do we delete files based on the suffix??? (1 Reply)
Discussion started by: saggiboy10
1 Replies

5. Shell Programming and Scripting

Move only files to a folder

I want to move all the files inside a directory to another directory which is inside that directory. Please help. Note: I want to move only the files. Note the folders. Thanks in advance. (4 Replies)
Discussion started by: brnl_basu
4 Replies

6. UNIX and Linux Applications

Move Files From One Folder To Another In UNIX

There are around 13 files in folder1.I need to move these files to another folder folder2,one file at a time, after checking whether a file exists in another folder folder3. If a file exists in folder3, we should not move files from folder1 to folder2. If there are no files in folder3, we need... (1 Reply)
Discussion started by: Jassz
1 Replies

7. Shell Programming and Scripting

Move all files but not folders to a new folder

Hi, I have a sub directory with a number of files and folders. What i want is a subdirectory with just folders and not files for cleanliness sake. So I want to move the files into the new folder but keep the folders in the same place. Move all files (but not folders) to new folder. I am... (4 Replies)
Discussion started by: Hopper_no1
4 Replies

8. Shell Programming and Scripting

Move files & folder structure

Hey, this might be a really basic question, but I'm new to Unix scripting. I'm trying to find a command to replicate a file structure from one location to another & move the actual files contained in the base directories, i.e. I have this structure - home/temp/test/dir1/ ... (3 Replies)
Discussion started by: SOCLA_CELT
3 Replies

9. Shell Programming and Scripting

move files to some folder

Hello folk, I want to move some *.jpg and *.JPG to some folder but i want move from current path not from subdirectories, wants to move 200 days old file. it will go in /dir/aa/target/*.jpg|*.JPG but not go inside /dir/aa/target/other-directories, Plese help (2 Replies)
Discussion started by: learnbash
2 Replies

10. UNIX for Dummies Questions & Answers

move files from folder thats are not empty

Hi, I would like to write a shell script that moves files from one folder to another without retrieving the error 'can not find file or folder' when the folder is empty. Any ideas, Thx in advance, Steven. (8 Replies)
Discussion started by: Steven
8 Replies
Login or Register to Ask a Question