Sponsored Content
Top Forums Shell Programming and Scripting Split a folder with huge number of files in n folders Post 302906805 by Scrutinizer on Monday 23rd of June 2014 03:32:48 PM
Old 06-23-2014
Another way to speed up could be to do the moves in the background:
Code:
cd XYZ || { echo "directory does not exist" >&2; exit 1 ;}
n=0
for i in *
do
  if [ $((n+=1)) -gt 20 ]; then
    n=1
    wait
  fi
  todir=../XYZ$n
  [ -d "$todir" ] || mkdir "$todir" 
  mv "$i" "$todir" &
done
wait

--
Probably the best way though would be to build randomly or serially selected lists of file names and feed them to mv operations to specific directories, while observing ARG_MAX
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

delete all folders/files and keep only the last 10 in a folder

Hi, I want to write a script that deletes all folders and keep the last 10 recent folders. I know the following: ls -ltr will sort the folders from old to recent. ls -ltr | awk '{print $9}' will list the folder names (with a blank line at the beginning) I want to get the 10th folder from... (3 Replies)
Discussion started by: melanie_pfefer
3 Replies

2. Shell Programming and Scripting

Split a huge data into few different files?!

Input file data contents: >seq_1 MSNQSPPQSQRPGHSHSHSHSHAGLASSTSSHSNPSANASYNLNGPRTGGDQRYRASVDA >seq_2 AGAAGRGWGRDVTAAASPNPRNGGGRPASDLLSVGNAGGQASFASPETIDRWFEDLQHYE >seq_3 ATLEEMAAASLDANFKEELSAIEQWFRVLSEAERTAALYSLLQSSTQVQMRFFVTVLQQM ARADPITALLSPANPGQASMEAQMDAKLAAMGLKSPASPAVRQYARQSLSGDTYLSPHSA... (7 Replies)
Discussion started by: patrick87
7 Replies

3. Shell Programming and Scripting

Compare 2 folders to find several missing files among huge amounts of files.

Hi, all: I've got two folders, say, "folder1" and "folder2". Under each, there are thousands of files. It's quite obvious that there are some files missing in each. I just would like to find them. I believe this can be done by "diff" command. However, if I change the above question a... (1 Reply)
Discussion started by: jiapei100
1 Replies

4. Shell Programming and Scripting

Move all files but not folders to a new folder

Hi, I have a sub directory with a number of files and folders. What i want is a subdirectory with just folders and not files for cleanliness sake. So I want to move the files into the new folder but keep the folders in the same place. Move all files (but not folders) to new folder. I am... (4 Replies)
Discussion started by: Hopper_no1
4 Replies

5. Shell Programming and Scripting

How to delete a huge number of files at a time

I met a problem on HPUX with 64G RAM and 20 CPU. There are 5 million files with file name from file0000001.dat to file9999999.dat, in the same directory, and with some other files with random names. I was trying to remove all the files from file0000001.dat to file9999999.dat at the same time.... (9 Replies)
Discussion started by: lisp21
9 Replies

6. Shell Programming and Scripting

Help- counting delimiter in a huge file and split data into 2 files

I’m new to Linux script and not sure how to filter out bad records from huge flat files (over 1.3GB each). The delimiter is a semi colon “;” Here is the sample of 5 lines in the file: Name1;phone1;address1;city1;state1;zipcode1 Name2;phone2;address2;city2;state2;zipcode2;comment... (7 Replies)
Discussion started by: lv99
7 Replies

7. Shell Programming and Scripting

moving files from one folder to many folders

I have a more than 10 K files in a folder. They are accumulated in a period of more than an year (Say from 13th July 2010 to 4th June 2011). I need to perform housekeeping on this. The requirement is to create a folder like 13Jul2010,14July2010,......3June2011,4June2010 and then from the main... (2 Replies)
Discussion started by: realspirituals
2 Replies

8. Shell Programming and Scripting

Symlink all files from one folder into all found folders

Hi. I have a folder which contains my application. I then have a flexible number of folders in another directory, called “sites”. It looks like this: -Application -- Test.html -- CSS --- Style.css -Sites --Site1 --Site2 I want to symlink all the files in the application folder... (1 Reply)
Discussion started by: Spadez
1 Replies

9. UNIX for Dummies Questions & Answers

Split a huge 7 GB File Based on Pattern into 4 files

Hi, I have a Huge 7 GB file which has around 1 million records, i want to split this file into 4 files to contain around 250k messages each. Please help me as Split command cannot work here as it might miss tags.. Format of the file is as below <!--###### ###### START-->... (6 Replies)
Discussion started by: KishM
6 Replies

10. Shell Programming and Scripting

Moving files and folders to another folder

I recently bought Synology server and realised it can run scripts. I would need fairly simple script which moves all files and folders from ARCHIVE folder to WORKING folder. I would also need to maintain folder structure as each of the folders may contain subfolders. How would I go about it as I am... (1 Reply)
Discussion started by: ###
1 Replies
LNDIR(1)						      General Commands Manual							  LNDIR(1)

NAME
lndir - create a shadow directory of symbolic links to another directory tree SYNOPSIS
lndir [ -silent ] [ -ignorelinks ] [ -withrevinfo ] fromdir [ todir ] DESCRIPTION
The lndir program makes a shadow copy todir of a directory tree fromdir, except that the shadow is not populated with real files but instead with symbolic links pointing at the real files in the fromdir directory tree. This is usually useful for maintaining source code for different machine architectures. You create a shadow directory containing links to the real source, which you will have usually mounted from a remote machine. You can build in the shadow tree, and the object files will be in the shadow directory, while the source files in the shadow directory are just symlinks to the real files. This scheme has the advantage that if you update the source, you need not propagate the change to the other architectures by hand, since all source in all shadow directories are symlinks to the real thing: just cd to the shadow directory and recompile away. The todir argument is optional and defaults to the current directory. The fromdir argument may be relative (e.g., ../src) and is relative to todir (not the current directory). Note that BitKeeper, CVS, CVS.adm, .git, .hg, RCS, SCCS, and .svn directories are shadowed only if the -withrevinfo flag is specified. Files with names ending in ~ are never shadowed. If you add files, simply run lndir again. New files will be silently added. Old files will be checked that they have the correct link. Deleting files is a more painful problem; the symlinks will just point into never never land. If a file in fromdir is a symbolic link, lndir will make the same link in todir rather than making a link back to the (symbolic link) entry in fromdir. The -ignorelinks flag changes this behavior. OPTIONS
-silent Normally lndir outputs the name of each subdirectory as it descends into it. The -silent option suppresses these status messages. -ignorelinks Causes the program to not treat symbolic links in fromdir specially. The link created in todir will point back to the corresponding (symbolic link) file in fromdir. If the link is to a directory, this is almost certainly the wrong thing. This option exists mostly to emulate the behavior the C version of lndir had in X11R6. Its use is not recommended. -withrevinfo Causes any source control manager subdirectories (those named BitKeeper, CVS, CVS.adm, .git, .hg, RCS, SCCS, or .svn) to be treated as any other directory, rather than ignored. DIAGNOSTICS
The program displays the name of each subdirectory it enters, followed by a colon. The -silent option suppresses these messages. A warning message is displayed if the symbolic link cannot be created. The usual problem is that a regular file of the same name already exists. If the link already exists but doesn't point to the correct file, the program prints the link name and the location where it does point. SEE ALSO
ln(1). X Version 11 lndir 1.0.3 LNDIR(1)
All times are GMT -4. The time now is 04:01 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy