Sponsored Content
Top Forums Shell Programming and Scripting Split a folder with huge number of files in n folders Post 302906661 by AlokKumbhare on Sunday 22nd of June 2014 05:08:05 PM
Old 06-22-2014
Quote:
Originally Posted by Don Cragun
What OS are you using?

Do any of the 350,000 files you're moving have names that contain any space, tab, or newline characters?
Hi Don,

I am on Ubuntu Server 12.04 LTS. The files do not contain any special characters except _

Actually the format is something like

<xyz>_n1_n2_n3.pbx

where <xyz> is a prefix containing only alphabets [a-z A-Z]
n1 is a number less than 10
n2 is a number less than 1000
n3 is a number less than 10000
pbx is the extension

---------- Post updated at 04:08 PM ---------- Previous update was at 04:06 PM ----------

Quote:
Originally Posted by MadeInGermany
Aren't 350 000 files too many arguments for for i in *?
Safer and faster is
Code:
find . -type f |
while read i

Thanks.. Although I did not get an error for too many files with for I in *, find seems faster and as you mentioned safer if the number of files increase.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

delete all folders/files and keep only the last 10 in a folder

Hi, I want to write a script that deletes all folders and keep the last 10 recent folders. I know the following: ls -ltr will sort the folders from old to recent. ls -ltr | awk '{print $9}' will list the folder names (with a blank line at the beginning) I want to get the 10th folder from... (3 Replies)
Discussion started by: melanie_pfefer
3 Replies

2. Shell Programming and Scripting

Split a huge data into few different files?!

Input file data contents: >seq_1 MSNQSPPQSQRPGHSHSHSHSHAGLASSTSSHSNPSANASYNLNGPRTGGDQRYRASVDA >seq_2 AGAAGRGWGRDVTAAASPNPRNGGGRPASDLLSVGNAGGQASFASPETIDRWFEDLQHYE >seq_3 ATLEEMAAASLDANFKEELSAIEQWFRVLSEAERTAALYSLLQSSTQVQMRFFVTVLQQM ARADPITALLSPANPGQASMEAQMDAKLAAMGLKSPASPAVRQYARQSLSGDTYLSPHSA... (7 Replies)
Discussion started by: patrick87
7 Replies

3. Shell Programming and Scripting

Compare 2 folders to find several missing files among huge amounts of files.

Hi, all: I've got two folders, say, "folder1" and "folder2". Under each, there are thousands of files. It's quite obvious that there are some files missing in each. I just would like to find them. I believe this can be done by "diff" command. However, if I change the above question a... (1 Reply)
Discussion started by: jiapei100
1 Replies

4. Shell Programming and Scripting

Move all files but not folders to a new folder

Hi, I have a sub directory with a number of files and folders. What i want is a subdirectory with just folders and not files for cleanliness sake. So I want to move the files into the new folder but keep the folders in the same place. Move all files (but not folders) to new folder. I am... (4 Replies)
Discussion started by: Hopper_no1
4 Replies

5. Shell Programming and Scripting

How to delete a huge number of files at a time

I met a problem on HPUX with 64G RAM and 20 CPU. There are 5 million files with file name from file0000001.dat to file9999999.dat, in the same directory, and with some other files with random names. I was trying to remove all the files from file0000001.dat to file9999999.dat at the same time.... (9 Replies)
Discussion started by: lisp21
9 Replies

6. Shell Programming and Scripting

Help- counting delimiter in a huge file and split data into 2 files

I’m new to Linux script and not sure how to filter out bad records from huge flat files (over 1.3GB each). The delimiter is a semi colon “;” Here is the sample of 5 lines in the file: Name1;phone1;address1;city1;state1;zipcode1 Name2;phone2;address2;city2;state2;zipcode2;comment... (7 Replies)
Discussion started by: lv99
7 Replies

7. Shell Programming and Scripting

moving files from one folder to many folders

I have a more than 10 K files in a folder. They are accumulated in a period of more than an year (Say from 13th July 2010 to 4th June 2011). I need to perform housekeeping on this. The requirement is to create a folder like 13Jul2010,14July2010,......3June2011,4June2010 and then from the main... (2 Replies)
Discussion started by: realspirituals
2 Replies

8. Shell Programming and Scripting

Symlink all files from one folder into all found folders

Hi. I have a folder which contains my application. I then have a flexible number of folders in another directory, called “sites”. It looks like this: -Application -- Test.html -- CSS --- Style.css -Sites --Site1 --Site2 I want to symlink all the files in the application folder... (1 Reply)
Discussion started by: Spadez
1 Replies

9. UNIX for Dummies Questions & Answers

Split a huge 7 GB File Based on Pattern into 4 files

Hi, I have a Huge 7 GB file which has around 1 million records, i want to split this file into 4 files to contain around 250k messages each. Please help me as Split command cannot work here as it might miss tags.. Format of the file is as below <!--###### ###### START-->... (6 Replies)
Discussion started by: KishM
6 Replies

10. Shell Programming and Scripting

Moving files and folders to another folder

I recently bought Synology server and realised it can run scripts. I would need fairly simple script which moves all files and folders from ARCHIVE folder to WORKING folder. I would also need to maintain folder structure as each of the folders may contain subfolders. How would I go about it as I am... (1 Reply)
Discussion started by: ###
1 Replies
refile(1mh)															       refile(1mh)

Name
       refile - file message in other folders

Syntax
       refile [ msgs ] [ +folder ] [ options ]

Description
       Use  the  command to move the specified message from the current folder to another folder.  You can refile messages in more than one folder
       by giving multiple folder names as arguments.

       If you do not specify a message, the current message is refiled.  You can refile a message other than the current  message  by  giving  its
       number  as  a msgs argument.  You can also refile more than one message at a time by specifying more than one message number, or a range of
       message numbers, or a message sequence.	See for more information on sequences.

       The current folder remains the same unless the -src option is specified; in that case, the source folder becomes  current.   Normally,  the
       last message specified becomes the current message.  However, if the -link option is used, the current message is not changed.

       If  the	Previous-Sequence:  entry is set in the file, in addition to defining the named sequences from the source folder, will also define
       those sequences for the destination folders.  See for information concerning the previous sequence.

Options
       -draft	 Refiles the draft message, or the current message in your folder, if you have one set up.  You cannot give a msgs  argument  when
		 you use this option.

       -file filename
		 Moves	a  file into a folder.	This option takes a file from its directory and places it in the named folder, as the next message
		 in the folder.  The file must be formatted as a legal mail message.  This means that the message must	have  the  minimum  header
		 fields separated from the body of the message by a blank line or a line of dashes.

       -help	 Prints a list of the valid options to this command.

       -link
       -nolink	 Keeps	a copy of the message in the source folder.  Normally, removes the messages from the original folder when it refiles them.
		 The -link option keeps a copy in the original folder, as well as filing a copy in the new folder.

       -preserve
       -nopreserve
		 Preserves the number of a message in the new folder.  Normally, when a message is refiled in to another folder, it is set to  the
		 next  available number in that folder.  The -preserve option keeps the number of the message the same in the new folder as it had
		 been in the old.

		 You cannot have two messages with the same number in one folder, so you should use this option with care.

       -src +folder
		 Specifies the source folder to take messages from.  Normally, messages are refiled from the current folder into  another  folder.
		 However, you can take messages from a different folder by using the -src +folder option to specify the alternative source folder.

Examples
       The following example refiles messages 3 and 5 in the folder
       % refile 3 5 +records

       The next example files the current message into two folders:
       % refile +jones +map

       The next example takes message 13 in the current folder and refiles it in the folder.  The message remains in the current folder as well as
       appearing in the folder.
       % refile -link 13 +test

       The next example takes a message from the folder when it is not the current folder, and places it in the folder
       % refile 3 -src +test +outbox

Profile Components
       Path:		 To determine your Mail directory

       Folder-Protect:	 To set protections when creating a new folder
       rmmproc: 	 Program to delete the message

Files
       The user profile.

See Also
       folder(1mh), mark(1mh), mh_profile(5mh)

																       refile(1mh)
All times are GMT -4. The time now is 09:14 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy