Sponsored Content
Top Forums Shell Programming and Scripting Split a folder with huge number of files in n folders Post 302906661 by AlokKumbhare on Sunday 22nd of June 2014 05:08:05 PM
Old 06-22-2014
Quote:
Originally Posted by Don Cragun
What OS are you using?

Do any of the 350,000 files you're moving have names that contain any space, tab, or newline characters?
Hi Don,

I am on Ubuntu Server 12.04 LTS. The files do not contain any special characters except _

Actually the format is something like

<xyz>_n1_n2_n3.pbx

where <xyz> is a prefix containing only alphabets [a-z A-Z]
n1 is a number less than 10
n2 is a number less than 1000
n3 is a number less than 10000
pbx is the extension

---------- Post updated at 04:08 PM ---------- Previous update was at 04:06 PM ----------

Quote:
Originally Posted by MadeInGermany
Aren't 350 000 files too many arguments for for i in *?
Safer and faster is
Code:
find . -type f |
while read i

Thanks.. Although I did not get an error for too many files with for I in *, find seems faster and as you mentioned safer if the number of files increase.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

delete all folders/files and keep only the last 10 in a folder

Hi, I want to write a script that deletes all folders and keep the last 10 recent folders. I know the following: ls -ltr will sort the folders from old to recent. ls -ltr | awk '{print $9}' will list the folder names (with a blank line at the beginning) I want to get the 10th folder from... (3 Replies)
Discussion started by: melanie_pfefer
3 Replies

2. Shell Programming and Scripting

Split a huge data into few different files?!

Input file data contents: >seq_1 MSNQSPPQSQRPGHSHSHSHSHAGLASSTSSHSNPSANASYNLNGPRTGGDQRYRASVDA >seq_2 AGAAGRGWGRDVTAAASPNPRNGGGRPASDLLSVGNAGGQASFASPETIDRWFEDLQHYE >seq_3 ATLEEMAAASLDANFKEELSAIEQWFRVLSEAERTAALYSLLQSSTQVQMRFFVTVLQQM ARADPITALLSPANPGQASMEAQMDAKLAAMGLKSPASPAVRQYARQSLSGDTYLSPHSA... (7 Replies)
Discussion started by: patrick87
7 Replies

3. Shell Programming and Scripting

Compare 2 folders to find several missing files among huge amounts of files.

Hi, all: I've got two folders, say, "folder1" and "folder2". Under each, there are thousands of files. It's quite obvious that there are some files missing in each. I just would like to find them. I believe this can be done by "diff" command. However, if I change the above question a... (1 Reply)
Discussion started by: jiapei100
1 Replies

4. Shell Programming and Scripting

Move all files but not folders to a new folder

Hi, I have a sub directory with a number of files and folders. What i want is a subdirectory with just folders and not files for cleanliness sake. So I want to move the files into the new folder but keep the folders in the same place. Move all files (but not folders) to new folder. I am... (4 Replies)
Discussion started by: Hopper_no1
4 Replies

5. Shell Programming and Scripting

How to delete a huge number of files at a time

I met a problem on HPUX with 64G RAM and 20 CPU. There are 5 million files with file name from file0000001.dat to file9999999.dat, in the same directory, and with some other files with random names. I was trying to remove all the files from file0000001.dat to file9999999.dat at the same time.... (9 Replies)
Discussion started by: lisp21
9 Replies

6. Shell Programming and Scripting

Help- counting delimiter in a huge file and split data into 2 files

I’m new to Linux script and not sure how to filter out bad records from huge flat files (over 1.3GB each). The delimiter is a semi colon “;” Here is the sample of 5 lines in the file: Name1;phone1;address1;city1;state1;zipcode1 Name2;phone2;address2;city2;state2;zipcode2;comment... (7 Replies)
Discussion started by: lv99
7 Replies

7. Shell Programming and Scripting

moving files from one folder to many folders

I have a more than 10 K files in a folder. They are accumulated in a period of more than an year (Say from 13th July 2010 to 4th June 2011). I need to perform housekeeping on this. The requirement is to create a folder like 13Jul2010,14July2010,......3June2011,4June2010 and then from the main... (2 Replies)
Discussion started by: realspirituals
2 Replies

8. Shell Programming and Scripting

Symlink all files from one folder into all found folders

Hi. I have a folder which contains my application. I then have a flexible number of folders in another directory, called “sites”. It looks like this: -Application -- Test.html -- CSS --- Style.css -Sites --Site1 --Site2 I want to symlink all the files in the application folder... (1 Reply)
Discussion started by: Spadez
1 Replies

9. UNIX for Dummies Questions & Answers

Split a huge 7 GB File Based on Pattern into 4 files

Hi, I have a Huge 7 GB file which has around 1 million records, i want to split this file into 4 files to contain around 250k messages each. Please help me as Split command cannot work here as it might miss tags.. Format of the file is as below <!--###### ###### START-->... (6 Replies)
Discussion started by: KishM
6 Replies

10. Shell Programming and Scripting

Moving files and folders to another folder

I recently bought Synology server and realised it can run scripts. I would need fairly simple script which moves all files and folders from ARCHIVE folder to WORKING folder. I would also need to maintain folder structure as each of the folders may contain subfolders. How would I go about it as I am... (1 Reply)
Discussion started by: ###
1 Replies
ROOTTAIL(1)						      General Commands Manual						       ROOTTAIL(1)

NAME
root-tail - print text directly to X11 root window SYNOPSIS
root-tail [-g|--geometry GEOMETRY] [-fn|--font FONTSPEC] [--color color] [--reload SEC COMMAND] [--shade] [--outline] [--minspace] [--noflicker] [-f|--fork] [--reverse] [--whole] [--partial] [--update] [--cont STRING] [--wordwrap] [--justify] [--noinitial] [--frame] [-id ID] [-i|--interval SECONDS] [-V] file1[,color[,desc]] [file2[,color[,desc]]] DESCRIPTION
Displays a given file anywhere on your X11 root window, i.e. it is kind of tail -f for multiple files using your desktop background as out- put window. All non-option arguments on the command line are files to be logged. A null desc (example: "/var/log/messages,red,") will prevent the printing of a description and the []'s. General Options --color COLOR Use COLOR as default. --font | -fn FONTSPEC Use font FONTSPEC. This can be either a fixed width font like -fn fixed or any font using -fn '-*-*-*-*-*-*-*-*-*-*-*-*-*-*' with the appropriate fields filled out (see xfontsel). Specifying a different FONTSPEC before each filename will cause each file to be displayed in a different font. -f | --fork Forks to the background. --reverse Display the files in reverse order, with the newest lines at the top. --whole Only display whole lines. If the last line of a file doesn't yet end with a newline character then wait until it does before dis- playing it. This is the default if more than one file is being displayed. --partial This is the opposite of the --whole option (see above). It displays lines even if they don't yet have a newline at the end. This is the default if only one file is being displayed. --update Update partial lines 'in place' if they are still on the screen when they are updated in their files. Using --update automatically turns on --partial. --cont STRING When partial lines are broken into two lines in the display, prefix the 2nd line with STRING. Defaults to "|| ". Specify the "--whole" argument to ensure partial lines are never displayed, or specify "--update" to attempt to "repair" broken lines in-place. --cont-color COLOR Use COLOR when displaying the continuation string (as optionally specified with the --cont option above). --wordwrap The default behaviour is to fit as much as possible onto each line of output, even if this means splitting a word between one line and the next. The --wordwrap argument splits lines at spaces if possible. --justify After wrapping long lines, attempt to justify the text to produce a smooth right-hand margin. Implies --wordwrap. --reload SEC COMMAND Re-display the file(s) and run COMMAND every SEC seconds. The default is to never re-display the file(s). --shade Add black shading to the font. --outline Add a black outline to the font (making redraws quite a bit slower). --minspace Use minimum linespace even when using shading or outlining. This might result in leftover pixels (dependign on font and logfile con- tent). --noflicker Use slower but flicker-free update. --noinitial Don't display the end of the file(s) initially. -id ID Use the given window ID for output instead of the root window. -i | --interval SECONDS Use the specified sleeping interval between checks instead of the default 2.4 seconds. Fractional values are OK. -V Print version information. --frame Draw a frame around the selected area. This is useful when trying to find the perfect geometry. EXAMPLE
root-tail -g 800x250+100+50 -font 10x20 /var/log/messages,green -font 12x24 /var/log/secure,red,'ALERT' BUGS
Some desktop environments open a virtual root window and make it difficult to share it. If you cannot see anything after starting root- tail, try to find a setting "allow programs on desktop" or similar, or manually specify a window id. Should you happen to find any bugs please fix them and send me a diff. NOTE: This program was modified by Marc Lehmann <pcg@goof.com>, who couldn't reach the original author. Please direct bug-reports etc. to pcg@goof.com. http://root-tail.plan9.de/ 4.2 Berkeley Distribution 2004-03-27 ROOTTAIL(1)
All times are GMT -4. The time now is 07:42 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy