Login or Register to Ask a Question and Join Our Community


UNIX for Dummies Questions & Answers

Take 100MB worth files from 200,000 Files

 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Take 100MB worth files from 200,000 Files
# 1  
Old 01-17-2013
Take 100MB worth files from 200,000 Files
Hi,
I have a process which creates almost 200K files. Each file ranging from 1kb to 5kb. This is almost 2GB of data in all files.
I have a requirement where the business needs only 100MB worth of files.
Is there a way to get files around 100MB (doesn't have to be exactly 100MB) from all the files.
Files can be random. All the filenames start with currentl datetime stamp.
for example:
Code:
20130116211758abc.txt
20130116211758def.txt 
etc

Any help would be much appreciated.
# 2  
Old 01-17-2013
That comes down to the application. Does it actually need all these files? If you can just zip them up, text compresses really well.

If the application needs all the files to be exactly where they are to operate, you can't really zip them up to save space.

The ideal thing to do would be to modify the application to be smarter with its files, but I suspect you don't have that option.

What's your system? What's your filesystem?
# 3  
Old 01-17-2013
Thanks Corona688.
Actually they do need all the files. In addition they need random sample files worth 100MB for their review (or whatever). I have to zip these 100MB files and put it on the server which they come pick up and review.
It is a SunOS unix system(SPARC-Enterprise-T5120).
vxfs filesystem
 
Login or Register to Ask a Question

Previous Thread | Next Thread

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Concatenate more than 200 files

Hi, I have a bunch of files (around 500) with the .log extension and each having a different name. Is there an easy way to concatenate them all into a single file? I've tried: cat *.log > complete.log and it seems that files are overlapping each other...thus the new file is not complete.... (2 Replies)
Discussion started by: amarn
2 Replies

2. Shell Programming and Scripting

Delete files having less than 200 lines

Hi All, I have some 30,000 files in one directory. The files look like this: computer networks router wire I want to remove those files which have less than 200 lines for example in the above file, the number of lines is 4. I am trying something like this: find /path/to/dir -type f... (5 Replies)
Discussion started by: shoaibjameel123
5 Replies

3. Shell Programming and Scripting

Listing 000 permission files

Hi all, is there a way to list all files with 000 permission and not get "Permission denied" message? I would like to have names of all those files in one text file. At the moment, i have this " find . -type d \( -name . -o -prune \) -perm 000 > text.txt " and it is not working. Can... (8 Replies)
Discussion started by: bb2
8 Replies

4. UNIX for Dummies Questions & Answers

Split 200.000 files into different subfolders

Dear UNIX-Community, can help me doing 2 things in Debian 5.0? 1.) Create 100 folders Format: ./0/0, ./0/1, ./0/2, ...,./0/9, ./1/0/, ..., ./9/9 2.) Move over 200.000 files into the subdirectories according to their last digits. 12398123.dat -> ./3/2/12398123.dat 48161.dat ->... (2 Replies)
Discussion started by: diadas
2 Replies

5. Shell Programming and Scripting

Compare 200,000 of rows in two text files

Friends, I have two very large plain text files with pipe delimited as below. Both files are not sorted. Both files have 200,000 of rows. FName|LName|Address|HPhNumber Is perl or shell script feasible for this task? Thanks, Prashant (1 Reply)
Discussion started by: ppat7046
1 Replies

6. Shell Programming and Scripting

Finding files in current directory when 100,000's files in current directory

Hi All I was wondering what is the most efficient way to find files in the current directory(that may contain 100,000's files), that meets a certain specified file type and of a certain age. I have experimented with the find command in unix but it also searches all sub directories. I have... (2 Replies)
Discussion started by: kewong007
2 Replies

7. Shell Programming and Scripting

Need help in creating 10,000 files in unix

hi all, I am an UNIX beginner. I need to create 10,000 .dat files in unix. and first file should have 10102009*1*14, second file should have 10102009*2*14, and the 10,000th file should contain 10102009*10000*14, (Comma is a must at end) The file name should be... (1 Reply)
Discussion started by: satishpv_2002
1 Replies

8. Shell Programming and Scripting

Fastest way to list a file in a folder containing 800,000 files using wildcard

Hi, I have a directory with possibly around 800,000 files in it. What is the fastest way to list file(s) in this directory with a wildcard. for example would ls -1 *.abcdefg.Z or find . -name "*.abcdefg.Z" be the fastest way to find all of the files that end with .abcdefg.Z... (6 Replies)
Discussion started by: jerardfjay
6 Replies

9. Shell Programming and Scripting

Help removing the prefix of more than 10,000 files

I have more than 10,000 files that all start with "raw_ddd_04_*". How can I remove the prefix on these files in a single command? (8 Replies)
Discussion started by: bbbngowc
8 Replies
Login or Register to Ask a Question