Visit Our UNIX and Linux User Community


How to generate a large file in Linux ?

 
Thread Tools Search this Thread
Top Forums UNIX for Beginners Questions & Answers How to generate a large file in Linux ?
# 1  
Old 09-06-2017
How to generate a large file in Linux ?

RHEL 7.3

In Linux, how can I create a large dummy file which is 15GB in size ?
# 2  
Old 09-06-2017
Code:
mkfile

?
# 3  
Old 09-06-2017
You don't state the OS version, so I don't know what tools you may already have to do this for you, and it depends what you want the file for.

There is a way in C that you open a new file, skipping to a huge offset then writing a single character before closing it. That might do the trick but it doesn't initially consume disk space and won't worry you for IO load. That may not be good for what you are trying to test. Additionally, if you restore it or copy it, the target file will be the full size, so that might mean that the filesystem does not restore properly if you suddenly fill it during a restore.

A good way can be to take a tar of a good chunk of disk (any files will do) then append it to itself several times, repeating until you get the desired size. Something like:-
Code:
cd /path/to/empty/space                                           # An empty directory to build the file
tar -cvf - /path/to/source/data | compress > my_base_file         # Get a chunk of something to start with.  Anything really, but probably best not devices, database files etc. that might be open

cat my_base_file my_base_file my_base_file > my_large_file        # Triple the file size
mv my_large_file  my_base_file                                    # Move it back to save space

Repeat the last two lines as needed to get a big enough file. If you need to trim it down again, you could perhaps:-
Code:
split -d -a 1 -b 15G my_large_file part. 2>/dev/null              # Break it into 15Gb chunks with a numeric suffix of length 1
mv part.0 my_large_file                                           # Save the one you actually want
rm part*                                                          # Clean up the rest of the junk

I've made it compress the file (you may prefer gzip) so that it is a true performance test, i.e. it can't be further compressed during transmission etc.


I hope that this helps,
Robin
This User Gave Thanks to rbatte1 For This Post:
# 4  
Old 09-06-2017
Most if not all Linux distros are lacking mkfile.
The usual workaround is with dd
Code:
dd count=15 bs=1G </dev/zero >dummy

This User Gave Thanks to MadeInGermany For This Post:
# 5  
Old 09-06-2017
Hi.

Some repetition, but also:
Code:
Generate specific-size file

        0) "dd", and other than Linux at:
           https://stackoverflow.com/questions/257844/quickly-create-a-large-file-on-a-linux-system

        1) mkfile (local), work-alike based on Solaris mkfile, q.v., dd wrapper
           Original from http://steve-parker.org/

        2) xfs_mkfile, part of xfsprogs, resides in /usr/sbin/
           Not restricted to XFS filesystems

        3) fallocate, preallocate or deallocate space to a file

        4) wsf (local), Write, create, sized file

        5) setosi (local), Set to size (length): make files same number of lines

        6) truncate, shrink or extend the size of a file to the specified size

Best wishes ... cheers, drl
These 3 Users Gave Thanks to drl For This Post:

Previous Thread | Next Thread
Test Your Knowledge in Computers #291
Difficulty: Easy
In August 2016, the United States government announced a new federal source-code policy. This policy mandates that at least 50% of custom source code developed by or for any agency of the federal government must be released as open-source software (OSS).
True or False?

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Needed script to FTP a File and generate a quality checksum file

hi all i want a script to FTP a file and should generate a quality checksum file means when I FTP a file from one server to another server it should generate a QC file which should contain timestamp,no.of records in that file Thanks in advance saikumar (3 Replies)
Discussion started by: hemanthsaikumar
3 Replies

2. UNIX for Dummies Questions & Answers

How to generate html reports through LINUX Scripting?

Hi All, I am trying to generate a weekly HTML report using LINUX Scripting. This will have record counts of some files. (like below) touch path/filename.html echo "Weekly Summary Report for Business Date : $P_BUS_DT">path/filename.html export A1=`cat path/filename1.txt |wc -l` echo "A1... (6 Replies)
Discussion started by: dsfreddie
6 Replies

3. UNIX for Dummies Questions & Answers

LINUX how to generate multiple line output file

Hi All, I am trying to generate a file through a LINUX scripting as shown below export field1=`cat dir/filename1.txt |wc -l` echo "field1 : $field1 >>dir/WeeklySummaryReport.txt export field2=`cat dir/filename2.txt |wc -l` echo "field2 : $field2 >>dir/WeeklySummaryReport.txt While... (5 Replies)
Discussion started by: dsfreddie
5 Replies

4. Shell Programming and Scripting

Linux - Script to generate the output delimited by Comma/Pipe

Hi All, I have a requirement where I need to go to a directory, list all the files that start with person* (for eg) & read the most recent file from the list of files. While browsing through the forum, i found that the command ls -t will list the files. I am trying to generate the output... (1 Reply)
Discussion started by: dsfreddie
1 Replies

5. Shell Programming and Scripting

Generate and copy files in UNICODE format from Linux to Windows

Hi,We have an interface which extracts data from database tables, spool the records into text files, and copy those files using SFTP to a windows server location. The target system loads these files into its database using some DTS packages in SQL Server. This interface of exporting text files... (13 Replies)
Discussion started by: urjagadeesh
13 Replies

6. Programming

Make cannot generate .ko linux kernel module

cannot generate .ko file on my linux, although it can generate module.symvers. But when I copy .c file and Makefile to another linux computer, there's no problem. The strange thing is: make is successfuly executed, and returned 0; make output: make -C /lib/modules/2.6.18-92.el5xen/build ... (4 Replies)
Discussion started by: vistastar
4 Replies

7. Shell Programming and Scripting

KSH - help needed for creating a script to generate xml file from text file

Dear Members, I have a table in Oracle DB and one of its column name is INFO which has data in text format which we need to fetch in a script and create an xml file of a new table from the input. The contents of a single cell of INFO column is like: Area:app - aam Clean Up Criteria:... (0 Replies)
Discussion started by: Yoodit
0 Replies

8. Red Hat

not large enough to display the application in red hat linux x win desktop

I use red hat linux es 5 I use startx to start the x-win desktop. But when I use vritual manager . The display application is too large so the bottom part for the application cannot show out. I cannot scroll down to get the display of bottm part . So, I do not know what button display at the... (0 Replies)
Discussion started by: chuikingman
0 Replies

9. Shell Programming and Scripting

need help in Parsing a CSV file and generate a new output file

Hi Scripting Gurus, I am trying to parse a csv file and generate a new output file. The input file will be a variable length in turns of rows and columns. output file will have 8 columns. we have three columns from the header for each set. just to give little bit more clarification each row... (15 Replies)
Discussion started by: vkr
15 Replies

10. UNIX for Dummies Questions & Answers

Script to generate text file from excel file

Hello, I have a excel file which has almost ten columns on the shared drive. I have to write a shell script to ftp that daily to unix server and then extract some columns from there and generate oracle standard text file. The columns should be in proper order and aligned properly, otherwise... (1 Reply)
Discussion started by: isingh786
1 Replies

Featured Tech Videos