LTO5 Catridge 1.5 TB Natvie capactiy unable to hold 1.44TB data in one volume

05-28-2013

Registered User

53, 2

Join Date: Apr 2009

Last Activity: 26 October 2015, 2:21 AM EDT

Posts: 53

Thanks Given: 5

Thanked 2 Times in 1 Post

OS : AIX
Version : 6100-08-02-1316
Filesystem Size which will be backed up: 1400 GB
LTO5 Cartridge Size : 1.5TB Native 3 TB Compressed
LTO5 Tape drive Attributes:
Software Compression: Using compress command
Command used to backup files: find * -print|backup -ivf '/dev/rmt0' '-U'

Code:

alt_pathing       no                    Enable Alternate Pathing Support                True
autoload          no                    Use Autoloading Feature at End-of-Tape          True
block_size        0    Block Size (0=Variable Length)                  True
compress          yes                   Use Hardware Compression on Tape                True
debug_trace       no                    Debug Trace Logging Enabled                     True
dev_status                              N/A                                             False
devtype           ULT3580-              Device Type                                     False
hh_refresh        yes                   Half height refresh Drive                       False
location                                Location                                        True
logging           no                    Activate volume information logging             True
lun_id            0x0                   Logical Unit Number                             True
max_log_size      500    Maximum size of log file (in # of entries)      True
new_name                                New Logical Name                                True
node_name         0x2001000e111484e1    World Wide Node Name                            False
primary_device    rmt0                  Primary Logical Device                          False
reserve_key                             Persistent Reservation Key                      True
reserve_type      reserve_6             Reservation Type                                True
retain_reserve    no                    Retain Reservation                              True
rew_immediate     no                    Use Immediate Bit in Rewind Commands            True
scsi_id           0x40fe1               SCSI Target ID                                  True
space_mode        SCSI                  Backward Space/Forward Space Record Mode        True
sys_encryption    no                    Use System Encryption FCP Proxy Manager         True
trace_logging     yes                   Trace Logging Enabled                           True
trailer_labels    no                    Trailer Label Processing                        True
wrt_encryption    custom                System Encryption for Write Commands at BOP     True
ww_name    0x2002000e111484e1    World Wide Port Name                            False

Method:
We have an empty filesystem having size 1400GB. We ftp all the dump files in this filesystem which were already compressed on other AIX lpars using compress command. When the 1400GB filesytem gets 100% full we start taking the backup on LTO5 cartridge using find * -print|backup -ivf '/dev/rmt0' '-U'

This is the whole procedure.

Moderator's Comments:

edit by bakunin: please use CODE-tags for output too. Thank you.

Last edited by bakunin; 05-28-2013 at 08:13 AM..

m_raheelahmed

View Public Profile for m_raheelahmed

Find all posts by m_raheelahmed

05-28-2013

Registered User

6,384, 2,214

Join Date: May 2005

Last Activity: 28 October 2019, 4:59 PM EDT

Location: In the leftmost byte of /dev/kmem

Posts: 6,384

Thanks Given: 143

Thanked 2,214 Times in 1,548 Posts

Quote:

Originally Posted by m_raheelahmed

Command used to backup files: find * -print|backup -ivf '/dev/rmt0' '-U'

This sheds a little more light onto the problem. Unfortunately it rules out my suggestion, because backup - unlike tar - will not expand sparse files. I am not sure how compress handles these.

As you have hardware compression enabled i suggest not using compress on the original files, because they can't compressed two times anyway. Let the hardware in the tape drive deal with compression and save CPU time.

I have to admit this will not solve the enigma, but it should improve matters nevertheless.

I hope this helps.

bakunin

bakunin

View Public Profile for bakunin

Find all posts by bakunin

05-28-2013

Registered User

53, 2

Join Date: Apr 2009

Last Activity: 26 October 2015, 2:21 AM EDT

Posts: 53

Thanks Given: 5

Thanked 2 Times in 1 Post

But let me inform you our uncompressed total data size is 1350 GB & when we compress the the data files it will become around 200 GB. So with the the help of compress we are able to hold 5-6 days data in a 1400 GB filesystem which will be backed up on cartridge once in a week. It is also save our storage space for not holding uncompressed data which is huge in size.

m_raheelahmed

View Public Profile for m_raheelahmed

Find all posts by m_raheelahmed

05-28-2013

Registered User

456, 108

Join Date: Nov 2012

Last Activity: 30 July 2019, 10:40 AM EDT

Location: on the road for work; home is private time

Posts: 456

Thanks Given: 10

Thanked 108 Times in 100 Posts

I see you are using the -v argument

Quote:

-v
Causes the backup command to display additional information
about the backup. When using the -v flag, the size of the file
as it exists on the archive is displayed in bytes. Additionally,
a total of these file sizes is displayed when all files have
been processed. Directories are listed with a size of 0.
Symbolic links are listed with the size of the symbolic link.
Hard links are listed with the size of the file, which is how
hard links are archived. Block and character devices, if they
were backed up, are listed with a size of 0.

Further, as the files are already compressed I would disable hardware compression - since it is unlikely it is giving you much assistance.

Together with the -v option you should be able to determine who much space backup thinks it has written to the device.

Last edited by MichaelFelt; 05-28-2013 at 11:20 AM.. Reason: oops: you are already using -v - went from suggestion to question!

MichaelFelt

View Public Profile for MichaelFelt

Find all posts by MichaelFelt

05-28-2013

Registered User

6,384, 2,214

Join Date: May 2005

Last Activity: 28 October 2019, 4:59 PM EDT

Location: In the leftmost byte of /dev/kmem

Posts: 6,384

Thanks Given: 143

Thanked 2,214 Times in 1,548 Posts

Quote:

Originally Posted by m_raheelahmed

Some things to clarify: you can compress a file only once. If you use the utility "compress" (or "gzip", "bzip2" or any other packing program), you have compressed it once, using some CPU time to do so. If you want to restore it you will need CPU time too to uncompress it.

If you use the hardware compression, on the other hand, you will still compress it (and about the same ratio as with "compress"), but use no CPU time to do so. The same, when you restore it.

The size data for the LTO5 tape are pure estimations. OK, 1500GB of uncompressed size is a fact, but "3000GB compressed" simply means: we estimate that our average packing rate will be 1:2, regardless of what is thrown in the drives way, so we say it has 1500 x 2 = 3000GB of compressed capacity. In reality the compression ratio one really achieves varies widely, depending on what a file contains. Picture data like bitmaps usually can easily be compressed by a factor of 1:10, the same with DB-files, while executable files give considerably lower ratios, typically ~1:2 with a LZ78-based algorithm.

That means: if you use solely hardware compression will put you in about the same situation as now or as using software compression (the "compress" utility) alone. Everything else is just a waste of resources, because in your current situation the software compression does all the work and the hardware compression achieves most likely absolutely nothing.

I hope this helps.

bakunin

bakunin

View Public Profile for bakunin

Find all posts by bakunin

AIX

LTO5 Catridge 1.5 TB Natvie capactiy unable to hold 1.44TB data in one volume

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Output large volume of data to CSV file

Discussion started by: dellanicholson

2. AIX

/usr/sbin/extendvg: Unable to extend volume group

Discussion started by: williamen

3. AIX

Unable to extend volume group(VG)

Discussion started by: murali969

4. AIX

Slow listing of LTo5 tape contents

Discussion started by: m_raheelahmed

5. Solaris

[SOLVED] How to initialize LTO5 Tape Drive?

Discussion started by: Mack1982

6. UNIX for Dummies Questions & Answers

Confusion Regarding Physical Volume,Volume Group,Logical Volume,Physical partition

Discussion started by: kashifsd17

7. UNIX for Dummies Questions & Answers

Unable to reduce the size of volume group?

Discussion started by: pinga123

8. UNIX for Dummies Questions & Answers

Unable to deactivate a Volume Group

Discussion started by: atanubanerji

9. HP-UX

Unable To Extend the Size of a Logical Volume File System

Discussion started by: Rob Sandifer

10. Shell Programming and Scripting

Playing with Volume of data

Discussion started by: darshanw