Sponsored Content
Top Forums Shell Programming and Scripting Fastest way calculating directory Post 302867895 by bakunin on Friday 25th of October 2013 07:48:26 AM
Old 10-25-2013
Any access to such a file system will tend to be slow because of the many necessary calls to stat(), not only the reading of the (basically) inode-structure by du. To minimize not only your problem at hand but all similar problems regarding this filesystem i suggest you move the vital FS information (that is: inodes and the like - all the metainformation) to some high-bandwidth storage, like an SSD.

I saw a similar problem (backup/restore of a huge GPFS with ~500TB of data) solved by introducing a 150GB SSD holding just the metadata. It reduced the necessary time from ~6 hours to ~90 minutes using the same hardware.

I hope this helps.

bakunin
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

fastest copy command

wich is the fastest command in HP-UX to copy an entire disk to dat tapes, or even disk to disk? thanks (0 Replies)
Discussion started by: vascobrito
0 Replies

2. Shell Programming and Scripting

Scripts for calculating size and remaining space of a directory automatically.

I would like to create a script for calculating size and remaining space of a directory automatically every 24 hours, then send an email to report to the admin. * POSIX and PERL are preferred. Can anyone help, please? (1 Reply)
Discussion started by: leonall
1 Replies

3. Shell Programming and Scripting

how to delete/remove directory in fastest way

hello i need help to remove directory . The directory is not empty ., it contains several sub directories and files inside that.. total number of files in one directory is 12,24,446 . rm -rf doesnt work . it is prompting for every file .. i want to delete without prompting and... (6 Replies)
Discussion started by: getdpg
6 Replies

4. UNIX for Advanced & Expert Users

Fastest way for searching the file

I want to search a file in fastest manner. Presently I am using 'find' command. But it is taking around 15min for searching. Is there any other method through which I can do it fast? (3 Replies)
Discussion started by: vaibhavbhat
3 Replies

5. AIX

Fastest way to count big amount of files in sub directory

Hi, what happened is we want to count all the files in a directory and inside this directory got many folders and so take long time to count it. Already run for about few minutes but still not done. The command we use to count is find . -type f | wc -l Just wondering if there is any other... (9 Replies)
Discussion started by: ngaisteve1
9 Replies

6. Solaris

The FASTEST copy method?

Hi Experts, I've been asked if there is a fast way to duplicate a file(10GB) and zip it at the same time. The zipped file would be FTP'd.....management is asking this. Maybe there is a better method all together? any ideas? CP will not cut it. Thanks in advance Harley (1 Reply)
Discussion started by: Harleyrci
1 Replies

7. Shell Programming and Scripting

calculating column summation in a directory of flat files

Hello Guru s I need your kind help to solve my below issue I have a directory of flat files and have to calculate sum of some columns from the flat file . Say for flat file 302 I need the column summation of 2 and 3 rd column For flat file 303 I need the column summation of 5 and... (2 Replies)
Discussion started by: Pratik4891
2 Replies

8. Shell Programming and Scripting

Calculating the epoch time from standard time using awk and calculating the duration

Hi All, I have the following time stamp data in 2 columns Date TimeStamp(also with milliseconds) 05/23/2012 08:30:11.250 05/23/2012 08:30:15.500 05/23/2012 08:31.15.500 . . etc From this data I need the following output. 0.00( row1-row1 in seconds) 04.25( row2-row1 in... (5 Replies)
Discussion started by: ks_reddy
5 Replies

9. Shell Programming and Scripting

How to Calculating space used in GB for any particular directory in UNIX?

How to Calculating space used in GB for any particular directory in unix Currently I am using : df -h which gives me space for each mout point ldndyn1:/vol/v01/dyn/sbcexp/dyn 1.1T 999G 29G 98% /sbcimp/dyn but I need for some internal particular directory... (3 Replies)
Discussion started by: RahulJoshi
3 Replies

10. Shell Programming and Scripting

Check fastest server and using it

hello we have upload some data in 15 servers in usa asia ... i consider to add new feature , script can detect download speed between localhost and destination and use fastest server, i have cut this part from a script which have this feature, download a xx MB file from all its source and... (0 Replies)
Discussion started by: nimafire
0 Replies
xfs_check(8)						      System Manager's Manual						      xfs_check(8)

NAME
xfs_check - check XFS filesystem consistency SYNOPSIS
xfs_check [ -i ino ] ... [ -b bno ] ... [ -f ] [ -s ] [ -v ] [ -l logdev ] device DESCRIPTION
xfs_check checks whether an XFS filesystem is consistent. It is normally run only when there is reason to believe that the filesystem has a consistency problem. The filesystem to be checked is specified by the device argument, which should be the disk or volume device for the filesystem. Filesystems stored in files can also be checked, using the -f flag. The filesystem should normally be unmounted or read-only during the execution of xfs_check. Otherwise, spurious problems are reported. OPTIONS
-f Specifies that the filesystem image to be processed is stored in a regular file at device (see the mkfs.xfs(8) -d file option). This might happen if an image copy of a filesystem has been made into an ordinary file. -l logdev Specifies the device where the filesystem's external log resides. Only for those filesystems which use an external log. See the mkfs.xfs(8) -l option, and refer to xfs(5) for a detailed description of the XFS log. -s Specifies that only serious errors should be reported. Serious errors are those that make it impossible to find major data struc- tures in the filesystem. This option can be used to cut down the amount of output when there is a serious problem, when the output might make it difficult to see what the real problem is. -v Specifies verbose output; it is impossibly long for a reasonably-sized filesystem. This option is intended for internal use only. -i ino Specifies verbose behavior for the specified inode ino. For instance, it can be used to locate all the blocks associated with a given inode. -b bno Specifies verbose behavior for the specific filesystem block at bno. For instance, it can be used to determine what a specific block is used for. The block number is a "file system block number". Conversion between disk addresses (i.e. addresses reported by xfs_bmap(8)) and file system blocks may be accomplished using xfs_db(8)'s convert command. Any output that is produced when xfs_check is not run in verbose mode indicates that the filesystem has an inconsistency. The filesystem can be repaired using either xfs_repair(8) to fix the filesystem in place, or by using xfsdump(8) and mkfs.xfs(8) to dump the filesystem, make a new filesystem, then use xfsrestore(8) to restore the data onto the new filesystem. Note that xfsdump may fail on a corrupt filesystem. However, if the filesystem is mountable, xfsdump can be used to try and save important data before repairing the filesystem with xfs_repair. If the filesystem is not mountable though, xfs_repair is the only viable option. DIAGNOSTICS
If the filesystem is completely corrupt, a core dump might be produced instead of the message device is not a valid filesystem If the filesystem is very large (has many files) then xfs_check might run out of memory. In this case the message out of memory is printed. The following is a description of the most likely problems and the associated messages. Most of the diagnostics produced are only meaning- ful with an understanding of the structure of the filesystem. agf_freeblks n, counted m in ag a The freeblocks count in the allocation group header for allocation group a doesn't match the number of blocks counted free. agf_longest n, counted m in ag a The longest free extent in the allocation group header for allocation group a doesn't match the longest free extent found in the allocation group. agi_count n, counted m in ag a The allocated inode count in the allocation group header for allocation group a doesn't match the number of inodes counted in the allocation group. agi_freecount n, counted m in ag a The free inode count in the allocation group header for allocation group a doesn't match the number of inodes counted free in the allocation group. block a/b expected inum 0 got i The block number is specified as a pair (allocation group number, block in the allocation group). The block is used multiple times (shared), between multiple inodes. This message usually follows a message of the next type. block a/b expected type unknown got y The block is used multiple times (shared). block a/b type unknown not expected The block is unaccounted for (not in the freelist and not in use). link count mismatch for inode nnn (name xxx), nlink m, counted n The inode has a bad link count (number of references in directories). rtblock b expected inum 0 got i The block is used multiple times (shared), between multiple inodes. This message usually follows a message of the next type. rtblock b expected type unknown got y The real-time block is used multiple times (shared). rtblock b type unknown not expected The real-time block is unaccounted for (not in the freelist and not in use). sb_fdblocks n, counted m The number of free data blocks recorded in the superblock doesn't match the number counted free in the filesystem. sb_frextents n, counted m The number of free real-time extents recorded in the superblock doesn't match the number counted free in the filesystem. sb_icount n, counted m The number of allocated inodes recorded in the superblock doesn't match the number allocated in the filesystem. sb_ifree n, counted m The number of free inodes recorded in the superblock doesn't match the number free in the filesystem. SEE ALSO
mkfs.xfs(8), xfsdump(8), xfsrestore(8), xfs_ncheck(8), xfs_repair(8), xfs(5). xfs_check(8)
All times are GMT -4. The time now is 11:07 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy