Parsing large files in Solaris 11 Post: 302952515

Sponsored Content

Top Forums Shell Programming and Scripting Parsing large files in Solaris 11 Post 302952515 by Scrutinizer on Wednesday 19th of August 2015 01:03:43 PM

08-19-2015

Moderator

Quote:

Originally Posted by Don Cragun

I assume you meant bs=4 instead of bs=3, but when processing a 1.2Gb file, dd will run noticeably faster with its default block size (512 bytes) or a larger size like bs=1024000. The dd bs=n parameter specifies how many bytes dd will read at a time from its input file and how many bytes at a time it will write to its output file.

With conv=unblock, it is just the conversion buffer size (specified by cbs=n) that determines the output line length produced by the dd utility.

Although I presume the system cache and the filesystem block size will soften the blow in the sense that usually this should translate into io sizes that are equivalent to the filesystem block size or multiples thereof, depending on how smart the filesystem is...

Scrutinizer

View Public Profile for Scrutinizer

Find all posts by Scrutinizer

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Problem with parsing a large file

Hi All, Following is the sample file and following is the op desired that is the last entry of each unique first field is required. My solution is as follows However the original file has around a million entries and around a 100,000 uniques first fields, so this soln....

2. Shell Programming and Scripting

Parsing a large log

I need to parse a large log say 300-400 mb The commands like awk and cat etc are taking time. Please help how to process. I need to process the log for certain values of current date. But I am unbale to do so.

3. Shell Programming and Scripting

parsing large CDR XML file

Dear Freind in the file attached how parse the data to be like a normal table :D

4. Shell Programming and Scripting

Divide large data files into smaller files

Hello everyone! I have 2 types of files in the following format: 1) *.fa >1234 ...some text... >2345 ...some text... >3456 ...some text... . . . . 2) *.info >1234

5. Solaris

How to safely copy full filesystems with large files (10Gb files)

Hello everyone. Need some help copying a filesystem. The situation is this: I have an oracle DB mounted on /u01 and need to copy it to /u02. /u01 is 500 Gb and /u02 is 300 Gb. The size used on /u01 is 187 Gb. This is running on solaris 9 and both filesystems are UFS. I have tried to do it using:...

6. UNIX for Advanced & Expert Users

Need help with configuring large packet size on Solaris 7 / e6500

We're running Solaris 7 on FDDI n/w on an E6500 host and wish to use MTU (packet size) > 1500, more like 3072 bytes to begin with and possibly up to 4096 bytes. Linux has /etc/network/interfaces. Does ANYONE remember the equivalent in Unix? When I do ifconfig eth0 mtu 4000, I get the error...

7. Shell Programming and Scripting

Help needed for parsing large XML with awk.

My XML structure looks like: <?xml version="1.0" encoding="UTF-8"?> <SearchRepository> <SearchItems> <SearchItem> ... </SearchItem> <SearchItem> ... ...

8. UNIX for Dummies Questions & Answers

How to display large file in Solaris?

Hi i want to see file in solaris which are eating space. like we have a listfiles command in AIX which show all the files in decreading order of the size . example of listfile command in this command i am able to all the huge file in root directory. do we have any similar command in...

9. Shell Programming and Scripting

Parsing a subset of data from a large matrix

I do have a large matrix of the following format and it is tab delimited ch-ab1-20 ch-bb2-23 ch-ab1-34 ch-ab1-24 er-cc1-45 bv-cc1-78 ch-ab1-20 0 2 3 4 5 6 ch-bb2-23 3 0 5 ...

LEARN ABOUT CENTOS

isosize

ISOSIZE(8)						       System Administration							ISOSIZE(8)

NAME

       isosize - output the length of an iso9660 filesystem

SYNOPSIS

       isosize [options] iso9660_image_file

DESCRIPTION

       This  command  outputs  the  length  of an iso9660 filesystem that is contained in the specified file.  This file may be a normal file or a
       block device (e.g. /dev/hdd or /dev/sr0).  In the absence of any options (and errors), it will output the size of the iso9660 filesystem in
       bytes.  This can now be a large number (>> 4 GB).

OPTIONS

       -x, --sectors
	      Show the block count and block size in human-readable form.  The output uses the term "sectors" for "blocks".

       -d, --divisor number
	      Only has an effect when -x is not given.	The value shown (if no errors) is the iso9660 file size in bytes divided by number.  So if
	      number is the block size then the shown value will be the block count.

       The size of the file (or block device) holding an iso9660 filesystem can be marginally larger than the actual size of the iso9660  filesys-
       tem.  One reason for this is that cd writers are allowed to add "run out" sectors at the end of an iso9660 image.

AVAILABILITY

       The  isosize  command  is  part	of the util-linux package and is available from Linux Kernel Archive <ftp://ftp.kernel.org/pub/linux/utils
       /util-linux/>.

util-linux							     June 2011								ISOSIZE(8)