Splitting a file into chunks of 1TB Post: 302842641

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

remove chunks of text from file

All, So, I have an ldif file that contains about 6500 users worth of data. Some users have a block of text I'd like to remove, while some don't. Example (block of text in question is the block starting with "authAuthority: ;Kerberosv5"): User with text block: # username, users,...

2. Shell Programming and Scripting

Split file into chunks of low & high byte

Hi guys, i have a question about spliting a binary file into 2 chunks. First chunk with all high bytes and the second one with all low bytes. What unix tools can i use? And how can this be performed? I looked in manpages of split and dd but this does not help. Thanks

3. Solaris

Multiple Backups to USB 1TB Drives using dd

First of all, great web site! I have been using it for a while but just registered today. It's been a great resource for me. Now, on to my issue.;) I'm geographically separated from six (Sun v245s) development servers that I have been asked to backup and restore as development is done and...

4. AIX

Can't backup more than ~1TB.

Hello all, I create a backup of a file system with hostA # find . | backup -iqvf /backup/hostA.fsA.backup After a while I get the following error message: backup medium write error: File too large Check backup media and rerun the backup /backup is a NFS mount and is the backup device...

5. UNIX for Dummies Questions & Answers

Awk: Print out overlapping chunks of file - rows 0-20,10-30,20-40 etc.

First time poster, but the forum has saved my bacon more times than... Lots. Anyway, I have a text file, and wanted to use Awk (or any other sensible program) to print out overlapping sections, or arbitrary length. To describe by example, for file 1 2 3 4 5 etc... I want the out put...

6. Shell Programming and Scripting

awk for splitting file in constant chunks

Hi gurus, I wanted to split main file in 20 files with 2500 lines in each file. My main file conatins total 2500*20 lines. Following awk I made, but it is breaking with error. awk '{ for (i = 1; i <= 20; i++) { starts=2500*$i-1; ends=2500*$i; NR>=starts && NR<=ends {f=My$i".txt"; print >> f;...

7. Shell Programming and Scripting

Reverse sort on delimited chunks within a file

Hello, I have a large file in which data of names is sorted according to their homographs. The database has the following structure:Each set of homographs with their corresponding equivalents in Devanagari is separated out from the next set by a hard return. An example will make this...

8. Shell Programming and Scripting

Deleting duplicated chunks in a file using awk/sed

Hi all, I'd always appreciate all helps from this site. I would like to delete duplicated chunks of strings on the same row(?). One chunk is comprised of four lines such as: path name starting point ending point voltage number I would like to delete duplicated chunks on the same...

9. Shell Programming and Scripting

Modification of perl script to split a large file into chunks of 5000 chracters

I have a perl script which splits a large file into chunks.The script is given below use strict; use warnings; open (FH, "<monolingual.txt") or die "Could not open source file. $!"; my $i = 0; while (1) { my $chunk; print "process part $i\n"; open(OUT, ">part$i.log") or die "Could...

10. UNIX for Beginners Questions & Answers

Split large file into smaller files without disturbing the entry chunks

Dears, Need you help with the below file manipulation. I want to split the file into 8 smaller files but without cutting/disturbing the entries (meaning every small file should start with a entry and end with an empty line). It will be helpful if you can provide a one liner command for this...

LEARN ABOUT SUSE

sum

sum(n)							     Cyclic Redundancy Checks							    sum(n)

__________________________________________________________________________________________________________________________________________________

NAME

       sum - Calculate a sum(1) compatible checksum

SYNOPSIS

       package require Tcl  8.2

       package require sum  ?1.1.0?

       ::crc::sum ?-bsd | -sysv? ?-format fmt? ?-chunksize size? [ -filename file | -channel chan | string ]

_________________________________________________________________

DESCRIPTION

       This  package  provides	a Tcl-only implementation of the sum(1) command which calculates a 16 bit checksum value from the input data.  The
       BSD sum algorithm is used by default but the SysV algorithm is also available.

COMMANDS

       ::crc::sum ?-bsd | -sysv? ?-format fmt? ?-chunksize size? [ -filename file | -channel chan | string ]
	      The command takes string data or a file name or a channel and returns a checksum value calculated using the  sum(1)  algorithm.  The
	      result is formatted using the format(n) specifier provided or as an unsigned integer (%u) by default.

OPTIONS

       -sysv  The SysV algorithm is fairly naive. The byte values are summed and any overflow is discarded. The lowest 16 bits are returned as the
	      checksum.  Input with the same content but different ordering will give the same result.

       -bsd   This algorithm is similar to the SysV version but includes a bit rotation step which provides a dependency on the order of the  data
	      values.

       -filename name
	      Return a checksum for the file contents instead of for parameter data.

       -channel chan
	      Return  a  checksum  for	the  contents  of the specified channel. The channel must be open for reading and should be configured for
	      binary translation. The channel will no be closed on completion.

       -chunksize size
	      Set the block size used when reading data from either files or channels. This value defaults to 4096.

       -format string
	      Return the checksum using an alternative format template.

EXAMPLES

       % crc::sum "Hello, World!"
       37287

       % crc::sum -format 0x%X "Hello, World!"
       0x91A7

       % crc::sum -file sum.tcl
       13392

AUTHORS

       Pat Thoyts

BUGS, IDEAS, FEEDBACK
       This document, and the package it describes, will undoubtedly contain bugs and other problems.  Please report such in the category  crc	of
       the  Tcllib  SF	Trackers [http://sourceforge.net/tracker/?group_id=12883].  Please also report any ideas for enhancements you may have for
       either package and/or documentation.

SEE ALSO

       cksum(n), crc32(n), sum(1)

KEYWORDS

       checksum, cksum, crc, crc32, cyclic redundancy check, data integrity, security, sum

COPYRIGHT

       Copyright (c) 2002, Pat Thoyts <patthoyts@users.sourceforge.net>

crc								       1.1.0								    sum(n)