Sponsored Content
Full Discussion: split file problem
Top Forums Shell Programming and Scripting split file problem Post 302350022 by hanu_oracle on Wednesday 2nd of September 2009 02:21:30 PM
Old 09-02-2009
Quote:
Originally Posted by methyl
"man split" it says "up to a maximum of 676 files" (26x26=676), but in some unixes you can have powers of 26 more.

If you don't have the "-a suffix_length" switch to "split" (which would fix the problem) and assuming you have unlimited disc space, I guess you could do two passes.

Split to 500 files of 20,000 lines.
Split each of the 500 files into 4 parts of 5,000 lines.
I think splitting 2 times is the double work & there is no manual activity also. script will run in the cronjob.
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Problem in split command

I want to split a file containing millions of records. I am issuing the command split -l 20000 filename which will split the file in 20K records each. It works fine except in some files, data after one particular field is lost( the field with space). Say the record is ... (4 Replies)
Discussion started by: superprogrammer
4 Replies

2. UNIX for Dummies Questions & Answers

Split a file with no pattern -- Split, Csplit, Awk

I have gone through all the threads in the forum and tested out different things. I am trying to split a 3GB file into multiple files. Some files are even larger than this. For example: split -l 3000000 filename.txt This is very slow and it splits the file with 3 million records in each... (10 Replies)
Discussion started by: madhunk
10 Replies

3. Shell Programming and Scripting

problem with awk for file split

Hi all, i have a .ksh script which is, among other stuff, splitting a file and saveing the filenames into variables for further processing: # file split before ftp and put result filenames into variables if ]; then awk '{close(f);f=$1}{sub("^","");print > f".TXT"}' $_ftpfile set B*.TXT... (0 Replies)
Discussion started by: spidermike
0 Replies

4. Shell Programming and Scripting

How to split a data file into separate files with the file names depending upon a column's value?

Hi, I have a data file xyz.dat similar to the one given below, 2345|98|809||x|969|0 2345|98|809||y|0|537 2345|97|809||x|544|0 2345|97|809||y|0|651 9685|98|809||x|321|0 9685|98|809||y|0|357 9685|98|709||x|687|0 9685|98|709||y|0|234 2315|98|809||x|564|0 2315|98|809||y|0|537... (2 Replies)
Discussion started by: nithins007
2 Replies

5. Shell Programming and Scripting

Split File by Pattern with File Names in Source File... Awk?

Hi all, I'm pretty new to Shell scripting and I need some help to split a source text file into multiple files. The source has a row with pattern where the file needs to be split, and the pattern row also contains the file name of the destination for that specific piece. Here is an example: ... (2 Replies)
Discussion started by: cul8er
2 Replies

6. Shell Programming and Scripting

Split a file into multiple files based on first two digits of file.

Hi , I do have a fixedwidth flatfile that has data for 10 different datasets each identified by the first two digits in the flatfile. 01 in the first two digit position refers to Set A 02 in the first two digit position refers to Set B and so on I want to genrate 10 different files from my... (6 Replies)
Discussion started by: okkadu
6 Replies

7. Shell Programming and Scripting

awk to split one field and print the last two fields within the split part.

Hello; I have a file consists of 4 columns separated by tab. The problem is the third fields. Some of the them are very long but can be split by the vertical bar "|". Also some of them do not contain the string "UniProt", but I could ignore it at this moment, and sort the file afterwards. Here is... (5 Replies)
Discussion started by: yifangt
5 Replies

8. Shell Programming and Scripting

Split file based on file size in Korn script

I need to split a file if it is over 2GB in size (or any size), preferably split on the lines. I have figured out how to get the file size using awk, and I can split the file based on the number of lines (which I got with wc -l) but I can't figure out how to connect them together in the script. ... (6 Replies)
Discussion started by: ssemple2000
6 Replies

9. Shell Programming and Scripting

How to split file into multiple files using awk based on 1 field in the file?

Good day all I need some helps, say that I have data like below, each field separated by a tab DATE NAME ADDRESS 15/7/2012 LX a.b.c 15/7/2012 LX1 a.b.c 16/7/2012 AB a.b.c 16/7/2012 AB2 a.b.c 15/7/2012 LX2 a.b.c... (2 Replies)
Discussion started by: alexyyw
2 Replies

10. UNIX for Beginners Questions & Answers

sed awk: split a large file to unique file names

Dear Users, Appreciate your help if you could help me with splitting a large file > 1 million lines with sed or awk. below is the text in the file input file.txt scaffold1 928 929 C/T + scaffold1 942 943 G/C + scaffold1 959 960 C/T +... (6 Replies)
Discussion started by: kapr0001
6 Replies
SPLIT(1P)						     POSIX Programmer's Manual							 SPLIT(1P)

PROLOG
This manual page is part of the POSIX Programmer's Manual. The Linux implementation of this interface may differ (consult the correspond- ing Linux manual page for details of Linux behavior), or the interface may not be implemented on Linux. NAME
split - split files into pieces SYNOPSIS
split [-l line_count][-a suffix_length][file[name]] split -b n[k|m][-a suffix_length][file[name]] DESCRIPTION
The split utility shall read an input file and write one or more output files. The default size of each output file shall be 1000 lines. The size of the output files can be modified by specification of the -b or -l options. Each output file shall be created with a unique suf- fix. The suffix shall consist of exactly suffix_length lowercase letters from the POSIX locale. The letters of the suffix shall be used as if they were a base-26 digit system, with the first suffix to be created consisting of all 'a' characters, the second with a 'b' replacing the last 'a', and so on, until a name of all 'z' characters is created. By default, the names of the output files shall be 'x', followed by a two-character suffix from the character set as described above, starting with "aa", "ab", "ac", and so on, and continuing until the suf- fix "zz", for a maximum of 676 files. If the number of files required exceeds the maximum allowed by the suffix length provided, such that the last allowable file would be larger than the requested size, the split utility shall fail after creating the last file with a valid suffix; split shall not delete the files it created with valid suffixes. If the file limit is not exceeded, the last file created shall contain the remainder of the input file, and may be smaller than the requested size. OPTIONS
The split utility shall conform to the Base Definitions volume of IEEE Std 1003.1-2001, Section 12.2, Utility Syntax Guidelines. The following options shall be supported: -a suffix_length Use suffix_length letters to form the suffix portion of the filenames of the split file. If -a is not specified, the default suffix length shall be two. If the sum of the name operand and the suffix_length option-argument would create a filename exceeding {NAME_MAX} bytes, an error shall result; split shall exit with a diagnostic message and no files shall be created. -b n Split a file into pieces n bytes in size. -b nk Split a file into pieces n*1024 bytes in size. -b nm Split a file into pieces n*1048576 bytes in size. -l line_count Specify the number of lines in each resulting file piece. The line_count argument is an unsigned decimal integer. The default is 1000. If the input does not end with a <newline>, the partial line shall be included in the last output file. OPERANDS
The following operands shall be supported: file The pathname of the ordinary file to be split. If no input file is given or file is '-', the standard input shall be used. name The prefix to be used for each of the files resulting from the split operation. If no name argument is given, 'x' shall be used as the prefix of the output files. The combined length of the basename of prefix and suffix_length cannot exceed {NAME_MAX} bytes. See the OPTIONS section. STDIN
See the INPUT FILES section. INPUT FILES
Any file can be used as input. ENVIRONMENT VARIABLES
The following environment variables shall affect the execution of split: LANG Provide a default value for the internationalization variables that are unset or null. (See the Base Definitions volume of IEEE Std 1003.1-2001, Section 8.2, Internationalization Variables for the precedence of internationalization variables used to determine the values of locale categories.) LC_ALL If set to a non-empty string value, override the values of all the other internationalization variables. LC_CTYPE Determine the locale for the interpretation of sequences of bytes of text data as characters (for example, single-byte as opposed to multi-byte characters in arguments and input files). LC_MESSAGES Determine the locale that should be used to affect the format and contents of diagnostic messages written to standard error. NLSPATH Determine the location of message catalogs for the processing of LC_MESSAGES . ASYNCHRONOUS EVENTS
Default. STDOUT
Not used. STDERR
The standard error shall be used only for diagnostic messages. OUTPUT FILES
The output files contain portions of the original input file; otherwise, unchanged. EXTENDED DESCRIPTION
None. EXIT STATUS
The following exit values shall be returned: 0 Successful completion. >0 An error occurred. CONSEQUENCES OF ERRORS
Default. The following sections are informative. APPLICATION USAGE
None. EXAMPLES
In the following examples foo is a text file that contains 5000 lines. 1. Create five files, xaa, xab, xac, xad, and xae: split foo 2. Create five files, but the suffixed portion of the created files consists of three letters, xaaa, xaab, xaac, xaad, and xaae: split -a 3 foo 3. Create three files with four-letter suffixes and a supplied prefix, bar_aaaa, bar_aaab, and bar_aaac: split -a 4 -l 2000 foo bar_ 4. Create as many files as are necessary to contain at most 20*1024 bytes, each with the default prefix of x and a five-letter suffix: split -a 5 -b 20k foo RATIONALE
The -b option was added to provide a mechanism for splitting files other than by lines. While most uses of the -b option are for transmit- ting files over networks, some believed it would have additional uses. The -a option was added to overcome the limitation of being able to create only 676 files. Consideration was given to deleting this utility, using the rationale that the functionality provided by this utility is available via the csplit utility (see csplit ). Upon reconsideration of the purpose of the User Portability Extension, it was decided to retain both this utility and the csplit utility because users use both utilities and have historical expectations of their behavior. Furthermore, the split- ting on byte boundaries in split cannot be duplicated with the historical csplit. The text " split shall not delete the files it created with valid suffixes" would normally be assumed, but since the related utility, csplit, does delete files under some circumstances, the historical behavior of split is made explicit to avoid misinterpretation. FUTURE DIRECTIONS
None. SEE ALSO
csplit COPYRIGHT
Portions of this text are reprinted and reproduced in electronic form from IEEE Std 1003.1, 2003 Edition, Standard for Information Technol- ogy -- Portable Operating System Interface (POSIX), The Open Group Base Specifications Issue 6, Copyright (C) 2001-2003 by the Institute of Electrical and Electronics Engineers, Inc and The Open Group. In the event of any discrepancy between this version and the original IEEE and The Open Group Standard, the original IEEE and The Open Group Standard is the referee document. The original Standard can be obtained online at http://www.opengroup.org/unix/online.html . IEEE
/The Open Group 2003 SPLIT(1P)
All times are GMT -4. The time now is 03:22 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy