Login or Register to Ask a Question and Join Our Community


Shell Programming and Scripting

split file problem

Login to Discuss or Reply to this Discussion in Our Community

 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting split file problem
# 8  
Old 09-03-2009
If use say "-a 4" this gives you up to 26x26x26x26=456,976 suffix combinations (aaaa-zzzz) which at 5000 records per file comes to 2,284,880,000 maximum input records.

Not sure if I understand your question. If your "-a" value is high enough you don't need to know the number of records in advance.

If you are creating large numbers of files be careful that you have enough inodes in the filesystem (df -i).
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. UNIX for Beginners Questions & Answers

sed awk: split a large file to unique file names

Dear Users, Appreciate your help if you could help me with splitting a large file > 1 million lines with sed or awk. below is the text in the file input file.txt scaffold1 928 929 C/T + scaffold1 942 943 G/C + scaffold1 959 960 C/T +... (6 Replies)
Discussion started by: kapr0001
6 Replies

2. Shell Programming and Scripting

How to split file into multiple files using awk based on 1 field in the file?

Good day all I need some helps, say that I have data like below, each field separated by a tab DATE NAME ADDRESS 15/7/2012 LX a.b.c 15/7/2012 LX1 a.b.c 16/7/2012 AB a.b.c 16/7/2012 AB2 a.b.c 15/7/2012 LX2 a.b.c... (2 Replies)
Discussion started by: alexyyw
2 Replies

3. Shell Programming and Scripting

Split file based on file size in Korn script

I need to split a file if it is over 2GB in size (or any size), preferably split on the lines. I have figured out how to get the file size using awk, and I can split the file based on the number of lines (which I got with wc -l) but I can't figure out how to connect them together in the script. ... (6 Replies)
Discussion started by: ssemple2000
6 Replies

4. Shell Programming and Scripting

awk to split one field and print the last two fields within the split part.

Hello; I have a file consists of 4 columns separated by tab. The problem is the third fields. Some of the them are very long but can be split by the vertical bar "|". Also some of them do not contain the string "UniProt", but I could ignore it at this moment, and sort the file afterwards. Here is... (5 Replies)
Discussion started by: yifangt
5 Replies

5. Shell Programming and Scripting

Split a file into multiple files based on first two digits of file.

Hi , I do have a fixedwidth flatfile that has data for 10 different datasets each identified by the first two digits in the flatfile. 01 in the first two digit position refers to Set A 02 in the first two digit position refers to Set B and so on I want to genrate 10 different files from my... (6 Replies)
Discussion started by: okkadu
6 Replies

6. Shell Programming and Scripting

Split File by Pattern with File Names in Source File... Awk?

Hi all, I'm pretty new to Shell scripting and I need some help to split a source text file into multiple files. The source has a row with pattern where the file needs to be split, and the pattern row also contains the file name of the destination for that specific piece. Here is an example: ... (2 Replies)
Discussion started by: cul8er
2 Replies

7. Shell Programming and Scripting

How to split a data file into separate files with the file names depending upon a column's value?

Hi, I have a data file xyz.dat similar to the one given below, 2345|98|809||x|969|0 2345|98|809||y|0|537 2345|97|809||x|544|0 2345|97|809||y|0|651 9685|98|809||x|321|0 9685|98|809||y|0|357 9685|98|709||x|687|0 9685|98|709||y|0|234 2315|98|809||x|564|0 2315|98|809||y|0|537... (2 Replies)
Discussion started by: nithins007
2 Replies

8. Shell Programming and Scripting

problem with awk for file split

Hi all, i have a .ksh script which is, among other stuff, splitting a file and saveing the filenames into variables for further processing: # file split before ftp and put result filenames into variables if ]; then awk '{close(f);f=$1}{sub("^","");print > f".TXT"}' $_ftpfile set B*.TXT... (0 Replies)
Discussion started by: spidermike
0 Replies

9. UNIX for Dummies Questions & Answers

Split a file with no pattern -- Split, Csplit, Awk

I have gone through all the threads in the forum and tested out different things. I am trying to split a 3GB file into multiple files. Some files are even larger than this. For example: split -l 3000000 filename.txt This is very slow and it splits the file with 3 million records in each... (10 Replies)
Discussion started by: madhunk
10 Replies

10. UNIX for Dummies Questions & Answers

Problem in split command

I want to split a file containing millions of records. I am issuing the command split -l 20000 filename which will split the file in 20K records each. It works fine except in some files, data after one particular field is lost( the field with space). Say the record is ... (4 Replies)
Discussion started by: superprogrammer
4 Replies
Login or Register to Ask a Question

LEARN ABOUT CENTOS

split

split(n)						       Tcl Built-In Commands							  split(n)

__________________________________________________________________________________________________________________________________________________

NAME

       split - Split a string into a proper Tcl list

SYNOPSIS

       split string ?splitChars?
_________________________________________________________________

DESCRIPTION

       Returns a list created by splitting string at each character that is in the splitChars argument.  Each element of the result list will con-
       sist of the characters from string that lie between instances of the characters in splitChars.  Empty list elements will  be  generated	if
       string contains adjacent characters in splitChars, or if the first or last character of string is in splitChars.  If splitChars is an empty
       string then each character of string becomes a separate element of the result list.  SplitChars defaults to the standard white-space  char-
       acters.

EXAMPLES

       Divide up a USENET group name into its hierarchical components:
	      split "comp.lang.tcl.announce" .
		    -> comp lang tcl announce

       See how the split command splits on every character in splitChars, which can result in information loss if you are not careful:
	      split "alpha beta gamma" "temp"
		    -> al {ha b} {} {a ga} {} a

       Extract the list words from a string that is not a well-formed list:
	      split "Example with {unbalanced brace character"
		    -> Example with {unbalanced brace character

       Split a string into its constituent characters
	      split "Hello world" {}
		    -> H e l l o { } w o r l d

   PARSING RECORD-ORIENTED FILES
       Parse a Unix /etc/passwd file, which consists of one entry per line, with each line consisting of a colon-separated list of fields:
	      ## Read the file
	      set fid [open /etc/passwd]
	      set content [read $fid]
	      close $fid

	      ## Split into records on newlines
	      set records [split $content "
"]

	      ## Iterate over the records
	      foreach rec $records {

		 ## Split into fields on colons
		 set fields [split $rec ":"]

		 ## Assign fields to variables and print some out...
		 lassign $fields 
		       userName password uid grp longName homeDir shell
		 puts "$longName uses [file tail $shell] for a login shell"
	      }

SEE ALSO

       join(n), list(n), string(n)

KEYWORDS

       list, split, string

Tcl																	  split(n)