Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: File splitting script help
Top Forums UNIX for Dummies Questions & Answers File splitting script help Post 302943283 by Don Cragun on Thursday 7th of May 2015 04:01:25 AM
Old 05-07-2015
Using:
Code:
awk -v POS="$Position" -v NXT="$NextPostion" -v PRF="$TargetfilePrefix" '{print >  PRF "_" substr($0, POS, NXT-POS+1) ".txt"}'  $Filename

could have a problem on some versions of awk because the precedence between concatenation of strings and the output redirection operator in print statements is not specified by the standards. Some implementations of awk will treat this code as:
Code:
awk -v POS="$Position" -v NXT="$NextPostion" -v PRF="$TargetfilePrefix" '{(print >  PRF) "_" substr($0, POS, NXT-POS+1) ".txt"}'  $Filename

(possibly giving a syntax error, and certainly not producing the output files you want) and others will treat it as:
Code:
awk -v POS="$Position" -v NXT="$NextPostion" -v PRF="$TargetfilePrefix" '{print > (PRF "_" substr($0, POS, NXT-POS+1) ".txt")}'  $Filename

Since it is working on your system, we can assume that your version of awk is doing the latter.

This code doesn't close and reopen files, so, if it doesn't give an error (too many open files) it should run pretty quickly. It might run slightly faster if you move appending the "_" out of the loop and just do it once instead of two million times:
Code:
awk -v POS="$Position" -v NXT="$NextPostion" -v PRF="${TargetfilePrefix}_" '{print >  (PRF substr($0, POS, NXT-POS+1) ".txt")}'  $Filename

As was noted before, if this fails on your two million record file with a too many open files error, you'll have to build in code to close and open files. If you open and close files for each output line, that will run considerably slower. But, doing anything smarter than that would require you to evaluate the input file to see if lines directed to the same output file are closely grouped in the input, if there are common occurrences of adjacent lines that will be directed to the same output file, etc. that can be used to make smarter decisions about when to close a file and when (if ever) a file needs to be reopened.

Since you couldn't even guess at how many output files would be produced from your input file, I assume you have not tried to evaluate any of the above questions that might help produce more efficient code if you do have to close and reopen files.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Help with shell script - splitting

Hi, I need to split the file lines in below format as. Input file : Sample.txt <Rule expression="DeliverToCompID IS NULL" invert="true"> <Rule field="PossDupFlag" value="Y" > <Rule expression="OrdStatus = '2' AND OrigClOrdID IS NULL"> Output... (5 Replies)
Discussion started by: manosubsulo
5 Replies

2. Shell Programming and Scripting

File splitting and grouping using unix script

Hello All, I have a small problem with file group/splitting and I am trying to get the best way to perform this in unix. I am trying with awk but need some suggestion what would be the best and fastest way to-do it. Here is the problem. I have a fixed length file with filled with product... (4 Replies)
Discussion started by: nandhan11891
4 Replies

3. Shell Programming and Scripting

script for splitting file

Can anyone help me in giving a script for the below scenario I have file from the source increamenting in size... I require to write a script witch will move the data to the new file once the file reaches 50MB of size . This needs If the first file reaches 50MB then my script has to generate... (3 Replies)
Discussion started by: Sudhakishore.P
3 Replies

4. UNIX for Dummies Questions & Answers

Is there any way of splitting the script (Noob Here).

I m writing a script to check Server Hardening. The problem is whenever i add new point it grows and it become very tedious to edit the script file. Is there any way of making them separate and call them from one base script? Is it possible to define global variable that can be accessed via... (5 Replies)
Discussion started by: pinga123
5 Replies

5. Shell Programming and Scripting

Splitting a file in to multiple files and passing each individual file to a command

I have an input file with contents like: MainFile.dat: 12247689|7896|77698080 16768900|hh78|78959390 12247689|7896|77698080 16768900|hh78|78959390 12247689|7896|77698080 16768900|hh78|78959390 12247689|7896|77698080 16768900|hh78|78959390 12247689|7896|77698080 16768900|hh78|78959390 ... (4 Replies)
Discussion started by: rkrish
4 Replies

6. Shell Programming and Scripting

Splitting a file and creating new files using Perl script

Hi All, I am new to Scripting language. I want to split a file and create several subfiles using Perl script. Example : File format : Sourcename ID Date Nbr SU IMYFDJ 9/17/2012 5552159976555 SU BWZMIG 9/14/2012 1952257857887 AR PEHQDF 11/26/2012 ... (13 Replies)
Discussion started by: Deepak9870
13 Replies

7. Shell Programming and Scripting

Splitting XML file on basis of line number into multiple file

Hi All, I have more than half million lines of XML file , wanted to split in four files in a such a way that top 7 lines should be present in each file on top and bottom line of should be present in each file at bottom. from the 8th line actual record starts and each record contains 15 lines... (14 Replies)
Discussion started by: ajju
14 Replies

8. Shell Programming and Scripting

Execution of loop :Splitting a single file into multiple .dat file

hdr=$(cut -c1 $path$file|head -1)#extract header”H” trl=$(cut -c|path$file|tail -1)#extract trailer “T” SplitFile=$(cut -c 50-250 $path 1$newfile |sed'$/ *$//' head -1')# to trim white space and extract table name If; then # start loop if it is a header While read I #read file Do... (4 Replies)
Discussion started by: SwagatikaP1
4 Replies

9. Shell Programming and Scripting

Script for splitting file of records into multiple files

Hello I have a file of following format HDR 1234 abc qwerty abc def ghi jkl HDR 4567 xyz qwerty abc def ghi jkl HDR 890 mno qwerty abc def ghi jkl HDR 1234 abc qwerty abc def ghi jkl HDR 1234 abc qwerty abc def ghi jkl -Need to split this into multiple files based on tag... (8 Replies)
Discussion started by: wincrazy
8 Replies

10. Shell Programming and Scripting

Splitting a text file into smaller files with awk, how to create a different name for each new file

Hello, I have some large text files that look like, putrescine Mrv1583 01041713302D 6 5 0 0 0 0 999 V2000 2.0928 -0.2063 0.0000 N 0 0 0 0 0 0 0 0 0 0 0 0 5.6650 0.2063 0.0000 N 0 0 0 0 0 0 0 0 0 0 0 0 3.5217 ... (3 Replies)
Discussion started by: LMHmedchem
3 Replies

LEARN ABOUT DEBIAN

ppmtosixel

ppmtosixel(1)						      General Commands Manual						     ppmtosixel(1)

NAME

       ppmtosixel - convert a portable pixmap into DEC sixel format

SYNOPSIS

       ppmtosixel [-raw] [-margin] [ppmfile]

DESCRIPTION

       Reads  a  portable  pixmap  as input.  Produces sixel commands (SIX) as output.	The output is formatted for color printing, e.g. for a DEC
       LJ250 color inkjet printer.

       If RGB values from the PPM file do not have maxval=100, the RGB values are rescaled.  A printer control header and a color assignment table
       begin the SIX file.  Image data is written in a compressed format by default.  A printer control footer ends the image file.

OPTIONS

       -raw   If  specified,  each  pixel  will  be explicitly described in the image file.  If -raw is not specified, output will default to com-
	      pressed format in which identical adjacent pixels are replaced by "repeat pixel" commands.  A raw file is often an order	of  magni-
	      tude larger than a compressed file and prints much slower.

       -margin
	      If  -margin  is not specified, the image will be start at the left margin (of the window, paper, or whatever).  If -margin is speci-
	      fied, a 1.5 inch left margin will offset the image.

PRINTING

       Generally, sixel files must reach the printer unfiltered.  Use the lpr -x option or cat filename > /dev/tty0?.

BUGS

       Upon rescaling, truncation of the least significant bits of RGB values may result in poor color conversion.  If the original PPM maxval was
       greater	than  100,  rescaling  also reduces the image depth.  While the actual RGB values from the ppm file are more or less retained, the
       color palette of the LJ250 may not match the colors on your screen.  This seems to be a printer limitation.

SEE ALSO

       ppm(5)

AUTHOR

       Copyright (C) 1991 by Rick Vinci.

								   26 April 1991						     ppmtosixel(1)
All times are GMT -4. The time now is 09:48 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy