Sponsored Content
Top Forums UNIX for Dummies Questions & Answers Strange behaviour when output to terminal vs file (awk) Post 302744999 by Don Cragun on Sunday 16th of December 2012 08:17:27 AM
Old 12-16-2012
Note that with 20 fields on each of 3,000 input records being combined into 200 output lines, your average output lines with have 300 fields and some lines could have many more. You don't give us any indication of what system you're using, nor of the contents of most of the input fields. The awk utility and most editors are only defined to work on text files, and by definition, lines in a text file can't be longer than LINE_MAX bytes (including the terminating newline character). (Try:
Code:
getconf LINE_MAX

to determine the value of LINE_MAX on your system. The standards only require that implementations support lines up to 2,048 bytes per line.) Are you sure that none of your output lines exceed LINE_MAX?

If you ask awk to print a line that is longer than LINE_MAX bytes long, the results are unspecified. If you use ed, ex, grep, sed, vi (or any of LOTS of other standard utilities that are described as processing text files) to read or write or create internal lines longer than LINE_MAX bytes long, the results are unspecified. There are very few standard text processing utilities that are defined to work on lines with arbitrary lengths (cut, fold, and paste).
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

A Strange Behaviour!!!

Can some-one give me a view to this : I have a directory in an unix server, having permissions r-xr-xr-x .This directory is basically a source directory. Now there is another directory basically the destination directory which has all the permissions. Note:I log in as not the owner,but user... (5 Replies)
Discussion started by: navojit dutta
5 Replies

2. UNIX for Advanced & Expert Users

Strange sed behaviour

$ echo a.bc | sed -e "s/\|/\\|/g" |a|.|b|c| $ Is the behavior of the sed statement expected ? Or is this a bug in sed ? OS details Linux 2.6.9-55.0.0.0.2.ELsmp #1 SMP Wed May 2 14:59:56 PDT 2007 i686 i686 i386 GNU/Linux (8 Replies)
Discussion started by: vino
8 Replies

3. UNIX for Dummies Questions & Answers

Strange Program behaviour

Had a strange thing going on with my code. It's ok I figured it out for myself.... (2 Replies)
Discussion started by: mrpugster
2 Replies

4. Shell Programming and Scripting

Strange behaviour with perl i/o?

Hi All, I got a strange problem here. I have a perl script which is fetching data from a database table and writing a file with that data. If i run that script from linux command line, the file it creates is a normal ascii text file without any binary character in it.But... (9 Replies)
Discussion started by: DILEEP410
9 Replies

5. Shell Programming and Scripting

Strange RegExp Behaviour

Hello, I was trying to identify lines who has a word of the following pattern "xyyx" (where x, and ys are different characters). I was trying the following grep - egrep '(\S)()\2\1' This pattern do catches the wanted pattern, but it also catches "GGGG" or "CCCC" patterns. I was trying to... (5 Replies)
Discussion started by: itskov
5 Replies

6. Shell Programming and Scripting

Strange terminal behaviour after killing ssh

Hi, We have a problem where occasionally an ssh will hang for no apparent reason preventing the rest of the script continuing. To deal with this I am trying to write a wrapper script to kill a hung ssh command after a specified period. The scripts use a sleep command running in the... (2 Replies)
Discussion started by: RECrerar
2 Replies

7. UNIX for Dummies Questions & Answers

Strange perl print output behaviour

Hi, Today I have found the following case in perl: print "length:$lengths\tsum:". $count{$lengths}+$count_pair{$lengths}."\tindi:$count{$lengths}\t$count_pair{$lengths}\n";This give output as That means the first part of print is not printing. Only the values after the additions are printed.... (5 Replies)
Discussion started by: gvj
5 Replies

8. Shell Programming and Scripting

Strange behaviour of arrays in awk

Imagine 2 files f1 f2: file1_l1_c1 code_to_find file1_l1_c3 file1_l2_c1 file1_code2 file1_l2_c3 file1_l3_c1 file1_code3 file1_l3_c3 file2_l1_c1 file2_l1_c2 code_to_find file2_l2_c1 file2_l2_c2 file2_code5 file2_l3_c1 file2_l3_c2 file2_code3 Say we want to print lines from f2 having... (5 Replies)
Discussion started by: ripat
5 Replies

9. Shell Programming and Scripting

awk Strange behaviour in AIX

Can someone please explain the strange behaviour.. I was just trying a few things to learn awk.. in the below code when I start the braces in the same line, the output is as expected, when I start at next line, output is displayed twice. Please see the file, code I tried and output below. ... (2 Replies)
Discussion started by: Kulasekar
2 Replies

10. Shell Programming and Scripting

Print Terminal Output Exactly how it Appears in the Terminal to a New Text File

Hello All, I have a text file containing output from a command that contains lots of escape/control characters that when viewed using vi or view, looks like jibberish. But when viewed using the cat command the output is formatted properly. Is there any way to take the output from the cat... (7 Replies)
Discussion started by: mrm5102
7 Replies
CUT(1P) 						     POSIX Programmer's Manual							   CUT(1P)

PROLOG
This manual page is part of the POSIX Programmer's Manual. The Linux implementation of this interface may differ (consult the correspond- ing Linux manual page for details of Linux behavior), or the interface may not be implemented on Linux. NAME
cut -- cut out selected fields of each line of a file SYNOPSIS
cut -b list [-n] [file...] cut -c list [file...] cut -f list [-d delim] [-s] [file...] DESCRIPTION
The cut utility shall cut out bytes (-b option), characters (-c option), or character-delimited fields (-f option) from each line in one or more files, concatenate them, and write them to standard output. OPTIONS
The cut utility shall conform to the Base Definitions volume of POSIX.1-2008, Section 12.2, Utility Syntax Guidelines. The application shall ensure that the option-argument list (see options -b, -c, and -f below) is a <comma>-separated list or <blank>-sepa- rated list of positive numbers and ranges. Ranges can be in three forms. The first is two positive numbers separated by a <hyphen> (low-high), which represents all fields from the first number to the second number. The second is a positive number preceded by a <hyphen> (-high), which represents all fields from field number 1 to that number. The third is a positive number followed by a <hyphen> (low-), which represents that number to the last field, inclusive. The elements in list can be repeated, can overlap, and can be specified in any order, but the bytes, characters, or fields selected shall be written in the order of the input data. If an element appears in the selec- tion list more than once, it shall be written exactly once. The following options shall be supported: -b list Cut based on a list of bytes. Each selected byte shall be output unless the -n option is also specified. It shall not be an error to select bytes not present in the input line. -c list Cut based on a list of characters. Each selected character shall be output. It shall not be an error to select characters not present in the input line. -d delim Set the field delimiter to the character delim. The default is the <tab>. -f list Cut based on a list of fields, assumed to be separated in the file by a delimiter character (see -d). Each selected field shall be output. Output fields shall be separated by a single occurrence of the field delimiter character. Lines with no field delim- iters shall be passed through intact, unless -s is specified. It shall not be an error to select fields not present in the input line. -n Do not split characters. When specified with the -b option, each element in list of the form low-high (<hyphen>-separated num- bers) shall be modified as follows: * If the byte selected by low is not the first byte of a character, low shall be decremented to select the first byte of the character originally selected by low. If the byte selected by high is not the last byte of a character, high shall be decre- mented to select the last byte of the character prior to the character originally selected by high, or zero if there is no prior character. If the resulting range element has high equal to zero or low greater than high, the list element shall be dropped from list for that input line without causing an error. Each element in list of the form low- shall be treated as above with high set to the number of bytes in the current line, not including the terminating <newline>. Each element in list of the form -high shall be treated as above with low set to 1. Each element in list of the form num (a single number) shall be treated as above with low set to num and high set to num. -s Suppress lines with no delimiter characters, when used with the -f option. Unless specified, lines with no delimiters shall be passed through untouched. OPERANDS
The following operand shall be supported: file A pathname of an input file. If no file operands are specified, or if a file operand is '-', the standard input shall be used. STDIN
The standard input shall be used only if no file operands are specified, or if a file operand is '-'. See the INPUT FILES section. INPUT FILES
The input files shall be text files, except that line lengths shall be unlimited. ENVIRONMENT VARIABLES
The following environment variables shall affect the execution of cut: LANG Provide a default value for the internationalization variables that are unset or null. (See the Base Definitions volume of POSIX.1-2008, Section 8.2, Internationalization Variables for the precedence of internationalization variables used to determine the values of locale categories.) LC_ALL If set to a non-empty string value, override the values of all the other internationalization variables. LC_CTYPE Determine the locale for the interpretation of sequences of bytes of text data as characters (for example, single-byte as opposed to multi-byte characters in arguments and input files). LC_MESSAGES Determine the locale that should be used to affect the format and contents of diagnostic messages written to standard error. NLSPATH Determine the location of message catalogs for the processing of LC_MESSAGES. ASYNCHRONOUS EVENTS
Default. STDOUT
The cut utility output shall be a concatenation of the selected bytes, characters, or fields (one of the following): "%s ", <concatenation of bytes> "%s ", <concatenation of characters> "%s ", <concatenation of fields and field delimiters> STDERR
The standard error shall be used only for diagnostic messages. OUTPUT FILES
None. EXTENDED DESCRIPTION
None. EXIT STATUS
The following exit values shall be returned: 0 All input files were output successfully. >0 An error occurred. CONSEQUENCES OF ERRORS
Default. The following sections are informative. APPLICATION USAGE
The cut and fold utilities can be used to create text files out of files with arbitrary line lengths. The cut utility should be used when the number of lines (or records) needs to remain constant. The fold utility should be used when the contents of long lines need to be kept contiguous. Earlier versions of the cut utility worked in an environment where bytes and characters were considered equivalent (modulo <backspace> and <tab> processing in some implementations). In the extended world of multi-byte characters, the new -b option has been added. The -n option (used with -b) allows it to be used to act on bytes rounded to character boundaries. The algorithm specified for -n guarantees that: cut -b 1-500 -n file > file1 cut -b 501- -n file > file2 ends up with all the characters in file appearing exactly once in file1 or file2. (There is, however, a <newline> in both file1 and file2 for each <newline> in file.) EXAMPLES
Examples of the option qualifier list: 1,4,7 Select the first, fourth, and seventh bytes, characters, or fields and field delimiters. 1-3,8 Equivalent to 1,2,3,8. -5,10 Equivalent to 1,2,3,4,5,10. 3- Equivalent to third to last, inclusive. The low-high forms are not always equivalent when used with -b and -n and multi-byte characters; see the description of -n. The following command: cut -d : -f 1,6 /etc/passwd reads the System V password file (user database) and produces lines of the form: <user ID>:<home directory> Most utilities in this volume of POSIX.1-2008 work on text files. The cut utility can be used to turn files with arbitrary line lengths into a set of text files containing the same data. The paste utility can be used to create (or recreate) files with arbitrary line lengths. For example, if file contains long lines: cut -b 1-500 -n file > file1 cut -b 501- -n file > file2 creates file1 (a text file) with lines no longer than 500 bytes (plus the <newline>) and file2 that contains the remainder of the data from file. (Note that file2 is not a text file if there are lines in file that are longer than 500 + {LINE_MAX} bytes.) The original file can be recreated from file1 and file2 using the command: paste -d "" file1 file2 > file RATIONALE
Some historical implementations do not count <backspace> characters in determining character counts with the -c option. This may be useful for using cut for processing nroff output. It was deliberately decided not to have the -c option treat either <backspace> or <tab> charac- ters in any special fashion. The fold utility does treat these characters specially. Unlike other utilities, some historical implementations of cut exit after not finding an input file, rather than continuing to process the remaining file operands. This behavior is prohibited by this volume of POSIX.1-2008, where only the exit status is affected by this prob- lem. The behavior of cut when provided with either mutually-exclusive options or options that do not work logically together has been deliber- ately left unspecified in favor of global wording in Section 1.4, Utility Description Defaults. The OPTIONS section was changed in response to IEEE PASC Interpretation 1003.2 #149. The change represents historical practice on all known systems. The original standard was ambiguous on the nature of the output. The list option-arguments are historically used to select the portions of the line to be written, but do not affect the order of the data. For example: echo abcdefghi | cut -c6,2,4-7,1 yields "abdefg". A proposal to enhance cut with the following option: -o Preserve the selected field order. When this option is specified, each byte, character, or field (or ranges of such) shall be written in the order specified by the list option-argument, even if this requires multiple outputs of the same bytes, characters, or fields. was rejected because this type of enhancement is outside the scope of the IEEE P1003.2b draft standard. FUTURE DIRECTIONS
None. SEE ALSO
Section 2.5, Parameters and Variables, fold, grep, paste The Base Definitions volume of POSIX.1-2008, Chapter 8, Environment Variables, Section 12.2, Utility Syntax Guidelines COPYRIGHT
Portions of this text are reprinted and reproduced in electronic form from IEEE Std 1003.1, 2013 Edition, Standard for Information Technol- ogy -- Portable Operating System Interface (POSIX), The Open Group Base Specifications Issue 7, Copyright (C) 2013 by the Institute of Electrical and Electronics Engineers, Inc and The Open Group. (This is POSIX.1-2008 with the 2013 Technical Corrigendum 1 applied.) In the event of any discrepancy between this version and the original IEEE and The Open Group Standard, the original IEEE and The Open Group Stan- dard is the referee document. The original Standard can be obtained online at http://www.unix.org/online.html . Any typographical or formatting errors that appear in this page are most likely to have been introduced during the conversion of the source files to man page format. To report such errors, see https://www.kernel.org/doc/man-pages/reporting_bugs.html . IEEE
/The Open Group 2013 CUT(1P)
All times are GMT -4. The time now is 05:01 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy