Sponsored Content
Top Forums Shell Programming and Scripting sort truncates line when they contain nulls Post 302186655 by massrobe on Thursday 17th of April 2008 05:43:19 PM
Old 04-17-2008
sort: warning: missing NEWLINE added

Hello Arthur. I have the same problem. Did you could fix it? Thanks


Quote:
Originally Posted by ArthurWaik
When I try to sort a file where some records contain nulls i.e. hex 00 the sort truncates the record when it reaches the null and writes message:

"sort: warning: missing NEWLINE added at end of input file myfile"

I'm assuming from this that the sort sees the null as a special character and acts accordingly. I could hack the file to replace the nulls with spaces but it would be great if I could tell the sort to accept the null as just another character in the record and not truncate.

Anybody got any ideas on options?
 

9 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

GREPing for Nulls

I just had a filesystem / file corruption issue on my HSP's server due to disk capacity limits and fileswapping. I discovered that certain files got corrupted when fileswapping was not successful and they ended up with a string of control characters, or what I believe to be nulls, in them. Does... (4 Replies)
Discussion started by: Dr. DOT
4 Replies

2. Shell Programming and Scripting

PS truncates the o/p

Hi , I have faced a strange situation in Solaris. the command ps -eo pid,args | grep 'SOMEPROCESS' truncates the output. outpt looks like 111 xxxxxxxxxxxxx SOMEPROCES 123 xxxxxxxxxxxxx SOMEPROCES 323 xxxxxxxxxxxxx SOMEPROCES The above doesn't return the complete command/args, infact if... (1 Reply)
Discussion started by: braindrain
1 Replies

3. UNIX for Advanced & Expert Users

who truncates the output? redirection? tty? Bug?

Hi, Output of running berkeley ps is truncated to 80 chars when using redirections. $ /usr/ucb/ps -e 12490|cat #truncated to 80 chars PID TT S TIME COMMAND 12490 pts/24 S 0:00 sleep 4000 aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa getting longer lines is done by changing the stty $... (7 Replies)
Discussion started by: fredy
7 Replies

4. Shell Programming and Scripting

Sort a file line by line alphabetically

infile: z y x c b a desired output: x y z a b c I don't want to sort the lines into this: a b c x y z nor this: c b a z y x The number of fields per line and number of lines is indeterminate. The field separator is always a space. Thanks for the use of your collective brains.... (11 Replies)
Discussion started by: H2OBoodle
11 Replies

5. Programming

Blanks vs: Nulls

I'm relatively new to Pro*C programming. In the following example: char name; EXEC SQL SELECT 'John Doe' INTO :name FROM DUAL; "John Doe" is in positions 0-7, blanks in 8-19, and a null in 20. I would really prefer the null to be in position 8 and I don't care what's after that. I wrote a... (1 Reply)
Discussion started by: ebock
1 Replies

6. Shell Programming and Scripting

include NULLs in line length check

Hello, I am checking the length of each line of a fixed length file and making sure all lines are 161 length. My problem is that some files contain null characters which gets stripped out of my echo. How do I have the NULLs included in my check? (and I cannot replace or sub the NULL values with... (10 Replies)
Discussion started by: ironmix
10 Replies

7. Shell Programming and Scripting

Sort a line and Insert sorted word(s) in a line

Hello, I am looking to automate a task - which is updating an existing access control instruction of a server and making sure that the attributes defined in the instruction is in sorted order. The instructions will be of a specific syntax. For example lets assume below listed is one of an... (6 Replies)
Discussion started by: sanjayroc
6 Replies

8. Shell Programming and Scripting

Replace nulls with a value in a file

Hi, I've a PIPE delimited file with about 5 fields. Sometimes the records in the 4th field is null, so I want to replace it based on values we get it on 2nd field in the same file. Following is an example. ABCD|X-TYPE 3.0|2010|X-TYPE|20000 CDEF|C-TYPE 2.5|2011|C-TYPE|10000 XYZ|LX... (4 Replies)
Discussion started by: rudoraj
4 Replies

9. Shell Programming and Scripting

/usr/bin/expect script truncates data

I have a script on a Linux machine that connects remotely, via telnet on a windows machine to launch several commands and colect their output. On the Linux machine the output of these commands is redirected in a file. The script: #!/usr/bin/expect log_user 0 spawn telnet 10.10.10.10... (6 Replies)
Discussion started by: black_fender
6 Replies
SORT(1) 						    BSD General Commands Manual 						   SORT(1)

NAME
sort -- sort or merge text files SYNOPSIS
sort [-bcdfHilmnrSsu] [-k field1[,field2]] [-o output] [-R char] [-T dir] [-t char] [file ...] DESCRIPTION
The sort utility sorts text files by lines. Comparisons are based on one or more sort keys extracted from each line of input, and are per- formed lexicographically. By default, if keys are not given, sort regards each input line as a single field. The following options are available: -c Check that the single input file is sorted. If the file is not sorted, sort produces the appropriate error messages and exits with code 1; otherwise, sort returns 0. sort -c produces no output. -H Ignored for compatibility with earlier versions of sort. -m Merge only; the input files are assumed to be pre-sorted. -o output The argument given is the name of an output file to be used instead of the standard output. This file can be the same as one of the input files. -S Don't use stable sort. Default is to use stable sort. -s Use stable sort, keeps records with equal keys in their original order. This is the default. Provided for compatibility with other sort implementations only. -T dir Use dir as the directory for temporary files. The default is the value specified in the environment variable TMPDIR or /tmp if TMPDIR is not defined. -u Unique: suppress all but one in each set of lines having equal keys. If used with the -c option, check that there are no lines with duplicate keys. The following options override the default ordering rules. When ordering options appear independent of key field specifications, the requested field ordering rules are applied globally to all sort keys. When attached to a specific key (see -k), the ordering options over- ride all global ordering options for that key. -d Only blank space and alphanumeric characters are used in making comparisons. -f Considers all lowercase characters that have uppercase equivalents to be the same for purposes of comparison. -i Ignore all non-printable characters. -l Sort by the string length of the field, not by the field itself. -n An initial numeric string, consisting of optional blank space, optional minus sign, and zero or more digits (including decimal point) is sorted by arithmetic value. (The -n option no longer implies the -b option.) -r Reverse the sense of comparisons. The treatment of field separators can be altered using these options: -b Ignores leading blank space when determining the start and end of a restricted sort key. A -b option specified before the first -k option applies globally to all -k options. Otherwise, the -b option can be attached independently to each field argument of the -k option (see below). Note that the -b option has no effect unless key fields are specified. -t char char is used as the field separator character. The initial char is not considered to be part of a field when determining key offsets (see below). Each occurrence of char is significant (for example, ``charchar'' delimits an empty field). If -t is not specified, the default field separator is a sequence of blank-space characters, and consecutive blank spaces do not delimit an empty field; further, the initial blank space is considered part of a field when determining key offsets. -R char char is used as the record separator character. This should be used with discretion; -R <alphanumeric> usually produces undesir- able results. The default record separator is newline. -k field1[,field2] Designates the starting position, field1, and optional ending position, field2, of a key field. The -k option replaces the obso- lescent options +pos1 and -pos2. The following operands are available: file The pathname of a file to be sorted, merged, or checked. If no file operands are specified, or if a file operand is -, the standard input is used. A field is defined as a minimal sequence of characters followed by a field separator or a newline character. By default, the first blank space of a sequence of blank spaces acts as the field separator. All blank spaces in a sequence of blank spaces are considered as part of the next field; for example, all blank spaces at the beginning of a line are considered to be part of the first field. Fields are specified by the -k field1[,field2] argument. A missing field2 argument defaults to the end of a line. The arguments field1 and field2 have the form m.n and can be followed by one or more of the letters b, d, f, i, l, n, and r, which correspond to the options discussed above. A field1 position specified by m.n (m, n > 0) is interpreted as the nth character in the mth field. A miss- ing .n in field1 means '.1', indicating the first character of the mth field; if the -b option is in effect, n is counted from the first non- blank character in the mth field; m.1b refers to the first non-blank character in the mth field. A field2 position specified by m.n is interpreted as the nth character (including separators) of the mth field. A missing .n indicates the last character of the mth field; m = 0 designates the end of a line. Thus the option -k v.x,w.y is synonymous with the obsolescent option +v-1.x-1-w-1.y; when y is omitted, -k v.x,w is synonymous with +v-1.x-1-w+1.0. The obsolescent +pos1 -pos2 option is still supported, except for -w.0b, which has no -k equivalent. ENVIRONMENT
If the following environment variable exists, it is used by sort. TMPDIR sort uses the contents of the TMPDIR environment variable as the path in which to store temporary files. FILES
/tmp/sort.* Default temporary files. outputNUMBER Temporary file which is used for output if output already exists. Once sorting is finished, this file replaces output (via link(2) and unlink(2)). EXIT STATUS
Sort exits with one of the following values: 0 Normal behavior. 1 On disorder (or non-uniqueness) with the -c option 2 An error occurred. SEE ALSO
comm(1), join(1), uniq(1), qsort(3), radixsort(3) HISTORY
A sort command appeared in Version 5 AT&T UNIX. This sort implementation appeared in 4.4BSD and is used since NetBSD 1.6. BUGS
Posix requires the locale's thousands separator be ignored in numbers. It may be faster to sort very large files in pieces and then explic- itly merge them. NOTES
This sort has no limits on input line length (other than imposed by available memory) or any restrictions on bytes allowed within lines. To protect data sort -o calls link(2) and unlink(2), and thus fails on protected directories. Input files should be text files. If file doesn't end with record separator (which is typically newline), the sort utility silently supplies one. The current sort uses lexicographic radix sorting, which requires that sort keys be kept in memory (as opposed to previous versions which used quick and merge sorts and did not.) Thus performance depends highly on efficient choice of sort keys, and the -b option and the field2 argument of the -k option should be used whenever possible. Similarly, sort -k1f is equivalent to sort -f and may take twice as long. BSD
December 18, 2010 BSD
All times are GMT -4. The time now is 01:34 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy