Sponsored Content
Top Forums UNIX for Dummies Questions & Answers duplicated lines not recognized by sort and uniq Post 302250104 by roussine on Wednesday 22nd of October 2008 05:38:55 PM
Old 10-22-2008
Quote:
Originally Posted by jim mcnamara
One has newlines 0xa the other has newlines 0xa and carriage returns 0xd - it is a DOS text file.

To fix it use dos2unix (or dos2ux on some machines)
Code:
dos2unix dostextfile.txt > unixfile

thank you, clear. one thing remains a mistery - both files were created under unix with unix commands (though different in the two cases) - how the dos endings could have ever got there?..
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

sort/uniq

I have a file: Fred Fred Fred Jim Fred Jim Jim If sort is executed on the listed file, shouldn't the output be?: Fred Fred Fred Fred Jim Jim Jim (3 Replies)
Discussion started by: jimmyflip
3 Replies

2. Shell Programming and Scripting

Sort, Uniq, Duplicates

Input File is : ------------- 25060008,0040,03, 25136437,0030,03, 25069457,0040,02, 80303438,0014,03,1st 80321837,0009,03,1st 80321977,0009,03,1st 80341345,0007,03,1st 84176527,0047,03,1st 84176527,0047,03, 20000735,0018,03,1st 25060008,0040,03, I am using the following in the script... (5 Replies)
Discussion started by: Amruta Pitkar
5 Replies

3. Shell Programming and Scripting

remove duplicated lines without sort

Hi Just wondering whether or not I can remove duplicated lines without sort For example, I use the command who, which shows users who are logging on. In some cases, it shows duplicated lines of users who are logging on more than one terminal. Normally, I would do who | cut -d" " -f1 |... (6 Replies)
Discussion started by: lalelle
6 Replies

4. Shell Programming and Scripting

Help with Uniq and sort

The key is first field i want only uniq record for the first field in file. I want the output as or output as Appreciate help on this (4 Replies)
Discussion started by: pinnacle
4 Replies

5. UNIX for Dummies Questions & Answers

Sort and uniq lines of a file while keeping a header line

So, I have a file that has some duplicate lines. The file has a header line that I would like to keep at the top. I could do this by extracting the header from the file, 'sort -u' the remaining lines, and recombine them. But they are quite big, so if there is a way to do it with a single... (1 Reply)
Discussion started by: Digby
1 Replies

6. Shell Programming and Scripting

Sort field and uniq

I have a flatfile A.txt 2012/12/04 14:06:07 |trees|Boards 2, 3|denver|mekong|mekong12 2012/12/04 17:07:22 |trees|Boards 2, 3|denver|mekong|mekong12 2012/12/04 17:13:27 |trees|Boards 2, 3|denver|mekong|mekong12 2012/12/04 14:07:39 |rain|Boards 1|tampa|merced|merced11 How do i sort and get... (3 Replies)
Discussion started by: sabercats
3 Replies

7. UNIX for Dummies Questions & Answers

Sort csv file by duplicated column value

hello, I have a large file (about 1gb) that is in a file similar to the following: I want to make it so that I can put all the duplicates where column 3 (delimited by the commas) are shown on top. Meaning all people with the same age are listed at the top. The command I used was ... (3 Replies)
Discussion started by: jl487
3 Replies

8. Shell Programming and Scripting

Uniq or sort -u or similar only between { }

Hi ! I am trying to remove doubbled entrys in a textfile only between delimiters. Like that example but i dont know how to do that with sort or similar. input: { aaa aaa } { aaa aaa } output: { aaa } { (8 Replies)
Discussion started by: fugitivus
8 Replies

9. UNIX for Dummies Questions & Answers

Uniq and sort -u

Hello all, Need to pick your brains, I have a 10Gb file where each row is a name, I am expecting about 50 names in total. So there are a lot of repetitions in clusters. So I want to do a sort -u file Will it be considerably faster or slower to use a uniq before piping it to sort... (3 Replies)
Discussion started by: senhia83
3 Replies

10. Shell Programming and Scripting

Sort & Uniq -u

Hi All, Below the actual file which i like to sort and Uniq -u /opt/oracle/work/Antony/Shell_Script> cat emp.1st 2233|a.k. shukula |g.m. |sales |12/12/52 |6000 1006|chanchal singhvi |director |sales |03/09/38 |6700... (8 Replies)
Discussion started by: Antony Ankrose
8 Replies
dos2unix(1)							   User Commands						       dos2unix(1)

NAME
dos2unix - convert text file from DOS format to ISO format SYNOPSIS
dos2unix [-ascii] [-iso] [-7] [-437 | -850 | -860 | -863 | -865] originalfile convertedfile DESCRIPTION
The dos2unix utility converts characters in the DOS extended character set to the corresponding ISO standard characters. This command can be invoked from either DOS or SunOS. However, the filenames must conform to the conventions of the environment in which the command is invoked. If the original file and the converted file are the same, dos2unix will rewrite the original file after converting it. OPTIONS
The following options are supported: -ascii Removes extra carriage returns and converts end of file characters in DOS format text files to conform to SunOS require- ments. -iso This is the default. It converts characters in the DOS extended character set to the corresponding ISO standard charac- ters. -7 Converts 8 bit DOS graphics characters to 7 bit space characters so that SunOS can read the file. On non-i386 systems, dos2unix will attempt to obtain the keyboard type to determine which code page to use. Otherwise, the default is US. The user may override the code page with one of the following options: -437 Use US code page -850 Use multilingual code page -860 Use Portuguese code page -863 Use French Canadian code page -865 Use Danish code page OPERANDS
The following operands are required: originalfile The original file in DOS format that is being converted to ISO format. convertedfile The new file in ISO format that has been converted from the original DOS file format. ATTRIBUTES
See attributes(5) for descriptions of the following attributes: +-----------------------------+-----------------------------+ | ATTRIBUTE TYPE | ATTRIBUTE VALUE | +-----------------------------+-----------------------------+ |Availability |SUNWesu | +-----------------------------+-----------------------------+ SEE ALSO
unix2dos(1), ls(1), attributes(5) DIAGNOSTICS
File filename not found, or no read permission The input file you specified does not exist, or you do not have read permission. Check with the SunOS command, ls -l (see ls(1)). Bad output filename filename, or no write permission The output file you specified is either invalid, or you do not have write permission for that file or the directory that contains it. Check also that the drive or diskette is not write-protected. Error while writing to temporary file An error occurred while converting your file, possibly because there is not enough space on the current drive. Check the amount of space on the current drive using the DIR command. Also be certain that the default diskette or drive is write-enabled (not write-pro- tected). Notice that when this error occurs, the original file remains intact. Translated temporary file name = filename. Could not rename temporary file to filename. The program could not perform the final step in converting your file. Your converted file is stored under the name indicated on the second line of this message. SunOS 5.10 14 Sep 2000 dos2unix(1)
All times are GMT -4. The time now is 08:13 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy