Sponsored Content
Top Forums Shell Programming and Scripting how many unique lines in a file Post 302355742 by ajp7701 on Wednesday 23rd of September 2009 03:05:39 PM
Old 09-23-2009
Bug how many unique lines in a file

I have a file, test.txt with approx 12,000 lines. Each line is a single word that looks like a hex address. There are many repeats. Over half of the lines are the same. I want to count how many UNIQUE lines there are.

#>more test.txt
Code:
0x123456
0x56AF23
0x99ABC1
0x123456
0x123456
0x99ABC1
0xADDE77
0x123456
0x123456
0x99ABC1

in this case there are 4 UNIQUE lines... How do I script this?
In other words, I need a script that only prints UNIQUE lines, then I can just wc (word count) the result.
Thank you very much!

It also might help to know that each line is a fixed amount of hex characters...the actual file that I need to count has 10 characters per line, for example 0x123456FF (so the above example isnt exact) but each line does begin with "0x" and then 8 hex characters.

Last edited by vgersh99; 09-23-2009 at 05:34 PM.. Reason: code tags, PLEASE!
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

extracting unique lines from text file

I have a file with 14million lines and I would like to extract all the unique lines from the file into another text file. For example: Contents of file1 happy sad smile happy funny sad I want to run a command against file one that only returns the unique lines (ie 1 line for happy... (3 Replies)
Discussion started by: soliberus
3 Replies

2. Shell Programming and Scripting

Comparing 2 files and return the unique lines in first file

Hi, I have 2 files file1 ******** 01-05-09|java.xls| 02-05-08|c.txt| 08-01-09|perl.txt| 01-01-09|oracle.txt| ******** file2 ******** 01-02-09|windows.xls| 02-05-08|c.txt| 01-05-09|java.xls| 08-02-09|perl.txt| 01-01-09|oracle.txt| ******** (8 Replies)
Discussion started by: shekhar_v4
8 Replies

3. Shell Programming and Scripting

get part of file with unique & non-unique string

I have an archive file that holds a batch of statements. I would like to be able to extract a certain statement based on the unique customer # (ie. 123456). The end for each statement is noted by "ENDSTM". I can find the line number for the beginning of the statement section with sed. ... (5 Replies)
Discussion started by: andrewsc
5 Replies

4. Shell Programming and Scripting

Remove All Lines Between Two Unique Lines

Hi all! Im wondering if its possible to remove all lines between two lines. Im working with a document like this: data1 data2 <Remove> data3 data4 </Remove> data5 data6 I need it to end up like this if that possible: data1 data2 data5 data6 There are multiple instances of... (2 Replies)
Discussion started by: Grizzly
2 Replies

5. UNIX for Advanced & Expert Users

In a huge file, Delete duplicate lines leaving unique lines

Hi All, I have a very huge file (4GB) which has duplicate lines. I want to delete duplicate lines leaving unique lines. Sort, uniq, awk '!x++' are not working as its running out of buffer space. I dont know if this works : I want to read each line of the File in a For Loop, and want to... (16 Replies)
Discussion started by: krishnix
16 Replies

6. Shell Programming and Scripting

Change unique file names into new unique filenames

I have 84 files with the following names splitseqs.1, spliseqs.2 etc. and I want to change the .number to a unique filename. E.g. change splitseqs.1 into splitseqs.7114_1#24 and change spliseqs.2 into splitseqs.7067_2#4 So all the current file names are unique, so are the new file names.... (1 Reply)
Discussion started by: avonm
1 Replies

7. Shell Programming and Scripting

compare 2 files and return unique lines in each file (based on condition)

hi my problem is little complicated one. i have 2 files which appear like this file 1 abbsss:aa:22:34:as akl abc 1234 mkilll:as:ss:23:qs asc abc 0987 mlopii:cd:wq:24:as asd abc 7866 file2 lkoaa:as:24:32:sa alk abc 3245 lkmo:as:34:43:qs qsa abc 0987 kloia:ds:45:56:sa acq abc 7805 i... (5 Replies)
Discussion started by: anurupa777
5 Replies

8. Shell Programming and Scripting

Combine multiple unique lines from event log text file into one line, use PERL or AWK?

I can't decide if I should use AWK or PERL after pouring over these forums for hours today I decided I'd post something and see if I couldn't get some advice. I've got a text file full of hundreds of events in this format: Record Number : 1 Records in Seq : ... (3 Replies)
Discussion started by: Mayday22
3 Replies

9. Shell Programming and Scripting

Transpose lines from individual blocks to unique lines

Hello to all, happy new year 2013! May somebody could help me, is about a very similar problem to the problem I've posted here where the member rdrtx1 and bipinajith helped me a lot. https://www.unix.com/shell-programming-scripting/211147-map-values-blocks-single-line-2.html It is very... (3 Replies)
Discussion started by: Ophiuchus
3 Replies

10. UNIX for Dummies Questions & Answers

Print unique lines without sort or unique

I would like to print unique lines without sort or unique. Unfortunately the server I am working on does not have sort or unique. I have not been able to contact the administrator of the server to ask him to add it for several weeks. (7 Replies)
Discussion started by: cokedude
7 Replies
WC(1)							    BSD General Commands Manual 						     WC(1)

NAME
wc -- word, line, and byte count SYNOPSIS
wc [-c | -m] [-Llw] [file ...] DESCRIPTION
The wc utility displays the number of lines, words, bytes and characters contained in each input file (or standard input, by default) to the standard output. A line is defined as a string of characters delimited by a <newline> character, and a word is defined as a string of char- acters delimited by white space characters. White space characters are the set of characters for which the iswspace(3) function returns true. If more than one input file is specified, a line of cumulative counts for all the files is displayed on a separate line after the out- put for the last file. The following options are available: -c The number of bytes in each input file is written to the standard output. -L The number of characters in the longest line of each input file is written to the standard output. -l The number of lines in each input file is written to the standard output. -m The number of characters in each input file is written to the standard output. -w The number of words in each input file is written to the standard output. When an option is specified, wc only reports the information requested by that option. The default action is equivalent to all the flags -clw having been specified. The following operands are available: file A pathname of an input file. If no file names are specified, the standard input is used and no file name is displayed. By default, the standard output contains a line for each input file of the form: lines words bytes file_name EXIT STATUS
The wc utility exits 0 on success, and >0 if an error occurs. SEE ALSO
iswspace(3) COMPATIBILITY
Historically, the wc utility was documented to define a word as a ``maximal string of characters delimited by <space>, <tab> or <newline> characters''. The implementation, however, didn't handle non-printing characters correctly so that `` ^D^E '' counted as 6 spaces, while ``foo^D^Ebar'' counted as 8 characters. 4BSD systems after 4.3BSD modified the implementation to be consistent with the documentation. This implementation defines a ``word'' in terms of the iswspace(3) function, as required by IEEE Std 1003.2 (``POSIX.2''). The -L option is a non-standard extension, compatible with the -L option of the GNU and FreeBSD wc utilities. STANDARDS
The wc utility conforms to IEEE Std 1003.2-1992 (``POSIX.2''). BSD
February 18, 2010 BSD
All times are GMT -4. The time now is 06:08 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy