I have a file, test.txt with approx 12,000 lines. Each line is a single word that looks like a hex address. There are many repeats. Over half of the lines are the same. I want to count how many UNIQUE lines there are.
#>more test.txt
in this case there are 4 UNIQUE lines... How do I script this?
In other words, I need a script that only prints UNIQUE lines, then I can just wc (word count) the result.
Thank you very much!
It also might help to know that each line is a fixed amount of hex characters...the actual file that I need to count has 10 characters per line, for example 0x123456FF (so the above example isnt exact) but each line does begin with "0x" and then 8 hex characters.
Last edited by vgersh99; 09-23-2009 at 05:34 PM..
Reason: code tags, PLEASE!
I have a file with 14million lines and I would like to extract all the unique lines from the file into another text file.
For example:
Contents of file1
happy
sad
smile
happy
funny
sad
I want to run a command against file one that only returns the unique lines (ie 1 line for happy... (3 Replies)
I have an archive file that holds a batch of statements. I would like to be able to extract a certain statement based on the unique customer # (ie. 123456). The end for each statement is noted by "ENDSTM".
I can find the line number for the beginning of the statement section with sed.
... (5 Replies)
Hi all! Im wondering if its possible to remove all lines between two lines. Im working with a document like this:
data1
data2
<Remove>
data3
data4
</Remove>
data5
data6
I need it to end up like this if that possible:
data1
data2
data5
data6
There are multiple instances of... (2 Replies)
Hi All,
I have a very huge file (4GB) which has duplicate lines. I want to delete duplicate lines leaving unique lines. Sort, uniq, awk '!x++' are not working as its running out of buffer space.
I dont know if this works : I want to read each line of the File in a For Loop, and want to... (16 Replies)
I have 84 files with the following names splitseqs.1, spliseqs.2 etc.
and I want to change the .number to a unique filename.
E.g.
change splitseqs.1 into splitseqs.7114_1#24
and
change spliseqs.2 into splitseqs.7067_2#4
So all the current file names are unique, so are the new file names.... (1 Reply)
hi
my problem is little complicated one. i have 2 files which appear like this
file 1
abbsss:aa:22:34:as akl abc 1234
mkilll:as:ss:23:qs asc abc 0987
mlopii:cd:wq:24:as asd abc 7866
file2
lkoaa:as:24:32:sa alk abc 3245
lkmo:as:34:43:qs qsa abc 0987
kloia:ds:45:56:sa acq abc 7805
i... (5 Replies)
I can't decide if I should use AWK or PERL after pouring over these forums for hours today I decided I'd post something and see if I couldn't get some advice.
I've got a text file full of hundreds of events in this format:
Record Number : 1
Records in Seq : ... (3 Replies)
Hello to all, happy new year 2013!
May somebody could help me, is about a very similar problem to the problem I've posted here where the member rdrtx1 and bipinajith helped me a lot.
https://www.unix.com/shell-programming-scripting/211147-map-values-blocks-single-line-2.html
It is very... (3 Replies)
I would like to print unique lines without sort or unique. Unfortunately the server I am working on does not have sort or unique. I have not been able to contact the administrator of the server to ask him to add it for several weeks. (7 Replies)
Discussion started by: cokedude
7 Replies
LEARN ABOUT NETBSD
wc
WC(1) BSD General Commands Manual WC(1)NAME
wc -- word, line, and byte count
SYNOPSIS
wc [-c | -m] [-Llw] [file ...]
DESCRIPTION
The wc utility displays the number of lines, words, bytes and characters contained in each input file (or standard input, by default) to the
standard output. A line is defined as a string of characters delimited by a <newline> character, and a word is defined as a string of char-
acters delimited by white space characters. White space characters are the set of characters for which the iswspace(3) function returns
true. If more than one input file is specified, a line of cumulative counts for all the files is displayed on a separate line after the out-
put for the last file.
The following options are available:
-c The number of bytes in each input file is written to the standard output.
-L The number of characters in the longest line of each input file is written to the standard output.
-l The number of lines in each input file is written to the standard output.
-m The number of characters in each input file is written to the standard output.
-w The number of words in each input file is written to the standard output.
When an option is specified, wc only reports the information requested by that option. The default action is equivalent to all the flags
-clw having been specified.
The following operands are available:
file A pathname of an input file.
If no file names are specified, the standard input is used and no file name is displayed.
By default, the standard output contains a line for each input file of the form:
lines words bytes file_name
EXIT STATUS
The wc utility exits 0 on success, and >0 if an error occurs.
SEE ALSO iswspace(3)COMPATIBILITY
Historically, the wc utility was documented to define a word as a ``maximal string of characters delimited by <space>, <tab> or <newline>
characters''. The implementation, however, didn't handle non-printing characters correctly so that `` ^D^E '' counted as 6 spaces, while
``foo^D^Ebar'' counted as 8 characters. 4BSD systems after 4.3BSD modified the implementation to be consistent with the documentation. This
implementation defines a ``word'' in terms of the iswspace(3) function, as required by IEEE Std 1003.2 (``POSIX.2'').
The -L option is a non-standard extension, compatible with the -L option of the GNU and FreeBSD wc utilities.
STANDARDS
The wc utility conforms to IEEE Std 1003.2-1992 (``POSIX.2'').
BSD February 18, 2010 BSD