Sponsored Content
Top Forums Shell Programming and Scripting Eliminate redundant data pairs Post 302383642 by joeyg on Wednesday 30th of December 2009 11:51:34 AM
Old 12-30-2009
Tools Be aware of original file data

If original data is not sorted, the uniq command will give strange results. You may need to sort the data prior to a uniq command.
I manually placed a few of the last entries at the beginning of the file, and see the results:
Code:
>uniq uniqsample.txt
0.2 17
0.0 3
0.0 6
0.0 7
0.0 11
0.1 17
0.2 17

 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

to check redundant file names

hi i have a very simple problem iam moving files from download to archive folder but before such a transfer want to make sure no two file of same are present in my download directory how to check for redundant file names i thought of using WC but it counts inside the file (lines and... (5 Replies)
Discussion started by: maverick
5 Replies

2. UNIX for Dummies Questions & Answers

Question is redundant but please advice

I am really really new to Unix. I'm lost with so many books around for different shell. I'm thinking of taking a course on Operating Systems but it contains a lot of Unix programming I think. For example, someone was talking about a "which" command. But I wasn't able to figure out what it does...... (10 Replies)
Discussion started by: Legend986
10 Replies

3. Shell Programming and Scripting

Eliminate unwanted data

Hi, I am stuck on writing a script that reads a file, retains wanted data but discards unwanted data from a CSV file. This is a sample of my table {not the actual data) ID Color tel_num Name color2 color3 abcdef green 5551212 jj88 red blue acbdfe yellow... (5 Replies)
Discussion started by: t524ube
5 Replies

4. Filesystems, Disks and Memory

ZFS Raidz not redundant?

My ZFS on debian media server just died in a power outage, the zpool status shows this: NAME STATE READ WRITE CKSUM tank UNAVAIL 0 0 0 insufficient replicas raidz1 UNAVAIL 0 0 0 corrupted data sda ONLINE 0 0 0 sdb ONLINE 0 0 0 sdf ONLINE 0 0 0 sdh ONLINE 0 0 0 sdi ONLINE 0 0 0 sdk ONLINE 0... (2 Replies)
Discussion started by: mastersarg
2 Replies

5. UNIX for Dummies Questions & Answers

Deleting all rows that contain redundant information

My input file looks like this: 1 rs4040617 0.08356 1 rs4040617 0.06799 1 rs2977612 0.07948 1 rs2977612 0.07882 1 rs2977612 0.07783 1 rs2977612 0.08142 1 rs2977612 0.07716 1 rs2977612 0.08356 1 rs2977612 0.06799 1 rs2980300 0.08356 1 rs2980300 0.08142 I want to delete all rows that... (1 Reply)
Discussion started by: evelibertine
1 Replies

6. Shell Programming and Scripting

Find redundant text in a file

I want to find which pattern or strings have occurred more than one time so that I can remove unnecessary redundancy. For example: If I have the sentence: A quick brown brown fox jumps jumps jumps over the lazy dog in a file, then I want to know that 1. the word "brown" has... (7 Replies)
Discussion started by: hbar
7 Replies

7. Shell Programming and Scripting

Edit a script in 2 redundant servers at once

Hi All, We have a redundant server system and every time we need any change in any particular script then we need to do it individually in each of the server. I want to create a shell script such that if i make changes in primary server and run the script then it should update the changes to ... (0 Replies)
Discussion started by: crimemastergogo
0 Replies

8. Shell Programming and Scripting

Reduce redundant file

Dear All, I have to reduce the redundancy of a file that is like this: a b 0 a c 0 a f 1 b a 1 b a 0 b c 1 d f 0 g h 1 f d 1 Basically, this file describe a network with relative nodes and edges. The nodes are the different letters and the edges are represented by the numbers (in... (7 Replies)
Discussion started by: giuliangiuseppe
7 Replies

9. Shell Programming and Scripting

Extracting data between two tag pairs

In a huge log file (43MB, 43k lines) I am trying to extract data between two tag pairs on same line and export it to a file so I can pull it into Excel for a report. One Pair is <Text>data I need</Text> Other pair follows on same line and is <TimeStamp>more data I need</TimeStamp> I would need... (2 Replies)
Discussion started by: NanookArctic
2 Replies

10. Shell Programming and Scripting

Eliminate or ignore asterisks in data when parsing

I have data file that has this in it: data.txt ......... ......... PPJ97**2017PPJ97**2017-03-21-13.35.15.887208********************START ERROR LOGGING****************** PPJ97**2017-03-21-13.35.15.887208** PROMPT APPLICATION ERROR ** PPJ97**2017-03-21-13.35.15.887208** IN TIMESTAMP |... (1 Reply)
Discussion started by: SkySmart
1 Replies
UNIQ(1) 						    BSD General Commands Manual 						   UNIQ(1)

NAME
uniq -- report or filter out repeated lines in a file SYNOPSIS
uniq [-c | -d | -u] [-i] [-f num] [-s chars] [input_file [output_file]] DESCRIPTION
The uniq utility reads the specified input_file comparing adjacent lines, and writes a copy of each unique input line to the output_file. If input_file is a single dash ('-') or absent, the standard input is read. If output_file is absent, standard output is used for output. The second and succeeding copies of identical adjacent input lines are not written. Repeated lines in the input will not be detected if they are not adjacent, so it may be necessary to sort the files first. The following options are available: -c Precede each output line with the count of the number of times the line occurred in the input, followed by a single space. -d Only output lines that are repeated in the input. -f num Ignore the first num fields in each input line when doing comparisons. A field is a string of non-blank characters separated from adjacent fields by blanks. Field numbers are one based, i.e., the first field is field one. -s chars Ignore the first chars characters in each input line when doing comparisons. If specified in conjunction with the -f option, the first chars characters after the first num fields will be ignored. Character numbers are one based, i.e., the first character is character one. -u Only output lines that are not repeated in the input. -i Case insensitive comparison of lines. ENVIRONMENT
The LANG, LC_ALL, LC_COLLATE and LC_CTYPE environment variables affect the execution of uniq as described in environ(7). EXIT STATUS
The uniq utility exits 0 on success, and >0 if an error occurs. COMPATIBILITY
The historic +number and -number options have been deprecated but are still supported in this implementation. SEE ALSO
sort(1) STANDARDS
The uniq utility conforms to IEEE Std 1003.1-2001 (``POSIX.1'') as amended by Cor. 1-2002. HISTORY
A uniq command appeared in Version 3 AT&T UNIX. BSD
December 17, 2009 BSD
All times are GMT -4. The time now is 04:31 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy