Sponsored Content
Top Forums Shell Programming and Scripting Help with remove duplicate content and only keep the first content detail Post 302481858 by R0H0N on Monday 20th of December 2010 05:19:11 AM
Old 12-20-2010
Quote:
Originally Posted by patrick87
Hi ROHON,
I just try it out.
It seems like can't get desired output result?
Thanks.

Code:
cat input_file | cut -f2 | uniq | while read line
do
   grep " ${line}$" input_file | head -1 >> output_file
done

R0H0N
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove duplicate line detail based on column one data

My input file: AVI.out <detail>named as the RRM .</detail> AVI.out <detail>Contains 1 RRM .</detail> AR0.out <detail>named as the tellurite-resistance.</detail> AWG.out <detail>Contains 2 HTH .</detail> ADV.out <detail>named as the DENR family.</detail> ADV.out ... (10 Replies)
Discussion started by: patrick87
10 Replies

2. Shell Programming and Scripting

Way to extract detail and its content above specific value problem asking

Input file: >position_10 sample:68711 coords:5453-8666 number:3 type:complete len:344 MSINQYSSDFHYHSLMWQQQQQQQQHQNDVVEEKEALFEKPLTPSDVGKLNRLVIPKQHA ERYFPLAAAAADAVEKGLLLCFEDEEGKPWRFRYSYWNSSQSYVLTKGWSRYVKEKHLDA NRTS* >position_4 sample:68711 coords:553-866 number:4 type:partial len:483... (7 Replies)
Discussion started by: patrick87
7 Replies

3. Shell Programming and Scripting

Help with remove duplicated content

Input file: hcmv-US25-2-3p hsa-3160-5 hcmv-US33 hsa-47 hcmv-UL70-3p hsa-4508 hcmv-UL70-3p hsa-4486 hcms-US25 hsa-360-5 hcms-US25 hsa-4 hcms-US25 hsa-458 hcms-US25 hsa-44812 . . Desired Output file: hcmv-US25-2-3p hsa-3160-5 hcmv-US33 hsa-47 hcmv-UL70-3p hsa-4508 hsa-4486... (3 Replies)
Discussion started by: perl_beginner
3 Replies

4. Shell Programming and Scripting

Help with remove duplicate content

Input file data_1 10 US data_1 2 US data_1 5 UK data_2 20 ENGLAND data_2 12 KOREA data_3 4 CHINA . . data_60 123 US data_60 23 UK data_60 45 US Desired output file data_1 10 US data_1 5 UK data_2 20 ENGLAND data_2 12 KOREA (2 Replies)
Discussion started by: perl_beginner
2 Replies

5. Shell Programming and Scripting

Help with replace duplicate content

Input file: CCNI data564_input1 264 CORO1A data564_input2 155 ABC-B data17_input1 3466 ABC-B data17_input2 1133 ABC-B data17_input3 2162 ABC-B data17_input4 2019 HNRNPA2B1 data95_input1 101 HNRNPA2B1 data95_input2 340 IFITM1 data105_input2 291 IFITM2 data105_input1 505... (3 Replies)
Discussion started by: cpp_beginner
3 Replies

6. Shell Programming and Scripting

Help with duplicate data content problem asking

Input file: A_69510335_ASD>aw 1199470 USA A_119571157_C>awe,QWEQE 113932840 USA C_34646666_qwe>TAWTT,G,TT 112736796 UK C_69510335_QW>T 1199470 USA D_70520237_WR>QEE,G 34459863 UK D_71380003_QWR>T 145418226 IK . Desired output: A_69510335_ASD>aw 1199470 USA... (1 Reply)
Discussion started by: perl_beginner
1 Replies

7. Shell Programming and Scripting

Help with duplicate common data content

Input file: #data_131 0 >content..._* 1 >content..._at_+/97.20% #data_137 0 >content..._* 1 >content..._at_+/97.20% 2 >seq..._* 3 >content..._at_+/97.20% 4 >content..._at_+/97.20% #data_141 0 >content..._* #data_150 0 >content..._* 1 >content..._at_+/97.20% 2 >seq..._* 3... (3 Replies)
Discussion started by: perl_beginner
3 Replies

8. Shell Programming and Scripting

Facing issues with Content-Type:application/x-download Content-Disposition:attachment

I am in the process of developing a perl cgi page. I had succeeded in developing the page but there are few errors/issues with the page. description about cgi page: My CGI page retrieves all the file names from an directory and displays the files in drop down menu for downloading the... (5 Replies)
Discussion started by: scriptscript
5 Replies

9. Shell Programming and Scripting

Remove the duplicate content in a file

Here is the contents of test.txt Dependencies Resolved Changes in packages about to be updated: ChangeLog for: 1:perl-Archive-Extract-0.38-131.el6_4.x86_64, - Resolves: #915692 - CVE-2013-1667 (DoS in rehashing code) Dependencies Resolved Changes in packages about to be updated: ... (5 Replies)
Discussion started by: ashokvpp
5 Replies

10. Shell Programming and Scripting

How to remove exisiting file content from a file and have to append new file content?

hi all, i had the below script x=`cat input.txt |wc -1` awk 'NR>1 && NR<'$x' ' input.txt > output.txt by using above script i am able to remove the head and tail part from the input file and able to append the output to the output.txt but if i run it for second time the output is... (2 Replies)
Discussion started by: hemanthsaikumar
2 Replies
uniq(1) 						      General Commands Manual							   uniq(1)

NAME
uniq - Removes or lists repeated lines in a file SYNOPSIS
Current Syntax uniq [-cdu] [-f fields] [-s chars] [input-file [output-file]] Obsolescent Syntax uniq [-cdu] [-fields] [+chars] [input-file [output-file]] The uniq command reads from the specified input_file, compares adjacent lines, removes the second and succeeding occurrences of a line, and writes to standard output. STANDARDS
Interfaces documented on this reference page conform to industry standards as follows: uniq: XCU5.0 Refer to the standards(5) reference page for more information about industry standards and associated tags. OPTIONS
Precedes each output line with a count of the number of times each line appears in the file. This option supersedes the -d and -u options. Displays repeated lines only. Ignores the first fields fields on each input line when doing comparisons, where fields is a positive deci- mal integer. A field is the maximal string matched by the basic regular expression: [[:blank:]]*[^[:blank:]]* If the fields argument specifies more fields than appear on an input line, a null string is used for comparisons. Ignores the spec- ified number of characters when doing comparisons. The chars argument is a positive decimal integer. If specified with the -f option, the first chars characters after the first fields fields are ignored. If the chars argument speci- fies more characters than remain on an input line, uniq uses a null string for comparison. Displays unique lines only. Equivalent to -f fields. (Obsolescent) Equivalent to -s chars. (Obsolescent) OPERANDS
A pathname for the input file. If this operand is omitted or specified as -, then standard input is read. A pathname for the output file. If this operand is omitted, then standard output is written. DESCRIPTION
The input_file and output_file arguments must be different files. If the input_file operand is not specified, or if it is -, uniq uses standard input. Repeated lines must be on consecutive lines to be found. You can arrange them with the sort command before processing. EXAMPLES
To delete repeated lines in the following file called fruit and save it to a file named newfruit, enter: uniq fruit newfruit The file fruit contains the following lines: apples apples bananas cherries cherries peaches pears The file newfruit contains the following lines: apples bananas cherries peaches pears EXIT STATUS
The following exit values are returned: Successful completion. An error occurred. ENVIRONMENT VARIABLES
The following environment variables affect the execution of uniq: Provides a default value for the internationalization variables that are unset or null. If LANG is unset or null, the corresponding value from the default locale is used. If any of the internationalization vari- ables contain an invalid setting, the utility behaves as if none of the variables had been defined. If set to a non-empty string value, overrides the values of all the other internationalization variables. Determines the locale for the interpretation of sequences of bytes of text data as characters (for example, single-byte as opposed to multibyte characters in arguments). Determines the locale for the for- mat and contents of diagnostic messages written to standard error. Determines the location of message catalogues for the processing of LC_MESSAGES. SEE ALSO
Commands: comm(1), sort(1) Standards: standards(5) uniq(1)
All times are GMT -4. The time now is 10:01 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy