Sponsored Content
Top Forums Shell Programming and Scripting Deleting repeated strings in column 2 Post 302319293 by panyam on Monday 25th of May 2009 02:30:31 AM
Old 05-25-2009
To Remove the repeated lines and to print one copy .

Code:
awk '/^Comment/ { print $1,substr($2,1,length($2)-1),$3,$4,$5,$6 }' inputfile.txt | uniq -ud

 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

repeated column data filter and make as a row

I need to get the output in row wise for the repeated column data Ex: Input: que = five ans = 5 que = six ans = 6 Required output: que = five six ans = 5 6 Any body can guide me?"""""" (2 Replies)
Discussion started by: vasanth_vadalur
2 Replies

2. Shell Programming and Scripting

delete repeated strings (tags) in a line and concatenate corresponding words

Hello friends! Each line of my input file has this format: word<TAB>tag1<blankspace>lemma<TAB>tag2<blankspace>lemma ... <TAB>tag3<blankspace>lemma Of this file I need to eliminate all the repeated tags (of the same word) in a line, as in the example here below, but conserving both (all) the... (2 Replies)
Discussion started by: mjomba
2 Replies

3. UNIX for Dummies Questions & Answers

Average for repeated elements in a column

I have a file that looks like this 452 025_E3 8 025_E3 82 025_F5 135 025_F5 5 025_F5 23 025_G2 38 025_G2 71 025_G2 9 026_A12 81 026_A12 10 026_A12 some of the elements in column2 are repeated. I want an output file that will extract the... (1 Reply)
Discussion started by: FelipeAd
1 Replies

4. Emergency UNIX and Linux Support

[Solved] Extract records based on a repeated column value

Hi guys, I need help in making a command to find some data. I have multiple files in which multiple records are present.. Each record is separated with a carriage return and in each record there are multiple fields with each field separated by "|" what i want is that I want to extract... (1 Reply)
Discussion started by: m_usmanayub
1 Replies

5. Shell Programming and Scripting

Finding most repeated entry in a column and giving the count

Please can you help in providing the most repeated entry in the 2nd column and give its count Here is an input file 1, This , is a forum 2, This , is a forum 1, There , is a forum 2, This , is not right Here the most repeated entry is "This" and count is 3 So output... (4 Replies)
Discussion started by: necro98
4 Replies

6. Shell Programming and Scripting

Deleting repeated lines by keeping only one.

Dear Buddies, Need ur help once again. I have a flat file with around 20 million lines (Huge file it is). However, many of the lines are of no use hence I want to remove it. To find and delete such lines we have certain codes written at the starting of each line. Basis that we can delete the... (2 Replies)
Discussion started by: anushree.a
2 Replies

7. Shell Programming and Scripting

Getting the most repeated column

Hi all , i want to get the most repeated column in my file File: name,ID adam,12345 ----1 adam,12345 ----2 adam,934 adam,12345 ----3 john,14 john,13 john,25 ----1 john,25 ----2 tom,1 -----1 tom,2 -----1 so my output to be (5 Replies)
Discussion started by: teefa
5 Replies

8. UNIX for Dummies Questions & Answers

Joining ends of strings in certain order with repeated ID's

I posted this a few days ago and got some help (Putting together substrings if pattern is matched - Page 2 | Unix Linux Forums | Shell Programming and Scripting) But I am now stuck on an issue that is similar but not the same really. I want to join parts of one line with parts of another line... (8 Replies)
Discussion started by: verse123
8 Replies

9. UNIX for Dummies Questions & Answers

Append no of times a column is repeated at the end

Hi folks, Iam working on a bash script, i need to print how many times column 2 repeated at the end of each line. Input.txt COL1 COL2 COL3 COL4 1 XX 45 N 2 YY 34 y 3 ZZ 44 N 4 XX 89 Y 5 XX 45 N 6 YY 84 D 7 ZZ 22 S Output.txt COL1 COL2 COL3 COL4 COL5 1 XX 45 N 3 2 YY 34... (6 Replies)
Discussion started by: tech_frk
6 Replies

10. Shell Programming and Scripting

Help tabulating file putting repeated strings as headers

Hi. May somebody help me with this. I´m trying to tabulate the following input file, but the desired output I´m getting is incorrect. I have access to GNU/LINUX (Ubuntu) and Cygwin Input file STAGE = 1 ID = 0 NAME = JFMSC TYPE = MLRR DFRUL = PERMISSION ADDR = 1001 RRUL =... (10 Replies)
Discussion started by: Ophiuchus
10 Replies
UNIQ(1) 						    BSD General Commands Manual 						   UNIQ(1)

NAME
uniq -- report or filter out repeated lines in a file SYNOPSIS
uniq [-c | -d | -u] [-i] [-f num] [-s chars] [input_file [output_file]] DESCRIPTION
The uniq utility reads the specified input_file comparing adjacent lines, and writes a copy of each unique input line to the output_file. If input_file is a single dash ('-') or absent, the standard input is read. If output_file is absent, standard output is used for output. The second and succeeding copies of identical adjacent input lines are not written. Repeated lines in the input will not be detected if they are not adjacent, so it may be necessary to sort the files first. The following options are available: -c Precede each output line with the count of the number of times the line occurred in the input, followed by a single space. -d Only output lines that are repeated in the input. -f num Ignore the first num fields in each input line when doing comparisons. A field is a string of non-blank characters separated from adjacent fields by blanks. Field numbers are one based, i.e. the first field is field one. -s chars Ignore the first chars characters in each input line when doing comparisons. If specified in conjunction with the -f option, the first chars characters after the first num fields will be ignored. Character numbers are one based, i.e. the first character is character one. -u Only output lines that are not repeated in the input. -i Case insensitive comparison of lines. DIAGNOSTICS
The uniq utility exits 0 on success, and >0 if an error occurs. COMPATIBILITY
The historic +number and -number options have been deprecated but are still supported in this implementation. SEE ALSO
sort(1) STANDARDS
The uniq utility is expected to be IEEE Std 1003.2 (``POSIX.2'') compatible. HISTORY
A uniq command appeared in Version 3 AT&T UNIX. BSD
June 6, 1993 BSD
All times are GMT -4. The time now is 10:20 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy