Sponsored Content
Top Forums Shell Programming and Scripting Remove duplicate lines after ignoring case and spaces between Post 302951527 by RudiC on Friday 7th of August 2015 05:41:44 AM
Old 08-07-2015
Any attempts from your side?

---------- Post updated at 11:41 ---------- Previous update was at 11:40 ----------

Howsoever, try
Code:
awk '
                        {(gsub(/ +/," "))}
!T[toupper($0)]++
' file
GRANT select on MANHPRD.S_PROD_INT TO OR_PHIL;
GRANT select on SCOTT.emp to JOHN;
grant select on scott.dept to hr;

This User Gave Thanks to RudiC For This Post:
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Removing duplicate lines ignore case

hi, I have the following input in file: abc ab a AB b c a C B When I use uniq -u file,the out put file is: abc ab AB c v B C (17 Replies)
Discussion started by: hellsd
17 Replies

2. Shell Programming and Scripting

how to remove duplicate lines

I have following file content (3 fields each line): 23 888 10.0.0.1 dfh 787 10.0.0.2 dssf dgfas 10.0.0.3 dsgas dg 10.0.0.4 df dasa 10.0.0.5 df dag 10.0.0.5 dfd dfdas 10.0.0.5 dfd dfd 10.0.0.6 daf nfd 10.0.0.6 ... as can be seen, that the third field is ip address and sorted. but... (3 Replies)
Discussion started by: fredao
3 Replies

3. UNIX for Dummies Questions & Answers

Remove Duplicate lines from File

I have a log file "logreport" that contains several lines as seen below: 04:20:00 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 06:38:08 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 07:11:05 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but... (18 Replies)
Discussion started by: Nysif Steve
18 Replies

4. UNIX for Dummies Questions & Answers

deleteing duplicate lines sing uniq while ignoring a column

I have a data set that has 4 columns, I want to know if I can delete duplicate lines while ignoring one of the columns, for example 10 chr1 ASF 30 15 chr1 ASF 20 5 chr1 ASF 30 6 chr2 EBC 15 4 chr2 EBC 30 ... I want to know if I can delete duplicate lines while ignoring column 1, so the... (5 Replies)
Discussion started by: japaneseguitars
5 Replies

5. Shell Programming and Scripting

sed ignoring case for search but respecting case for subtitute

Hi I want to make string substitution ignoring case for search but respecting case for subtitute. Ex changing all occurences of "original" in a file to "substitute": original becomes substitute Origninal becomes Substitute ORIGINAL becomes SUBSTITUTE I know this a little special but it's not... (1 Reply)
Discussion started by: kmchen
1 Replies

6. Shell Programming and Scripting

Remove duplicate lines

Hi, I have a huge file which is about 50GB. There are many lines. The file format likes 21 rs885550 0 9887804 C C T C C C C C C C 21 rs210498 0 9928860 0 0 C C 0 0 0 0 0 0 21 rs303304 0 9941889 A A A A A A A A A A 22 rs303304 0 9941890 0 A A A A A A A A A The question is that there are a few... (4 Replies)
Discussion started by: zhshqzyc
4 Replies

7. UNIX for Dummies Questions & Answers

Remove Duplicate Lines

Hi I need this output. Thanks. Input: TAZ YET FOO FOO VAK TAZ BAR Output: YET VAK BAR (10 Replies)
Discussion started by: tara123
10 Replies

8. Shell Programming and Scripting

Count duplicate lines ignoring certain columns

I have this structure: col1 col2 col3 col4 col5 27 xxx 38 aaa ttt 2 xxx 38 aaa yyy 1 xxx 38 aaa yyy I need to collapse duplicate lines ignoring column 1 and add values of duplicate lines (col1) so it will look like this: col1 col2 col3 col4 col5 27 xxx 38 aaa ttt ... (3 Replies)
Discussion started by: coppuca
3 Replies

9. Shell Programming and Scripting

Remove lines containing 2 or more duplicate strings

Within my text file i have several thousand lines of text with some lines containing duplicate strings/words. I would like to entirely remove those lines which contain the duplicate strings. Eg; One and a Two Unix.com is the Best This as a Line Line Example duplicate sentence with the word... (22 Replies)
Discussion started by: martinsmith
22 Replies

10. Shell Programming and Scripting

How to remove duplicate lines?

Hi All, I am storing the result in the variable result_text using the below code. result_text=$(printf "$result_text\t\n$name") The result_text is having the below text. Which is having duplicate lines. file and time for the interval 03:30 - 03:45 file and time for the interval 03:30 - 03:45 ... (4 Replies)
Discussion started by: nalu
4 Replies
OD(1)                                                              User Commands                                                             OD(1)

NAME
od - dump files in octal and other formats SYNOPSIS
od [OPTION]... [FILE]... od [-abcdfilosx]... [FILE] [[+]OFFSET[.][b]] od --traditional [OPTION]... [FILE] [[+]OFFSET[.][b] [+][LABEL][.][b]] DESCRIPTION
Write an unambiguous representation, octal bytes by default, of FILE to standard output. With more than one FILE argument, concatenate them in the listed order to form the input. With no FILE, or when FILE is -, read standard input. If first and second call formats both apply, the second format is assumed if the last operand begins with + or (if there are 2 operands) a digit. An OFFSET operand means -j OFFSET. LABEL is the pseudo-address at first byte printed, incremented when dump is progressing. For OFFSET and LABEL, a 0x or 0X prefix indicates hexadecimal; suffixes may be . for octal and b for multiply by 512. Mandatory arguments to long options are mandatory for short options too. -A, --address-radix=RADIX output format for file offsets; RADIX is one of [doxn], for Decimal, Octal, Hex or None --endian={big|little} swap input bytes according the specified order -j, --skip-bytes=BYTES skip BYTES input bytes first -N, --read-bytes=BYTES limit dump to BYTES input bytes -S BYTES, --strings[=BYTES] output strings of at least BYTES graphic chars; 3 is implied when BYTES is not specified -t, --format=TYPE select output format or formats -v, --output-duplicates do not use * to mark line suppression -w[BYTES], --width[=BYTES] output BYTES bytes per output line; 32 is implied when BYTES is not specified --traditional accept arguments in third form above --help display this help and exit --version output version information and exit Traditional format specifications may be intermixed; they accumulate: -a same as -t a, select named characters, ignoring high-order bit -b same as -t o1, select octal bytes -c same as -t c, select printable characters or backslash escapes -d same as -t u2, select unsigned decimal 2-byte units -f same as -t fF, select floats -i same as -t dI, select decimal ints -l same as -t dL, select decimal longs -o same as -t o2, select octal 2-byte units -s same as -t d2, select decimal 2-byte units -x same as -t x2, select hexadecimal 2-byte units TYPE is made up of one or more of these specifications: a named character, ignoring high-order bit c printable character or backslash escape d[SIZE] signed decimal, SIZE bytes per integer f[SIZE] floating point, SIZE bytes per float o[SIZE] octal, SIZE bytes per integer u[SIZE] unsigned decimal, SIZE bytes per integer x[SIZE] hexadecimal, SIZE bytes per integer SIZE is a number. For TYPE in [doux], SIZE may also be C for sizeof(char), S for sizeof(short), I for sizeof(int) or L for sizeof(long). If TYPE is f, SIZE may also be F for sizeof(float), D for sizeof(double) or L for sizeof(long double). Adding a z suffix to any type displays printable characters at the end of each output line. BYTES is hex with 0x or 0X prefix, and may have a multiplier suffix: b 512 KB 1000 K 1024 MB 1000*1000 M 1024*1024 and so on for G, T, P, E, Z, Y. EXAMPLES
od -A x -t x1z -v Display hexdump format output od -A o -t oS -w16 The default output format used by od AUTHOR
Written by Jim Meyering. REPORTING BUGS
GNU coreutils online help: <http://www.gnu.org/software/coreutils/> Report od translation bugs to <http://translationproject.org/team/> COPYRIGHT
Copyright (C) 2017 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>. This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. SEE ALSO
Full documentation at: <http://www.gnu.org/software/coreutils/od> or available locally via: info '(coreutils) od invocation' GNU coreutils 8.28 January 2018 OD(1)
All times are GMT -4. The time now is 03:22 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy