Sponsored Content
Top Forums Shell Programming and Scripting Remove duplicate lines after ignoring case and spaces between Post 302951719 by summer_cherry on Monday 10th of August 2015 03:47:45 AM
Old 08-10-2015
python

Code:
cache={}
with open("a.txt") as file:
	for line in file:
		line=line.replace("\n","")
		key=" ".join([i.lower() for i in filter(lambda x: x!="",line.split(" "))])
		if key not in cache:
			print(key)
			cache[key]=1

 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Removing duplicate lines ignore case

hi, I have the following input in file: abc ab a AB b c a C B When I use uniq -u file,the out put file is: abc ab AB c v B C (17 Replies)
Discussion started by: hellsd
17 Replies

2. Shell Programming and Scripting

how to remove duplicate lines

I have following file content (3 fields each line): 23 888 10.0.0.1 dfh 787 10.0.0.2 dssf dgfas 10.0.0.3 dsgas dg 10.0.0.4 df dasa 10.0.0.5 df dag 10.0.0.5 dfd dfdas 10.0.0.5 dfd dfd 10.0.0.6 daf nfd 10.0.0.6 ... as can be seen, that the third field is ip address and sorted. but... (3 Replies)
Discussion started by: fredao
3 Replies

3. UNIX for Dummies Questions & Answers

Remove Duplicate lines from File

I have a log file "logreport" that contains several lines as seen below: 04:20:00 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 06:38:08 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 07:11:05 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but... (18 Replies)
Discussion started by: Nysif Steve
18 Replies

4. UNIX for Dummies Questions & Answers

deleteing duplicate lines sing uniq while ignoring a column

I have a data set that has 4 columns, I want to know if I can delete duplicate lines while ignoring one of the columns, for example 10 chr1 ASF 30 15 chr1 ASF 20 5 chr1 ASF 30 6 chr2 EBC 15 4 chr2 EBC 30 ... I want to know if I can delete duplicate lines while ignoring column 1, so the... (5 Replies)
Discussion started by: japaneseguitars
5 Replies

5. Shell Programming and Scripting

sed ignoring case for search but respecting case for subtitute

Hi I want to make string substitution ignoring case for search but respecting case for subtitute. Ex changing all occurences of "original" in a file to "substitute": original becomes substitute Origninal becomes Substitute ORIGINAL becomes SUBSTITUTE I know this a little special but it's not... (1 Reply)
Discussion started by: kmchen
1 Replies

6. Shell Programming and Scripting

Remove duplicate lines

Hi, I have a huge file which is about 50GB. There are many lines. The file format likes 21 rs885550 0 9887804 C C T C C C C C C C 21 rs210498 0 9928860 0 0 C C 0 0 0 0 0 0 21 rs303304 0 9941889 A A A A A A A A A A 22 rs303304 0 9941890 0 A A A A A A A A A The question is that there are a few... (4 Replies)
Discussion started by: zhshqzyc
4 Replies

7. UNIX for Dummies Questions & Answers

Remove Duplicate Lines

Hi I need this output. Thanks. Input: TAZ YET FOO FOO VAK TAZ BAR Output: YET VAK BAR (10 Replies)
Discussion started by: tara123
10 Replies

8. Shell Programming and Scripting

Count duplicate lines ignoring certain columns

I have this structure: col1 col2 col3 col4 col5 27 xxx 38 aaa ttt 2 xxx 38 aaa yyy 1 xxx 38 aaa yyy I need to collapse duplicate lines ignoring column 1 and add values of duplicate lines (col1) so it will look like this: col1 col2 col3 col4 col5 27 xxx 38 aaa ttt ... (3 Replies)
Discussion started by: coppuca
3 Replies

9. Shell Programming and Scripting

Remove lines containing 2 or more duplicate strings

Within my text file i have several thousand lines of text with some lines containing duplicate strings/words. I would like to entirely remove those lines which contain the duplicate strings. Eg; One and a Two Unix.com is the Best This as a Line Line Example duplicate sentence with the word... (22 Replies)
Discussion started by: martinsmith
22 Replies

10. Shell Programming and Scripting

How to remove duplicate lines?

Hi All, I am storing the result in the variable result_text using the below code. result_text=$(printf "$result_text\t\n$name") The result_text is having the below text. Which is having duplicate lines. file and time for the interval 03:30 - 03:45 file and time for the interval 03:30 - 03:45 ... (4 Replies)
Discussion started by: nalu
4 Replies
nljust(1)						      General Commands Manual							 nljust(1)

NAME
nljust - justify lines, left or right, for printing SYNOPSIS
digits] seq] just] mode] order] margin] width] ck] [file ...] DESCRIPTION
formats for printing data written in languages with a right-to-left orientation. It is designed to be used with the and the commands (see pr(1) and lp(1)). reads the concatenation of input files (or standard input if none are given) and produces on standard output a right-to-left formatted ver- sion of its input. If appears as an input file name, reads standard input at that point. Use to delimit the end of options. formats input files for all languages that are read from right to left. For languages that have a left-to-right orientation, the command merely copies input files to standard output. Options recognizes the following options: Justify data for all languages, including those having a left-to-right text orientation. By default only right-to-left language data is justified. For all other languages, input files are directly copied to standard output. Select enhanced printer shapes for some Arabic characters. With this option, two-character combinations of laam and alif are replaced by a single character. Triggers ISO 8859-6 interpretation of the data. Processes digits for output as hindi, western, or both. digits can be or both. Use seq as the escape sequence to select the primary character set. This escape sequence is used by languages that have too many characters to be accommodated by ASCII in a single 256-character set. In these cases, the seq escape sequence can be used to select the non-ASCII character set. The escape character itself(0x1b) is not given on the command line. Hewlett-Packard escape sequences are used by default. If just is left justify print lines. If just is right-justify print lines starting from the (designated or default) print width column. The default is right justification. Replace leading spaces with alternative spaces. Some right-to-left character sets have a non-ASCII or alternative space. This option can be useful when filtering out- put (see pr(1)). With right justification, the option causes line numbers to be placed immediately to the right of the tab character. Without the option, right justification causes line numbers to be placed at the print-width column. By default, leading spaces are not replaced by alternative spaces. Indicate mode of any file to be formatted. Mode refers to the text orientation of the file when it was created. If mode is assume Latin mode. If mode is assume non-Latin mode. By default, mode information is obtained from the environment variable. Do not terminate lines containing printable characters with a new-line. By default, print lines are terminated by new-lines. Indicate data order of any file to be formatted. The text orientation of a file can affect the way its data is arranged. If order is assume keyboard order. If order is assume screen order. By default, order information is obtained from the environment variable. Truncate print lines that do not fit the designated or default line length. Print lines are folded (that is, wrapped to next line) by default. Expand input tabs to column positions k+1, 2*k+1, 3*k+1, etc. Tab characters in the input are expanded to the appropriate number of spaces. If k is 0 or is omitted, default tab settings at every eighth position is assumed. If cd (any non-digit character) is given, it is treated as the input tab character. The default for c is the tab character. always expands input tabs. This option provides a way to change the tab character and setting. If this option is specified, at least one of the parameters c or k must be given. Designate a number as the print margin. The print margin is the column where truncation or folding takes place. The print margin determines how many characters appear on a single line and can never exceed the print width. The print margin is relative to the justifica- tion. If the print margin is 80, folding or truncation occurs at column 80 starting from the right during a right jus- tification. Similarly, folding or truncation occurs at column 80 starting from the left during a left justification. By default, the print margin is set to column 80. Designates a number as the print width. The print width is the maximum number of columns in the print line. Print width determines the start of text during a right justification. The larger the print width, the further to the right the text will start. By default, an 80-column print width is used. EXTERNAL INFLUENCES
Environment Variables The environment variable determines the mode and order of the file. The syntax of is [mode][_order]. mode describes the mode of a file where represents Latin mode and represents non-Latin mode. Non-Latin mode is assumed for values other than and order describes the data order of a file where is keyboard and is screen. Keyboard order is assumed for values other than and Mode and order information in can be overridden from the command line. The environment variable determines the direction of a language (left-to-right or right-to-left) and whether context analysis of characters is necessary. The environment variable determines whether a language has alternative numbers. The environment variable determines the language in which messages are displayed. International Code Set Support Single-byte character code sets are supported. EXAMPLES
Right justify on a 132-column printer with a print margin at column 80 (the default): Right justify output of with line numbers on a 132-column printer with a print margin at column 132: WARNINGS
If with line numbers option) is piped to the separator character must be a tab(0x09). It is the user's responsibility to ensure that the environment variable accurately reflects the status of the file. Mode and justification must be consistent. Only non-Latin-mode files can be right justified in a meaningful way. Similarly, only Latin- mode files can be safely left justified. If mode and justification do not match, the results are undefined. If present, alternative numbers always have a left-to-right orientation. The command is HP proprietary, not portable to other vendors' systems, and will not be provided in future HP-UX releases. AUTHOR
was developed by HP. SEE ALSO
forder(1), lp(1), pr(1), strord(3C). nljust(1)
All times are GMT -4. The time now is 01:50 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy