Sponsored Content
Top Forums Shell Programming and Scripting Delete lines in file containing duplicate strings, keeping longer strings Post 302556123 by raidzero on Friday 16th of September 2011 04:39:40 PM
Old 09-16-2011
duplicate lines are not always consecutive, and the item names can vary Smilie


string1 may be defined at line 18, and then string1 might be defined again at like 818...

---------- Post updated at 04:39 PM ---------- Previous update was at 04:36 PM ----------

but you know what? corona, your solution seems to work Smilie

my awk-fu is weak
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Grep and delete lines except the lines with strings

Hi I am writing a script which should read a file and search for certain strings 'approved' or 'removed' and retain only those lines that contain the above strings. Ex: file name 'test' test: approved package waiting for approval package disapproved package removed package approved... (14 Replies)
Discussion started by: vj8436
14 Replies

2. UNIX for Dummies Questions & Answers

Delete lines with duplicate strings based on date

Hey all, a relative bash/script newbie trying solve a problem. I've got a text file with lots of lines that I've been able to clean up and format with awk/sed/cut, but now I'd like to remove the lines with duplicate usernames based on time stamp. Here's what the data looks like 2007-11-03... (3 Replies)
Discussion started by: mattv
3 Replies

3. UNIX for Dummies Questions & Answers

Delete strings in file1 based on the list of strings in file2

Hello guys, should be a very easy questn for you: I need to delete strings in file1 based on the list of strings in file2. like file2: word1_word2_ word3_word5_ word3_word4_ word6_word7_ file1: word1_word2_otherwords..,word3_word5_others... (7 Replies)
Discussion started by: roussine
7 Replies

4. Shell Programming and Scripting

Delete lines starting with these strings

Platform : RHEL 5.8 I have text file called myapplication.log . In this file, I have around 800 lines which start with the followng three strings PWRBRKER-3493 PWRBRKER-7834 SCHEDULER-ERROR How can I delete these lines in one go ? (13 Replies)
Discussion started by: omega3
13 Replies

5. Shell Programming and Scripting

Getting lines between two strings with duplicate set of data

if I have the following lines in a file app.log some lines here <AAAA> abc <id>123456789</id> ddd </AAAA>some lines here too <BBBB> abc <id>123456789</id> ddd </BBBB>some lines here too <AAAA> xyz <id>987654321</id> ssss </AAAA>some lines here again... How do I get the... (5 Replies)
Discussion started by: nariwithu
5 Replies

6. Shell Programming and Scripting

Delete duplicate strings in a line

Hi, i need help to remove duplicates in my file. The problem is i need to delete one duplicate for each line only. the input file as follows and it is not tab delimited:- The output need to remove 2nd word (in red) that duplicate with 1st word (in blue). Other duplicates should remained... (12 Replies)
Discussion started by: redse171
12 Replies

7. UNIX for Dummies Questions & Answers

Replace some strings keeping others

I want to replace strings in test2 according to test1 table. In doing so, I`m losing records that I dont need to replace, please suggest modifications. what i have $ cat > test1 a b c d   $ cat > test2 a a a d d   what i tried $ awk ' BEGIN {FS=OFS=" "} FNR==NR{a=$2;next}... (2 Replies)
Discussion started by: senhia83
2 Replies

8. Shell Programming and Scripting

Remove lines containing 2 or more duplicate strings

Within my text file i have several thousand lines of text with some lines containing duplicate strings/words. I would like to entirely remove those lines which contain the duplicate strings. Eg; One and a Two Unix.com is the Best This as a Line Line Example duplicate sentence with the word... (22 Replies)
Discussion started by: martinsmith
22 Replies

9. UNIX for Beginners Questions & Answers

How to pass strings from a list of strings from another file and create multiple files?

Hello Everyone , Iam a newbie to shell programming and iam reaching out if anyone can help in this :- I have two files 1) Insert.txt 2) partition_list.txt insert.txt looks like this :- insert into emp1 partition (partition_name) (a1, b2, c4, s6, d8) select a1, b2, c4, (2 Replies)
Discussion started by: nubie2linux
2 Replies

10. UNIX for Beginners Questions & Answers

Use strings from nth field from one file to match strings in entire line in another file, awk

I cannot seem to get what should be a simple awk one-liner to work correctly and cannot figure out why. I would like to use patterns from a specific field in one file as regex to search for matching strings in the entire line ($0) of another file. I would like to output the lines of File2 which... (1 Reply)
Discussion started by: jvoot
1 Replies
tr(1)							      General Commands Manual							     tr(1)

NAME
tr - translate characters SYNOPSIS
string1 string2 string1 string1 string1 string1 DESCRIPTION
copies the standard input to the standard output with substitution or deletion of selected characters. Input characters from string1 are replaced with the corresponding characters in string2. If necessary, string1 and string2 can be quoted to avoid pattern matching by the shell. recognizes the following command line options: Translates on a byte-by-byte basis. When this flag is specified does not support extended characters. Complements the set of characters in string1, which is the set of all characters in the current character set, as defined by the current setting of except for those actually specified in the string1 argument. These characters are placed in the array in ascending collation sequence, as defined by the current setting of Deletes all occurrences of input characters or collating elements found in the array specified in string1. If and are both specified, all characters except those specified by string1 are deleted. The contents of string2 are ignored, unless is also specified. Note, however, that the same string cannot be used for both the and the flags; when both flags are specified, both string1 (used for deletion) and string2 (used for squeezing) are required. If is not specified, each input character or collating element found in the array specified by string1 is replaced by the character or collating element in the same relative position specified by string2. Replaces any character specified in string1 that occurs as a string of two or more repeating characters as a single instance of the character in string2. If the string2 contains a character class, the argument's array contains all of the characters in that character class. For example: In a case conversion, however, the string2 array contains only those characters defined as the second characters in each of the or character pairs, as appropriate. For example: The following abbreviation conventions can be used to introduce ranges of characters, repeated characters or single-character collating elements into the strings: c1-c2 or Stands for the range of collating elements c1 through c2, inclusive, as defined by the current setting of the locale category. Stands for all the characters belonging to the defined character class, as defined by the current setting of locale category. The following character class names will be accepted when spec- ified in string1: or Character classes are expanded in collation order. When the and flags are specified together, any of the character class names are accepted in string2; otherwise, only character class names or are accepted in string2 and then only if the corresponding character class and respectively) is specified in the same relative position in string1. Such a specification is interpreted as a request for case conversion. When appears in string1 and appears in string2, the arrays contain the characters from the mapping in the category of the current locale. When appears in string1 and appears in string2, the arrays contain the characters from the map- ping in the category of the current locale. Stands for all the characters or collating elements belonging to the same equivalence class as c, as defined by the current setting of locale category. An equivalence class expression is allowed only in string1, or in string2 when it is being used by the combined and options. Stands for n repetitions of a. If the first digit of n is n is considered octal; otherwise, n is treated as a decimal value. A zero or missing n is interpreted as large enough to extend string2-based sequence to the length of the string1-based sequence. The escape character can be used as in the shell to remove special meaning from any character in a string. In addition, followed by 1, 2, or 3 octal digits represents the character whose ASCII code is given by those digits. An ASCII NULL character in string1 or string2 can be represented only as an escaped character; i.e. as but is treated like other characters and translated correctly if so specified. NULL characters in the input are not stripped out unless the option is given. EXTERNAL INFLUENCES
Environment Variables provides a default value for the internationalization variables that are unset or null. If is unset or null, the default value of "C" (see lang(5)) is used. If any of the internationalization variables contains an invalid setting, will behave as if all internationalization variables are set to "C". See environ(5). If set to a non-empty string value, overrides the values of all the other internationalization variables. determines the interpretation of text as single and/or multi-byte characters, the classification of characters as printable, and the char- acters matched by character class expressions in regular expressions. determines the locale that should be used to affect the format and contents of diagnostic messages written to standard error and informa- tive messages written to standard output. determines the location of message catalogues for the processing of RETURN VALUE
exits with one of the following values: All input was processed successfully. An error occurred. EXAMPLES
For the ASCII character set and default collation sequence, create a list of all the words in file1, one per line in file2, where a word is taken to be a maximal string of alphabetics. Quote the strings to protect the special characters from interpretation by the shell (012 is the ASCII code for a new-line (line feed) character): Same as above, but for all character sets and collation sequences: Translate all lower case characters in file1 to upper case and write the result to standard output. Use an equivalence class to identify accented variants of the base character in file1, strip them of diacritical marks and write the result to file2: Translate each digit in file1 to a (number sign), and write the result to file2. The (asterisk) tells to repeat the (number sign) enough times to make the second string as long as the first one. AUTHOR
was developed by OSF and HP. SEE ALSO
ed(1), sh(1), ascii(5), environ(5), lang(5), regexp(5). STANDARDS CONFORMANCE
tr(1)
All times are GMT -4. The time now is 10:49 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy