Sponsored Content
Top Forums Shell Programming and Scripting Filter uniq field values (non-substring) Post 302900881 by vgersh99 on Friday 9th of May 2014 09:38:11 AM
Old 05-09-2014
no-no, this is all good - no ill feelings here!
I see your point - and yes, we solved this case earlier - I need to revert to my previous code:
Code:
awk '{
      for(i=1;i<=c;i++) {
        if (!(i in a)) continue
        if (index(a[i],$2) && index(b[i],$4))
           next

        if (index($2, a[i]) && index($4,b[i])) {
           delete a[i]
           delete b[i]
           delete all[i]
        }
      }

      a[++c]=$2
      b[c]=$4
      all[c]=$0
   }
END {
   for (i=1; i<=c;i++) if (i in all) print all[i]
}' myFile

It's a bit ugly. If you have an improvement idea, I'd be grateful to see it as well.
Also I cannot think of an easy non-convoluted way to avoid a full scan of the already cached entries for every new record/line (in order to improve the performance)...

Thanks for staying on this thread!

Last edited by vgersh99; 05-09-2014 at 10:54 AM..
These 2 Users Gave Thanks to vgersh99 For This Post:
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

Uniq using only the first field

Hi all, I have a file that contains a list of codes (shown below). I want to 'uniq' the file using only the first field. Anyone know an easy way of doing it? Cheers, Dave ##### Input File ##### 1xr1 1xws 1yxt 1yxu 1yxv 1yxx 2o3p 2o63 2o64 2o65 1xr1 1xws 1yxt 1yxv 1yxx 2o3p 2o63 2o64... (8 Replies)
Discussion started by: Digby
8 Replies

2. UNIX for Dummies Questions & Answers

How to uniq third field in a file

Hi ; I have a question regarding the uniq command in unix How do I uniq 3rd field in a file ? original file : zoom coord 39 18652 39 18652 zoom coord 39 18653 39 18653 zoom coord 39 18818 39 18818 zoom coord 39 18840 39 18840 zoom coord 41 15096 41 15096 zoom... (1 Reply)
Discussion started by: babycakes
1 Replies

3. Shell Programming and Scripting

How to use uniq on a certain field?

How can I use uniq on a certain field or what else could I use? If I want to use uniq on the second field and the output would remove one of the lines with a 5. bob 5 hand jane 3 leg jon 4 head chris 5 lungs (1 Reply)
Discussion started by: Bandit390
1 Replies

4. Shell Programming and Scripting

filter the uniq record problem

Anyone can help for filter the uniq record for below example? Thank you very much Input file 20090503011111|test|abc 20090503011112|tet1|abc|def 20090503011112|test1|bcd|def 20090503011131|abc|abc 20090503011131|bbc|bcd 20090503011152|bcd|abc 20090503011151|abc|abc... (8 Replies)
Discussion started by: bleach8578
8 Replies

5. Shell Programming and Scripting

Uniq based on first field

Hi New to unix. I want to display only the unrepeated lines from a file using first field. Ex: 1234 uname1 status1 1235 uname2 status2 1234 uname3 status3 1236 uname5 status5 I used sort filename | uniq -u output: 1234 uname1 status1 1235 uname2 status2 1234 uname3 status3 1236... (10 Replies)
Discussion started by: venummca
10 Replies

6. Shell Programming and Scripting

Sort field and uniq

I have a flatfile A.txt 2012/12/04 14:06:07 |trees|Boards 2, 3|denver|mekong|mekong12 2012/12/04 17:07:22 |trees|Boards 2, 3|denver|mekong|mekong12 2012/12/04 17:13:27 |trees|Boards 2, 3|denver|mekong|mekong12 2012/12/04 14:07:39 |rain|Boards 1|tampa|merced|merced11 How do i sort and get... (3 Replies)
Discussion started by: sabercats
3 Replies

7. Shell Programming and Scripting

Printing uniq first field with the the highest second field

Hi All, I am searching for a script which will produce an output file with the uniq first field with the second field having highest value among all the duplicates.. The output file will produce only the uniqs which are duplicate 3 times.. Input file X 9 B 5 A 1 Z 9 T 4 C 9 A 4... (13 Replies)
Discussion started by: ailnilanjan
13 Replies

8. Shell Programming and Scripting

Grok filter to extract substring from path and add to host field in logstash

Hii, I am reading data from files by defining path as *.log etc, Files names are like app1a_test2_heep.log , cdc2a_test3_heep.log etc How to configure logstash so that the part of string that is string before underscore (app1a, cdc2a..) should be grepped and added to host field and... (7 Replies)
Discussion started by: Ravi Kishore
7 Replies

9. Shell Programming and Scripting

HELP - uniq values per column

Hi All, I am trying to output uniq values per column. see file below. can you please assist? Thank you in advance. cat names joe allen ibm joe smith ibm joe allen google joe smith google rachel allen google desired output is: joe allen google rachel smith ibm (5 Replies)
Discussion started by: Apollo
5 Replies

10. Shell Programming and Scripting

awk to update field using matching value in file1 and substring in field in file2

In the awk below I am trying to set/update the value of $14 in file2 in bold, using the matching NM_ in $12 or $9 in file2 with the NM_ in $2 of file1. The lengths of $9 and $12 can be variable but what is consistent is the start pattern will always be NM_ and the end pattern is always ;... (2 Replies)
Discussion started by: cmccabe
2 Replies
GIT-REVERT(1)							    Git Manual							     GIT-REVERT(1)

NAME
git-revert - Revert some existing commits SYNOPSIS
git revert [--[no-]edit] [-n] [-m parent-number] [-s] [-S[<keyid>]] <commit>... git revert --continue git revert --quit git revert --abort DESCRIPTION
Given one or more existing commits, revert the changes that the related patches introduce, and record some new commits that record them. This requires your working tree to be clean (no modifications from the HEAD commit). Note: git revert is used to record some new commits to reverse the effect of some earlier commits (often only a faulty one). If you want to throw away all uncommitted changes in your working directory, you should see git-reset(1), particularly the --hard option. If you want to extract specific files as they were in another commit, you should see git-checkout(1), specifically the git checkout <commit> -- <filename> syntax. Take care with these alternatives as both will discard uncommitted changes in your working directory. OPTIONS
<commit>... Commits to revert. For a more complete list of ways to spell commit names, see gitrevisions(7). Sets of commits can also be given but no traversal is done by default, see git-rev-list(1) and its --no-walk option. -e, --edit With this option, git revert will let you edit the commit message prior to committing the revert. This is the default if you run the command from a terminal. -m parent-number, --mainline parent-number Usually you cannot revert a merge because you do not know which side of the merge should be considered the mainline. This option specifies the parent number (starting from 1) of the mainline and allows revert to reverse the change relative to the specified parent. Reverting a merge commit declares that you will never want the tree changes brought in by the merge. As a result, later merges will only bring in tree changes introduced by commits that are not ancestors of the previously reverted merge. This may or may not be what you want. See the revert-a-faulty-merge How-To[1] for more details. --no-edit With this option, git revert will not start the commit message editor. -n, --no-commit Usually the command automatically creates some commits with commit log messages stating which commits were reverted. This flag applies the changes necessary to revert the named commits to your working tree and the index, but does not make the commits. In addition, when this option is used, your index does not have to match the HEAD commit. The revert is done against the beginning state of your index. This is useful when reverting more than one commits' effect to your index in a row. -S[<keyid>], --gpg-sign[=<keyid>] GPG-sign commits. The keyid argument is optional and defaults to the committer identity; if specified, it must be stuck to the option without a space. -s, --signoff Add Signed-off-by line at the end of the commit message. See the signoff option in git-commit(1) for more information. --strategy=<strategy> Use the given merge strategy. Should only be used once. See the MERGE STRATEGIES section in git-merge(1) for details. -X<option>, --strategy-option=<option> Pass the merge strategy-specific option through to the merge strategy. See git-merge(1) for details. SEQUENCER SUBCOMMANDS
--continue Continue the operation in progress using the information in .git/sequencer. Can be used to continue after resolving conflicts in a failed cherry-pick or revert. --quit Forget about the current operation in progress. Can be used to clear the sequencer state after a failed cherry-pick or revert. --abort Cancel the operation and return to the pre-sequence state. EXAMPLES
git revert HEAD~3 Revert the changes specified by the fourth last commit in HEAD and create a new commit with the reverted changes. git revert -n master~5..master~2 Revert the changes done by commits from the fifth last commit in master (included) to the third last commit in master (included), but do not create any commit with the reverted changes. The revert only modifies the working tree and the index. SEE ALSO
git-cherry-pick(1) GIT
Part of the git(1) suite NOTES
1. revert-a-faulty-merge How-To file:///usr/share/doc/git/html/howto/revert-a-faulty-merge.html Git 2.17.1 10/05/2018 GIT-REVERT(1)
All times are GMT -4. The time now is 01:56 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy