Sponsored Content
Top Forums Shell Programming and Scripting Grep: check if a string comes up twice Post 302999197 by rbatte1 on Thursday 15th of June 2017 07:10:50 AM
Old 06-15-2017
Beware that for large input files, you are reading the entire list.txt for every entry in query.txt

There may be better ways to approach this, but it depends how much you need to know. If you just need to know that ABC (or whatever) has been repeated and you don't care what the first string part is, then you might be better with this:-
Code:
cut -f2 -d"$(printf "\t")" list.txt | sort | uniq -c | grep -f query.txt | grep -Ev "^      1 "

It is a few pipes, but will read list.txt once.

If you then need to go back and get the leading string information, we can probably work on that.


So,
  • Does that help?
  • Does it confuse?
  • Is it irrelevant because the files are trivial?
  • Does anyone else have a better way? I'm open to suggestions too!


I hope that this helps,
Robin
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

read string, check string length and cut

Hello All, Plz help me with: I have a csv file with data separated by ',' and optionally enclosed by "". I want to check each of these values to see if they exceed the specified string length, and if they do I want to cut just that value to the max length allowed and keep the csv format as it... (9 Replies)
Discussion started by: ozzy80
9 Replies

2. UNIX for Dummies Questions & Answers

| help | unix | grep - Can I use grep to return a string with exactly n matches?

Hello, I looking to use grep to return a string with exactly n matches. I'm building off this: ls -aLl /bin | grep '^.\{9\}x' | tr -s ' ' -rwxr-xr-x 1 root root 632816 Nov 25 2008 vi -rwxr-xr-x 1 root root 632816 Nov 25 2008 view -rwxr-xr-x 1 root root 16008 May 25 2008... (7 Replies)
Discussion started by: MykC
7 Replies

3. Shell Programming and Scripting

Grep for a string and then grep using a string from that result

Hello, Thanks in advance for the query. There is a log file abcd.log which has multible line like this. "hello1" , "hello2", "hello3" , "hello4" , "hello5" I want to grep for the lines which has "hello4" & "hello5" and use "hello2" to grep the same log file again. All these should... (8 Replies)
Discussion started by: kzenthil
8 Replies

4. Shell Programming and Scripting

grep on string and printing line after until another string has been found

Hello Everyone, I just started scripting this week. I have no background in programming or scripting. I'm working on a script to grep for a variable in a log file Heres what the log file looks like. The x's are all random clutter xxxxxxxxxxxxxxxxxxxxx START: xxxxxxxxxxxx... (7 Replies)
Discussion started by: rxc23816
7 Replies

5. Shell Programming and Scripting

Grep a string and write a value to next line of found string

Hi, I have two variables x and y. i need to find a particular string in a file, a workflow name and then insert the values of x and y into the next lines of the workflow name. basically it is like as below wf_xxxxxx $$a= $$b= $$c= figo $$d=bentley i need to grep the 'wf_xxxx' and then... (6 Replies)
Discussion started by: angel12345
6 Replies

6. Shell Programming and Scripting

Grep a string from input file and delete next three lines including the line contains string in xml

Hi, 1_strings file contains $ cat 1_strings /home/$USER/Src /home/Valid /home/Review$ cat myxml <projected value="some string" path="/home/$USER/Src"> <input 1/> <estimate value/> <somestring/> </projected> <few more lines > <projected value="some string" path="/home/$USER/check">... (4 Replies)
Discussion started by: greet_sed
4 Replies

7. Shell Programming and Scripting

grep exact string from files and write to filename when string present in file

I am attempting to grep an exact string from a series of files within a directory and append that output to the filename when it is present in the file. I've been after this all day with no luck. Thanks for your help in advance :wall:. (4 Replies)
Discussion started by: JC_1
4 Replies

8. Shell Programming and Scripting

Check file for string existence before appending it with string

I want to append file with a string but before doing that i want to check if this string already exist in that file.I tried with grep on Solaris 10 but unsuccessful.Man pages from grep seems to suggest if the string is found command status will be 0 and if not 1.But i am not finding it.May be i... (2 Replies)
Discussion started by: sahil_shine
2 Replies

9. Shell Programming and Scripting

Grep string in files and list file names that contain the string

Hi, I have a list of zipped files. I want to grep for a string in all files and get a list of file names that contain the string. But without unzipping them before that, more like using something like gzcat. My OS is: SunOS test 5.10 Generic_142900-13 sun4u sparc SUNW,SPARC-Enterprise (8 Replies)
Discussion started by: apenkov
8 Replies

10. Shell Programming and Scripting

Grep a string and count following lines starting with another string

I have a large dataset with following structure; C 0001 Carbon D SAR001 methane D SAR002 ethane D SAR003 propane D SAR004 butane D SAR005 pentane C 0002 Hydrogen C 0003 Nitrogen C 0004 Oxygen D SAR011 ozone D SAR012 super oxide C 0005 Sulphur D SAR013... (3 Replies)
Discussion started by: Syeda Sumayya
3 Replies
POE::Filter::Grep(3pm)					User Contributed Perl Documentation				    POE::Filter::Grep(3pm)

NAME
POE::Filter::Grep - select or remove items based on simple rules SYNOPSIS
#!perl use POE qw( Wheel::FollowTail Filter::Line Filter::Grep Filter::Stackable ); POE::Session->create( inline_states => { _start => sub { my $parse_input_as_lines = POE::Filter::Line->new(); my $select_sudo_log_lines = POE::Filter::Grep->new( Put => sub { 1 }, Get => sub { my $input = shift; return $input =~ /sudo[d+]/i; }, ); my $filter_stack = POE::Filter::Stackable->new( Filters => [ $parse_input_as_lines, # first on get, last on put $select_sudo_log_lines, # first on put, last on get ] ); $_[HEAP]{tailor} = POE::Wheel::FollowTail->new( Filename => "/var/log/system.log", InputEvent => "got_log_line", Filter => $filter_stack, ); }, got_log_line => sub { print "Log: $_[ARG0] "; } } ); POE::Kernel->run(); exit; DESCRIPTION
POE::Filter::Grep selects or removes items based on simple tests. It may be used to filter input, output, or both. This filter is named and modeled after Perl's built-in grep() function. POE::Filter::Grep is designed to be combined with other filters through POE::Filter::Stackable. In the "SYNOPSIS" example, a filter stack is created to parse logs as lines and remove all entries that don't pertain to a sudo process. (Or if your glass is half full, the stack only selects entries that DO mention sudo.) PUBLIC FILTER METHODS
In addition to the usual POE::Filter methods, POE::Filter::Grep also supports the following. new new() constructs a new POE::Filter::Grep object. It must either be called with a single Code parameter, or both a Put and a Get parameter. The values for Code, Put, and Get are code references that, when invoked, return true to select an item or false to reject it. A Code function will be used for both input and output, while Get and Put functions allow input and output to be filtered in different ways. The item in question will be passed as the function's sole parameter. sub reject_bidoofs { my $pokemon = shift; return 1 if $pokemon ne "bidoof"; return; } my $gotta_catch_nearly_all = POE::Filter::Grep->new( Code => &reject_bidoofs, ); Enforce read-only behavior: my $read_only = POE::Filter::Grep->new( Get => sub { 1 }, Put => sub { 0 }, ); modify modify() changes a POE::Filter::Grep object's behavior at run-time. It accepts the same parameters as new(), and it replaces the existing tests with new ones. # Don't give away our Dialgas. $gotta_catch_nearly_all->modify( Get => sub { 1 }, Put => sub { return shift() ne "dialga" }, ); SEE ALSO
POE::Filter for more information about filters in general. POE::Filter::Stackable for more details on stacking filters. BUGS
None known. AUTHORS &; COPYRIGHTS The Grep filter was contributed by Dieter Pearcey. Documentation is provided by Rocco Caputo. Please see the POE manpage for more information about authors and contributors. perl v5.14.2 2012-05-15 POE::Filter::Grep(3pm)
All times are GMT -4. The time now is 04:09 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy