Sponsored Content
Top Forums Shell Programming and Scripting Find all lines in file such that each word on that line appears in at least n lines of the file Post 302999219 by RudiC on Thursday 15th of June 2017 10:53:02 AM
Old 06-15-2017
I'm certain Don Cragun will accept the apologies. The forum maintainers' attitude is less to not to become useless - people in here REALLY like to help with also minor problems - but to keep up the quality of IT education. If a student fills in the homework form including institution, course and professor, s/he will be helped to develop in the right direction and find a solution of his/her own; c.f. https://www.unix.com/homework-and-coursework-questions/. By the way, vague comments on a person's company like "chemical" or "administration" would have sufficed, or even you telling us you're a hobbyist.

Back to your problem. Outputting the entire line that satisfies a condition means either keep ALL lines in memory (demanding for BIG files) or run through the input file twice - once for counting, once for printing. This is the approach in here:
Code:
awk 'NR == FNR {CNT[$1]++; CNT[$3]++;CNT[$5]++; CNT[$7]++; next} CNT[$1] > 1 && CNT[$3] > 1 && CNT[$5] > 1 && CNT[$7] > 1 ' file file
5^5 + 18^2 = 15^3 + 74^1    (3125, 324, 3375, 74)
5^5 + 32^2 = 8^4 + 53^1    (3125, 1024, 4096, 53)
5^5 + 60^1 = 14^3 + 21^2    (3125, 60, 2744, 441)

For increasing the count limit, set all the 1 s to 3 for the four comparisons in the second part.
And, yes, you're right: awk is a very powerful tool for text file analyses...
This User Gave Thanks to RudiC For This Post:
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

shellscript to find a line in between a particular set of lines of a text file

i have a file a.txt and following is only one portion. I want to search <branch value="/dev36/AREA/" include="yes"></branch> present in between <template_file name="Approve External" path="core/approve/bin" and </template_file> where the no of lines containing "<branch value= " is increasing ... (2 Replies)
Discussion started by: millan
2 Replies

2. Shell Programming and Scripting

Find 5 lines and replace with 18 line in sql file where it contains multiple blocks.

My sql file xyz_abc.sql in this file there are multiple sql block in this block I need to find the following block rem Subset Rows (&&tempName.*) CREATE VIEW &&tempName.* AS SELECT * FROM &&tempName.* WHERE f is not null and replace with following code rem Subset Rows... (9 Replies)
Discussion started by: Zaheer.mic
9 Replies

3. Shell Programming and Scripting

Unix help to find blank lines in a file and print numbers on that line

Hi, I would like to know how to solve one of my problems using expert unix commands. I have a file with occasional blank lines; for example; dertu frthu fghtu frtty frtgy frgtui frgtu ghrye frhutp frjuf I need to edit the file so that the file looks like this; (10 Replies)
Discussion started by: Lucky Ali
10 Replies

4. UNIX for Dummies Questions & Answers

find uniq lines in file, using the first field of line

Hello all, new to unix and have just found the forum. I think I will be here quite often, and hope that in time i will be able to provide soem help, role on not being a newbie anymore :) I have a question which iI am hoping someone could help me with. If i have a file with lines in in thus... (8 Replies)
Discussion started by: grom
8 Replies

5. Shell Programming and Scripting

print lines from a file containing key word

i have a file containing over 1 million records,and i want to print about 300,000 line containing a some specific words. file has content. eg 1,rrt,234 3,fgt,678 4,crf,456 5,cde,drt 6,cfg,123 and i want to print the line with the word fgt,crf this is just an example,my file is so... (2 Replies)
Discussion started by: tomjones
2 Replies

6. UNIX for Dummies Questions & Answers

how to find a word in a file that appears next to a given keyword

Hi Experts, I have a file which contains some text. i need to print the word next to a given keyword. Please help. Ex: test.txt ===================== NEXT HOST ===================== AEADBAS001 access-list 1 permit xxxxxxxxxxxxxx ip access-list extended BLA_Outgoing_Filter... (6 Replies)
Discussion started by: mwrg
6 Replies

7. Shell Programming and Scripting

Get last lines of file after last line with word TEST

i need to get least lines of file after last word TEST in file, and send that lines to mail example of file structure: TEST 10.10.2010 jdfjdnjfndjfndnfkdk djfjdnfjkdjkfnjkdfk jdfjdjfnjdjnfjkdnfjk TEST 11.10.2010 jdjfnjdnfdkdfjdfjdnk jdnfjdnjkfndnfjdnfjk fjdnfjkndnfdfnjdnfjk TEST... (6 Replies)
Discussion started by: waso
6 Replies

8. Shell Programming and Scripting

Read all lines after a string appears in the file.

Hi All, I want to read all lines after a perticular string {SET UP VALUES}apprears in the file. SET UP values contains direcory, number of days and file type. Step1: Read all lines below SET UP VALUES string. Step2: If set up values are not present in each record then read from default... (4 Replies)
Discussion started by: Nagaraja Akkiva
4 Replies

9. UNIX for Advanced & Expert Users

How to find a string in a line in UNIX file and delete that line and previous 3 lines ?

Hi , i have a file with data as below.This is same file. But actual file contains to many rows. i want to search for a string "Field 039 00" and delete that line and previous 3 lines in that file.. Can some body suggested me how can i do using either sed or awk command ? Field 004... (7 Replies)
Discussion started by: vadlamudy
7 Replies

10. UNIX for Beginners Questions & Answers

Search for word in huge logfile and need to continue to print few lines from that line til find date

Guys i need an idea for one logic..in shell scripting am struggling with a logic...So the thing is... i need to search for a word in a huge log file and i need to continue to print few more lines from that line and the consecutive line has to end when it finds the line with date..because i know... (1 Reply)
Discussion started by: Prathi
1 Replies
Biblio::Isis(3pm)					User Contributed Perl Documentation					 Biblio::Isis(3pm)

NAME
Biblio::Isis - Read CDS/ISIS, WinISIS and IsisMarc database SYNOPSIS
use Biblio::Isis; my $isis = new Biblio::Isis( isisdb => './cds/cds', ); for(my $mfn = 1; $mfn <= $isis->count; $mfn++) { print $isis->to_ascii($mfn)," "; } DESCRIPTION
This module will read ISIS databases created by DOS CDS/ISIS, WinIsis or IsisMarc. It can be used as perl-only alternative to OpenIsis mod- ule which seems to depriciate it's old "XS" bindings for perl. It can create hash values from data in ISIS database (using "to_hash"), ASCII dump (using "to_ascii") or just hash with field names and packed values (like "^asomething^belse"). Unique feature of this module is ability to "include_deleted" records. It will also skip zero sized fields (OpenIsis has a bug in XS bind- ings, so fields which are zero sized will be filled with random junk from memory). It also has support for identifiers (only if ISIS database is created by IsisMarc), see "to_hash". This module will always be slower than OpenIsis module which use C library. However, since it's written in perl, it's platform independent (so you don't need C compiler), and can be easily modified. I hope that it creates data structures which are easier to use than ones cre- ated by OpenIsis, so reduced time in other parts of the code should compensate for slower performance of this module (speed of reading ISIS database is rarely an issue). METHODS
new Open ISIS database my $isis = new Biblio::Isis( isisdb => './cds/cds', read_fdt => 1, include_deleted => 1, hash_filter => sub { my ($v,$field_number) = @_; $v =~ s#foo#bar#g; }, debug => 1, join_subfields_with => ' ; ', ); Options are described below: isisdb This is full or relative path to ISIS database files which include common prefix of ".MST", and ".XRF" and optionally ".FDT" (if using "read_fdt" option) files. In this example it uses "./cds/cds.MST" and related files. read_fdt Boolean flag to specify if field definition table should be read. It's off by default. include_deleted Don't skip logically deleted records in ISIS. hash_filter Filter code ref which will be used before data is converted to hash. It will receive two arguments, whole line from current field (in $_[0]) and field number (in $_[1]). debug Dump a lot of debugging output even at level 1. For even more increase level. join_subfields_with Define delimiter which will be used to join repeatable subfields. This option is included to support lagacy application written against version older than 0.21 of this module. By default, it disabled. See "to_hash". ignore_empty_subfields Remove all empty subfields while reading from ISIS file. count Return number of records in database print $isis->count; fetch Read record with selected MFN my $rec = $isis->fetch(55); Returns hash with keys which are field names and values are unpacked values for that field like this: $rec = { '210' => [ '^aNew York^cNew York University press^dcop. 1988' ], '990' => [ '2140', '88', 'HAY' ], }; mfn Returns current MFN position my $mfn = $isis->mfn; to_ascii Returns ASCII output of record with specified MFN print $isis->to_ascii(42); This outputs something like this: 210 ^aNew York^cNew York University press^dcop. 1988 990 2140 990 88 990 HAY If "read_fdt" is specified when calling "new" it will display field names from ".FDT" file instead of numeric tags. to_hash Read record with specified MFN and convert it to hash my $hash = $isis->to_hash($mfn); It has ability to convert characters (using "hash_filter") from ISIS database before creating structures enabling character re-mapping or quick fix-up of data. This function returns hash which is like this: $hash = { '210' => [ { 'c' => 'New York University press', 'a' => 'New York', 'd' => 'cop. 1988' } ], '990' => [ '2140', '88', 'HAY' ], }; You can later use that hash to produce any output from ISIS data. If database is created using IsisMarc, it will also have to special fields which will be used for identifiers, "i1" and "i2" like this: '200' => [ { 'i1' => '1', 'i2' => ' ' 'a' => 'Goa', 'f' => 'Valdo D'Arienzo', 'e' => 'tipografie e tipografi nel XVI secolo', } ], In case there are repeatable subfields in record, this will create following structure: '900' => [ { 'a' => [ 'foo', 'bar', 'baz' ], }] Or in more complex example of 902 ^aa1^aa2^aa3^bb1^aa4^bb2^cc1^aa5 it will create 902 => [ { a => ["a1", "a2", "a3", "a4", "a5"], b => ["b1", "b2"], c => "c1" }, ], This behaviour can be changed using "join_subfields_with" option to "new", in which case "to_hash" will always create single value for each subfield. This will change result to: This method will also create additional field 000 with MFN. There is also more elaborative way to call "to_hash" like this: my $hash = $isis->to_hash({ mfn => 42, include_subfields => 1, }); Each option controll creation of hash: mfn Specify MFN number of record include_subfields This option will create additional key in hash called "subfields" which will have original record subfield order and index to that sub- field like this: 902 => [ { a => ["a1", "a2", "a3", "a4", "a5"], b => ["b1", "b2"], c => "c1", subfields => ["a", 0, "a", 1, "a", 2, "b", 0, "a", 3, "b", 1, "c", 0, "a", 4], } ], join_subfields_with Define delimiter which will be used to join repeatable subfields. You can specify option here instead in "new" if you want to have per- record control. hash_filter You can override "hash_filter" defined in "new" using this option. tag_name Return name of selected tag print $isis->tag_name('200'); read_cnt Read content of ".CNT" file and return hash containing it. print Dumper($isis->read_cnt); This function is not used by module (".CNT" files are not required for this module to work), but it can be useful to examine your index (while debugging for example). unpack_cnt Unpack one of two 26 bytes fixed length record in ".CNT" file. Here is definition of record: off key description size 0: IDTYPE BTree type s 2: ORDN Nodes Order s 4: ORDF Leafs Order s 6: N Number of Memory buffers for nodes s 8: K Number of buffers for first level index s 10: LIV Current number of Index Levels s 12: POSRX Pointer to Root Record in N0x l 16: NMAXPOS Next Available position in N0x l 20: FMAXPOS Next available position in L0x l 24: ABNORMAL Formal BTree normality indicator s length: 26 bytes This will fill $self object under "cnt" with hash. It's used by "read_cnt". BUGS
Some parts of CDS/ISIS documentation are not detailed enough to exmplain some variations in input databases which has been tested with this module. When I was in doubt, I assumed that OpenIsis's implementation was right (except for obvious bugs). However, every effort has been made to test this module with as much databases (and programs that create them) as possible. I would be very greatful for success or failure reports about usage of this module with databases from programs other than WinIsis and IsisMarc. I had tested this against ouput of one "isis.dll"-based application, but I don't know any details about it's version. VERSIONS
As this is young module, new features are added in subsequent version. It's a good idea to specify version when using this module like this: use Biblio::Isis 0.23 Below is list of changes in specific version of module (so you can target older versions if you really have to): 0.24 Added "ignore_empty_subfields" 0.23 Added "hash_filter" to "to_hash" Fixed bug with documented "join_subfields_with" in "new" which wasn't implemented 0.22 Added field number when calling "hash_filter" 0.21 Added "join_subfields_with" to "new" and "to_hash". Added "include_subfields" to "to_hash". 0.20 Added "$isis->mfn", support for repeatable subfields and "$isis->to_hash({ mfn => 42, ... })" calling convention AUTHOR
Dobrica Pavlinusic CPAN ID: DPAVLIN dpavlin@rot13.org http://www.rot13.org/~dpavlin/ This module is based heavily on code from "LIBISIS.PHP" library to read ISIS files V0.1.1 written in php and (c) 2000 Franck Martin <franck@sopac.org> and released under LGPL. COPYRIGHT
This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself. The full text of the license can be found in the LICENSE file included with this module. SEE ALSO
Biblio::Isis::Manual for CDS/ISIS manual appendix F, G and H which describe file format OpenIsis web site <http://www.openisis.org> perl4lib site <http://perl4lib.perl.org> perl v5.8.8 2007-05-18 Biblio::Isis(3pm)
All times are GMT -4. The time now is 03:43 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy