Extract lines that have entries in VI file

 
Thread Tools Search this Thread
Top Forums UNIX for Beginners Questions & Answers Extract lines that have entries in VI file
# 8  
Old 12-20-2016
Quote:
Originally Posted by RudiC
man grep:

Does /usr/xpg4/bin/grep offer the -f option? If yes, create small but representative samples of dump and VI files each, run the command and post the results.

hi Rudic

whe i did sample with only 3 entries in the VI file it work ok , when i make my bigger sample (165 entry in VI ) it again copy all the dump lines (which is about 4M lines for about 180K entry )

Code:
-bash-3.00$ vi infile3.csv
"infile3.csv" [New file] 
LKHYA_AMDR2:1-0-6-22,
MFYSLKHYA_AMDR2:1-0-6-23,
MFYSLKHYA_AMDR2:1-0-7-1,
~
~
~
-bash-3.00$ 
-bash-3.00$ /usr/xpg4/bin/grep -f infile3.csv EP-ports-faults_20161218.csv > outfile3.csv

-rw-r--r--   1 motive   other       4012 Dec 20 20:02 outfile3.csv
-rw-r--r--   1 motive   other       4691 Dec 20 20:09 infile4.csv
-rw-r--r--   1 motive   other    223682851 Dec 20 20:10 outfile4.csv
-bash-3.00$

# 9  
Old 12-20-2016
Well, is it possible that all lines are correctly selected, i.e. are represented in the VI file? I recommend to do some intermediate steps, e.g. 20 and then 50 entries in VI.
This User Gave Thanks to RudiC For This Post:
# 10  
Old 12-20-2016
hi Rudi

i did and it works fine now with all entries (165) , the first line in VI file was empty , after i remove it it works fine , i wonder if this empty line make the command skip the execution and take full copy of the dump? as it was finish in no time and copy the whole big file in no time.
# 11  
Old 12-20-2016
Do you have Perl in your system?

copy and paste and save as searcher.pl
Run as
Code:
perl searcher.pl EP-ports-faults_20161218.csv

or
Code:
perl searcher.pl EP-ports-faults_201612*.csv

if you want to process multiple files.

The filename with search strings is infile2.csv, you can change it in the code
The filename with the result is outfile2.csv, you can change it in the code

Code:
#!/usr/bin/perl

use strict;
use warnings;

sub pattern {
    my $f = shift;
    chomp(my @pat = <$f>);
    join "|", @pat;
}

my $patterns_file = "infile2.csv";
open my $fh, '<', $patterns_file or die;
my $search_pattern = pattern($fh);
close $fh;

my $match_searches = "outfile2.csv";
open my $wfh, '>', $match_searches or die;

while(<>) {
    print $wfh $_ if /$search_pattern/;
}
close $wfh;

# 12  
Old 12-20-2016
Quote:
Originally Posted by is2_egypt
hi Rudi

i did and it works fine now with all entries (165) , the first line in VI file was empty , after i remove it it works fine , i wonder if this empty line make the command skip the execution and take full copy of the dump? as it was finish in no time and copy the whole big file in no time.
An empty line when using -f means that one of the patterns that it tries is the empty string and that is always a positive hit, i.e. it matches everything...
Code:
$ printf "%s\n" aa bbbb ccc | grep ''
aa
bbbb
ccc

This User Gave Thanks to Scrutinizer For This Post:
# 13  
Old 12-21-2016
The /usr/xpg4/bin/grep is quite slow with "-f bigfile".
Better use /usr/xpg4/bin/awk or nawk and a hashed array lookup
Code:
nawk -F, 'NR==FNR { A[$1]; next } ($1 in A)' file1.csv dump

Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Extract lines from a file

Hi all; Here is my file which contains a list of files (recent versions of files are in red). This file is dynamic, files versions can change at any time (versions can increment) filename ------------------------------------------------------- ... (8 Replies)
Discussion started by: chercheur111
8 Replies

2. Shell Programming and Scripting

Want to extract certain lines from big file

Hi All, I am trying to get some lines from a file i did it with while-do-loop. since the files are huge it is taking much time. now i want to make it faster. The requirement is the file will be having 1 million lines. The format is like below. ##transaction, , , ,blah, blah... (38 Replies)
Discussion started by: mad man
38 Replies

3. Shell Programming and Scripting

How to extract certain lines from a file?

Hi guys I have a several thousands line file in the following format: n817 -------------------------------------------------- n842 -------------------------------------------------- n877 -------------------------------------------------- n513 /bb/data/rmt2db.lrl:JBSKDB 31915 75... (4 Replies)
Discussion started by: aoussenko
4 Replies

4. Shell Programming and Scripting

Extract particular lines from a file

Hi all, I have a file with many records with information as given below ID A16L2_HUMAN Reviewed; 619 AA. AC Q8NAA4; A5PL30; B2RPK5; Q658V4; Q6PID3; Q8NBG0; DT 20-MAY-2008, integrated into UniProtKB/Swiss-Prot. DT 20-MAY-2008, sequence version 2. DT ... (1 Reply)
Discussion started by: kaav06
1 Replies

5. Shell Programming and Scripting

Extract some lines from one file and add those lines to current file

hi, i have two files. file1.sh echo "unix" echo "linux" file2.sh echo "unix linux forums" now the output i need is $./file2.sh unix linux forums (3 Replies)
Discussion started by: snreddy_gopu
3 Replies

6. Shell Programming and Scripting

To extract the required lines from a file

I have a files in a directory in this format data data data ---BEGIN CERT----- data data data ---END CERT ----- Now, I want to extract the lines starting from --BEGIN CERT-- and write the contents till the end of file into a new file.How can I do this for all the files in the... (1 Reply)
Discussion started by: sureshg
1 Replies

7. Shell Programming and Scripting

Extract lines from a file automatically. Please a Help

hello, hope you can help me: ive got a file called archivos The content or structure of this file is ./chu0/filechu ./chu1/filechu I extract each line from this file manually and redirect to a file, and it Works fine, so the command line is: awk ‘/chu0/ {print $0}' < archivos >... (8 Replies)
Discussion started by: alexcol
8 Replies

8. Shell Programming and Scripting

How to extract a sequence of n lines from a file

Hi I want to be able to extract a sequence of n lines from a file. ideas, commands and suggestions would be highly appreciated. Thanks (4 Replies)
Discussion started by: 0ktalmagik
4 Replies

9. Shell Programming and Scripting

how to extract a range of lines from a file

I am reading a file that contains over 5000 lines and I want to assign it to a shell variable array (which has a restriction of 1024 rows). I had an idea that if I could grab 1000 record hunks of the file, and pipe the records out, that I could perform a loop until I got to the end and process 1000... (5 Replies)
Discussion started by: beilstwh
5 Replies

10. UNIX for Dummies Questions & Answers

extract specific lines from file

hi, how would i extract a range of lines in a file by using the line number? ex: file contains: 1 title 2 i want 3 this part 4 to be taken out 5 from this file 6 and sent to 7 another file 8 not needed 9 end of file In this case, i want to copy line number 2 to 7 on a new... (2 Replies)
Discussion started by: apalex
2 Replies
Login or Register to Ask a Question