Sponsored Content
Top Forums UNIX for Dummies Questions & Answers Filter records in a huge text file from a filter text file Post 302919666 by Smiling Dragon on Friday 3rd of October 2014 01:07:24 AM
Old 10-03-2014
Code:
#!/bin/sh
while read line
do
  firstword=`echo "$line" | awk '{ print $1 }'`
  if egrep "^$firstword\$" filterfile.txt
  then
    echo "$line"
  fi
done

Depending on your OS, you might need to use grep, egrep, or grep -e.
This will be pretty slow if your filter list get's excessively long, but will scale well enough regardless of file size as it doesn't pull everything into memory.
This User Gave Thanks to Smiling Dragon For This Post:
 

10 More Discussions You Might Find Interesting

1. Windows & DOS: Issues & Discussions

Filter data from text file

Hi All We have got a text file, which has data dumped from 60 tables. From these 60 tables of data we need data from 4 tables only. I tried assigning line numbers to filter out data, but it is not working as intended. below is the sample file ----Table1----- 3,dfs,43,df 4,sd,5,edd... (18 Replies)
Discussion started by: b_sri
18 Replies

2. Shell Programming and Scripting

filter out all the records which are having space in the 8th filed of my file

I have a file which is having fileds separtaed by delimiter. Ex: C;4498;qwa;cghy;;;;40;;222122 C;4498;sample;city;;;;34 2;;222123 C;4498;qwe;xcbv;;;;34-2;;222124 C;4498;jj;sffz;;;;41;;222120 C;4498;eert;qwq;;;;34 A;;222125 C;4498;jj;szxzzd;;;;34;;222127 out of these records I... (3 Replies)
Discussion started by: indusri
3 Replies

3. Shell Programming and Scripting

Shell script to read lines in a text file and filter user data

hi all, I have this file with some user data. example: $cat myfile.txt FName|LName|Gender|Company|Branch|Bday|Salary|Age aaaa|bbbb|male|cccc|dddd|19900814|15000|20| eeee|asdg|male|gggg|ksgu|19911216||| aara|bdbm|male|kkkk|acke|19931018||23| asad|kfjg|male|kkkc|gkgg|19921213|14000|24|... (4 Replies)
Discussion started by: srimal
4 Replies

4. Shell Programming and Scripting

Filter records in a file using AWK

I want to filter records in one of my file using AWK command (or anyother command). I am using the below code awk -F@ '$1=="0003"&&"$2==20100402" print {$0}' $INPUT > $OUTPUT I want to pass the 0003 and 20100402 values through a variable. How can I do this? Any help is much... (1 Reply)
Discussion started by: gpaulose
1 Replies

5. Shell Programming and Scripting

Filter a .kml file (xml) with data set from text file

I have a .kml file. So I want filter the .kml to get only the tags that have this numeric codes that they are in a text file 11951 11952 74014 11964 11965 11969 11970 11971 11972 60149 74018 74023 86378 11976 11980 11983 11984 11987 (5 Replies)
Discussion started by: pcoj33
5 Replies

6. UNIX for Dummies Questions & Answers

Solaris - Filter columns in text file and adding new column

Hello, I am very now to this, hope you can help, I am looking into editing a file in Solaris, with dinamic collums (lenght varies) and I need 2 things to be made, the fist is to filter the first column and third column from the file bellow file.txt, and create a new file with the 2 filtered... (8 Replies)
Discussion started by: jpbastos
8 Replies

7. Shell Programming and Scripting

Filter records based on 2nd file

Hello, I want to filter records of a file if they fall in range associated with a second file. First the chr number (2nd col of 1st file and 1st col of 2nd file) needs to be matched. Then if the 3rd col of the first file falls within any of the ranges specified by the 2nd and 3rd cols , then... (4 Replies)
Discussion started by: ritakadm
4 Replies

8. Shell Programming and Scripting

Filter all the lines with minimum specified length of words of a text file

Hi Can someone tell me which script will work best (in terms of speed and simplicity to write and run) for a large text file to filter all the lines with a minimum specified length of words ? A sample script with be definitely of great help !!! Thanks in advance. :) (4 Replies)
Discussion started by: my_Perl
4 Replies

9. Shell Programming and Scripting

Shell script to filter records in a zip file that contains matching columns from another file

Not sure if this is the correct forum for this question. I have two files. file1.zip, file2 Input: file1.zip col1, col2 , col3 a , b , 0:0:0:0:0:c436:9346:d40b x, y, 0:0:0:0:0:880:39f9:c9a7 m, n , 0:0:0:0:0:80c7:9161:fe00 file2.txt col1 c4:36:93:46:d4:0b... (1 Reply)
Discussion started by: anil.v
1 Replies

10. UNIX for Beginners Questions & Answers

Filter records from a log file based on timestamp

Dear Experts, I have a log file that contains a timestamp, I would like to filter record from that file based on timestamp. For example refer below file - cat sample.txt Jan 19 20:51:48 mukul-Vostro-14-3468 systemd: pam_unix(systemd-user:session): session opened for user root by (uid=0)... (6 Replies)
Discussion started by: mukulverma2408
6 Replies
egrep(1)							   User Commands							  egrep(1)

NAME
egrep - search a file for a pattern using full regular expressions SYNOPSIS
/usr/bin/egrep [-bchilnsv] -e pattern_list [file...] /usr/bin/egrep [-bchilnsv] -f file [file...] /usr/bin/egrep [-bchilnsv] pattern [file...] /usr/xpg4/bin/egrep [-bchilnqsvx] -e pattern_list [-f file] [file...] /usr/xpg4/bin/egrep [-bchilnqsvx] [-e pattern_list] -f file [file...] /usr/xpg4/bin/egrep [-bchilnqsvx] pattern [file...] DESCRIPTION
The egrep (expression grep) utility searches files for a pattern of characters and prints all lines that contain that pattern. egrep uses full regular expressions (expressions that have string values that use the full set of alphanumeric and special characters) to match the patterns. It uses a fast deterministic algorithm that sometimes needs exponential space. If no files are specified, egrep assumes standard input. Normally, each line found is copied to the standard output. The file name is printed before each line found if there is more than one input file. /usr/bin/egrep The /usr/bin/egrep utility accepts full regular expressions as described on the regexp(5) manual page, except for ( and ), ( and ), { and }, < and >, and , and with the addition of: 1. A full regular expression followed by + that matches one or more occurrences of the full regular expression. 2. A full regular expression followed by ? that matches 0 or 1 occurrences of the full regular expression. 3. Full regular expressions separated by | or by a NEWLINE that match strings that are matched by any of the expressions. 4. A full regular expression that can be enclosed in parentheses ()for grouping. Be careful using the characters $, *, [, ^, |, (, ), and in full regular expression, because they are also meaningful to the shell. It is safest to enclose the entire full regular expression in single quotes (a'a'). The order of precedence of operators is [], then *?+, then concatenation, then | and NEWLINE. /usr/xpg4/bin/egrep The /usr/xpg4/bin/egrep utility uses the regular expressions described in the EXTENDED REGULAR EXPRESSIONS section of the regex(5) manual page. OPTIONS
The following options are supported for both /usr/bin/egrep and /usr/xpg4/bin/egrep: -b Precede each line by the block number on which it was found. This can be useful in locating block numbers by context (first block is 0). -c Print only a count of the lines that contain the pattern. -e pattern_list Search for a pattern_list (full regular expression that begins with a -). -f file Take the list of full regular expressions from file. -h Suppress printing of filenames when searching multiple files. -i Ignore upper/lower case distinction during comparisons. -l Print the names of files with matching lines once, separated by NEWLINEs. Does not repeat the names of files when the pattern is found more than once. -n Precede each line by its line number in the file (first line is 1). -s Work silently, that is, display nothing except error messages. This is useful for checking the error status. -v Print all lines except those that contain the pattern. /usr/xpg4/bin/egrep The following options are supported for /usr/xpg4/bin/egrep only: -q Quiet. Does not write anything to the standard output, regardless of matching lines. Exits with zero status if an input line is selected. -x Consider only input lines that use all characters in the line to match an entire fixed string or regular expression to be matching lines. OPERANDS
The following operands are supported: file A path name of a file to be searched for the patterns. If no file operands are specified, the standard input is used. /usr/bin/egrep pattern Specify a pattern to be used during the search for input. /usr/xpg4/bin/egrep pattern Specify one or more patterns to be used during the search for input. This operand is treated as if it were specified as -epat- tern_list.. USAGE
See largefile(5) for the description of the behavior of egrep when encountering files greater than or equal to 2 Gbyte ( 2^31 bytes). ENVIRONMENT VARIABLES
See environ(5) for descriptions of the following environment variables that affect the execution of egrep: LC_COLLATE, LC_CTYPE, LC_MES- SAGES, and NLSPATH. EXIT STATUS
The following exit values are returned: 0 If any matches are found. 1 If no matches are found. 2 For syntax errors or inaccessible files (even if matches were found). ATTRIBUTES
See attributes(5) for descriptions of the following attributes: /usr/bin/egrep +-----------------------------+-----------------------------+ | ATTRIBUTE TYPE | ATTRIBUTE VALUE | +-----------------------------+-----------------------------+ |Availability |SUNWcsu | +-----------------------------+-----------------------------+ |CSI |Not Enabled | +-----------------------------+-----------------------------+ /usr/xpg4/bin/egrep +-----------------------------+-----------------------------+ | ATTRIBUTE TYPE | ATTRIBUTE VALUE | +-----------------------------+-----------------------------+ |Availability |SUNWxcu4 | +-----------------------------+-----------------------------+ |CSI |Enabled | +-----------------------------+-----------------------------+ SEE ALSO
fgrep(1), grep(1), sed(1), sh(1), attributes(5), environ(5), largefile(5), regex(5), regexp(5), XPG4(5) NOTES
Ideally there should be only one grep command, but there is not a single algorithm that spans a wide enough range of space-time trade-offs. Lines are limited only by the size of the available virtual memory. /usr/xpg4/bin/egrep The /usr/xpg4/bin/egrep utility is identical to /usr/xpg4/bin/grep -E. See grep(1). Portable applications should use /usr/xpg4/bin/grep -E. SunOS 5.11 24 Mar 2006 egrep(1)
All times are GMT -4. The time now is 03:58 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy