Visit Our UNIX and Linux User Community


Bash : Checking Large file for specific lines


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Bash : Checking Large file for specific lines
# 1  
Old 06-22-2018
Bash : Checking Large file for specific lines

Morning ..

I have a file with approximately 1000 lines. I want to check that the file contains, for example, 100 lines.
Something like whats given below is ugly. And even if I create a function I have to call it 100 times.
I may need to look through multiple files at times.

Is there a "cleaner" way to do this ?

Code:
x=$(grep "now is the time" 1000linefile.txt)
 if [ -z $x ]
  then
   echo "now is the time not found"
 fi 

y=$(grep "for all good men" 1000linefile.txt)
 if [ -z $y ]
  then
   echo "for all good men not found"
 fi

Thanks !!! AND HAPPY FRIEDAY Smilie Smilie

---------- Post updated at 11:15 AM ---------- Previous update was at 10:48 AM ----------

Well I guess I answered my own question .. once I though about it.
If *.txt contains the 1000 line file(s) and linestocheckfile contains the 100 lines that I want to be sure is in the large file.

Code:
for x in `ls *.txt`
do
 while read line
  do
   z=$(grep "$line" $x)
    if [ -z "$z" ]
     then
      echo "$line is missing from $x"
    fi
 done < linestocheckfile
done

# 2  
Old 06-22-2018
a bit simplified:
Code:
if [ $(grep -c mySearchPattern myFile) == 0 ]; then
   echo 'not found'
else
   echo 'found'
fi

This User Gave Thanks to vgersh99 For This Post:
# 3  
Old 06-22-2018
I would go further:-
Code:
if $(grep -q "now is the time" 1000linefile.txt)
then
   echo "It's a HIT"
else
   echo "Missed"
fi

The extra benefit is that the -q flag will cause the grep to exit immediately that it has matched, so if you have a properly large file (more than just a few disk blocks), it won't have to read it all if it finds a match, saving you IO time.

Potentially you could extend the search using an expression, but that's up to you knowing what you are looking for. If you want to know the state of each you would have to run twice right through the file. If you want to know if either is listed, then:-
Code:
if $(egrep -q "now is the time|for all good men" 1000linefile.txt)
then
   echo "It's a HIT"
else
   echo "Missed"
fi




I hope that this helps,
Robin
These 2 Users Gave Thanks to rbatte1 For This Post:
# 4  
Old 06-22-2018
I'd propose to drop the "command substitution" $( ... ) as it's not needed in above construct.


If you have all target lines in a file, and you are sure they occur not more than once in the large file, you could try
Code:
if [ $(grep -cwf targetlines 1000linesfile) = $(wc -l < targetlines) ]
  then    echo "Found all"
   else    echo "Some missing"
   fi

To allow for duplicates of target lines, try $(grep -owf targetlines 1000linesfile | sort -u | wc -l) in the if construct above.

Last edited by RudiC; 06-22-2018 at 03:18 PM..
This User Gave Thanks to RudiC For This Post:
# 5  
Old 06-22-2018
Thanks guys !!

Previous Thread | Next Thread
Test Your Knowledge in Computers #345
Difficulty: Medium
Scott McNealy received $238 million USD when the Oracle acquisition closed.
True or False?

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Bash: copying lines with specific character to files with same name as copied line.

I am trying to make my script as simple as a possible but, I am not sure if the way I am approaching is necessarily the most efficient or effective it can be. What I am mainly trying to fix is a for loop to remove a string from the specified files and within this loop I am trying to copy the lines... (2 Replies)
Discussion started by: Allie_gastrator
2 Replies

2. Shell Programming and Scripting

Adding lines to a large file

Hello, I have a relatively large text file (25,000K) consisting of records of data. For each record, I need to create a new line based on what is already there. Every record has a block that looks like, M END > <ID> 1 > <SOURCE> KEGG > <SOURCE_ID> C00002 > <NAME> ATP;... (4 Replies)
Discussion started by: LMHmedchem
4 Replies

3. UNIX for Dummies Questions & Answers

Quick UNIX command to display specific lines in the middle of a file from/to specific word

This could be a really dummy question. I have a log text file. What unix command to extract line from specific string to another specific string. Is it something similar to?: more +/"string" file_name Thanks (4 Replies)
Discussion started by: aku
4 Replies

4. Shell Programming and Scripting

Cutting specific line of a file by checking condition

testfile.csv 0","1125209",,"689202CBx18888",,"49",,,"NONMC",,,,,"01112010",,,,,,,"MTM- "1","",,"689202ABx19005",,"49",,,"NONMC",,,,,"01072010",,,,,,,"MTM- testfile.csv looks like above format if the second column is null then get 23rd column and store in a different varible .. add all the... (1 Reply)
Discussion started by: mgant
1 Replies

5. Shell Programming and Scripting

[bash help]Adding multiple lines of text into a specific spot into a text file

I am attempting to insert multiple lines of text into a specific place in a text file based on the lines above or below it. For example, Here is a portion of a zone file. IN NS ns1.domain.tld. IN NS ns2.domain.tld. IN ... (2 Replies)
Discussion started by: cdn_humbucker
2 Replies

6. Shell Programming and Scripting

remove a specific line in a LARGE file

Hi guys, i have a really big file, and i want to remove a specific line. sed -i '5d' fileThis doesn't really work, it takes a lot of time... The whole script is supposed to remove every word containing less than 5 characters and currently looks like this: #!/bin/bash line="1"... (2 Replies)
Discussion started by: blubbiblubbkekz
2 Replies

7. UNIX for Dummies Questions & Answers

how to display specific lines of a specific file

are there any basic commands that can display lines 99 - 101 of the /etc/passwd file? I'm thinking use of head and tail, but I forget what numbers to use and where to put /etc/passwd in the command. (2 Replies)
Discussion started by: raidkridley
2 Replies

8. Shell Programming and Scripting

Searching a specific line in a large file

Hey All Can any one please suggest the procedure to search a part of line in a very large file in which log entries are entered with very high speed. i have trued with grep and egrep grep 'text text text' <file-name> egrep 'text text text' <file-name> here 'text text text' is... (4 Replies)
Discussion started by: NIMISH AGARWAL
4 Replies

9. Shell Programming and Scripting

Count specific character(s) very large file

I'm trying to count the number of 2 specific characters in a very large file. I'd like to avoid using gsub because its taking too long. I was thinking something like: awk '-F' { t += NF - 1 } END {print t}' infile > outfile which isn't working Any ideas would be great. (3 Replies)
Discussion started by: dcfargo
3 Replies

10. UNIX for Dummies Questions & Answers

Help with selecting specific lines in a large file

Hello, I need to select the 3 lines above as well as below a search string, including the search string. I have been trying various combinations using sed command without any success. Can anuone help please. Thanking (2 Replies)
Discussion started by: tansha
2 Replies

Featured Tech Videos