Large file search


 
Thread Tools Search this Thread
Top Forums UNIX for Dummies Questions & Answers Large file search
# 1  
Old 08-18-2003
Large file search

How do I search through a file that is so large (over 1GB) and find out exactly on what the exact pattern is found. I've used grep and know the line exist but what it returns is not complete because there are blobs in the data and there are control characters, etc and part of the data may beyond 2048 characters limit in vi...

If I can figure out exactly what line number the pattern is found in this large file I can at least attempt to do a head|tail|more combinations or split it correctly...

I'd like to find it using vi editor but the file's too large..

Thanks.
Gian
# 2  
Old 08-18-2003
"blobs in the data" .... I don't believe that I've encountered that particular complaint before. But I can help with the rest.

grep has a -n option to print line numbers. But you really should switch to sed. sed can process data rather quickly. It can handle large files well. And it has plenty of power. Here are a few examples:

sed -n '/<pattern>/p' < datafile
sed -n '/<pattern>/=' < datafile
sed -n /<pattern>/{ =;p;}' < datafile
sed -n 25p < datafile
sed -n 25,30p < datafile

The first one displays the lines that match the pattern.
The second displays the line numbers of the matches.
The third one does both, using a single pass though the file.
Then we print line 25, then lines 25 through 30.

And you can use the l command (letter l) instead of the p command. The l command will render the line readable if it has special characters or is very long.
# 3  
Old 08-20-2003
Thank you!!! This is perfect.
 
Login or Register to Ask a Question

Previous Thread | Next Thread

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Bash script search, improve performance with large files

Hello, For several of our scripts we are using awk to search patterns in files with data from other files. This works almost perfectly except that it takes ages to run on larger files. I am wondering if there is a way to speed up this process or have something else that is quicker with the... (15 Replies)
Discussion started by: SDohmen
15 Replies

2. UNIX for Beginners Questions & Answers

Search spaces in 5th column in large file

i have a file having 5 columns with more than million records. And i want to search using UNIX command to find if there are any spaces in 5th column. any please help. (1 Reply)
Discussion started by: sivakumar.p
1 Replies

3. UNIX for Beginners Questions & Answers

sed awk: split a large file to unique file names

Dear Users, Appreciate your help if you could help me with splitting a large file > 1 million lines with sed or awk. below is the text in the file input file.txt scaffold1 928 929 C/T + scaffold1 942 943 G/C + scaffold1 959 960 C/T +... (6 Replies)
Discussion started by: kapr0001
6 Replies

4. Shell Programming and Scripting

Large search replace using sed results in memory problem.

I have one big file of size 9GB (big_file.txt). This big file has sentences and paragraphs like any usual English document. I have another file consisting of replacement strings for sed to use. The file name is replace.sed and each entry in one line looks like this: s/\<shout\>/shout/g s/\<b is... (2 Replies)
Discussion started by: shoaibjameel123
2 Replies

5. Shell Programming and Scripting

Read in search strings from text file, search for string in second text file and output to CSV

Hi guys, I have a text file named file1.txt that is formatted like this: 001 , ID , 20000 002 , Name , Brandon 003 , Phone_Number , 616-234-1999 004 , SSNumber , 234-23-234 005 , Model , Toyota 007 , Engine ,V8 008 , GPS , OFF and I have file2.txt formatted like this: ... (2 Replies)
Discussion started by: An0mander
2 Replies

6. Shell Programming and Scripting

Retrieving the relevant search from search file in the main file

I have two files: file 1: hello.com neo.com,japan.com,example.com news.net xyz.com, telecom.net, highlands.net, software.com example2.com earth.net, abc.gov.uk file 2: neo.com example.com abc.gov.uk file 2 are the search keys to search in file 1 if any of the search... (7 Replies)
Discussion started by: csim_mohan
7 Replies

7. Shell Programming and Scripting

highly specific search and replace for a large number of files

hey guys, I have a directory with about 600 files. I need to find a specific word inside a command and replace only that instance of the word in many files. For example, lets say I have a command called 'foo' in many files. One of the input arguments of the 'foo' call is 'bar'. The word 'bar'... (5 Replies)
Discussion started by: ksubrama
5 Replies

8. Shell Programming and Scripting

Script to search a large file with a list of terms in another file

Hi- I am trying to search a large file with a number of different search terms that are listed one per line in 3 different files. Most importantly I need to be able to do a case insensitive search. I have tried just using egrep -f but it doesn't seam to be able to handle the -i option when... (3 Replies)
Discussion started by: dougzilla
3 Replies

9. Shell Programming and Scripting

Search for multiple lines in large file

Hi, I have a requirement to search for a string in a large log file along with few lines before and after the the string. The following script was sufficient to search such an entry. STRING_TO_GREP="$1" FILE_TO_GREP="$2" NUMBER_OF_LINES_BEFORE=$3 NUMBER_OF_LINES_AFTER=$4 for i in `grep... (3 Replies)
Discussion started by: praveen123
3 Replies

10. Shell Programming and Scripting

Performance issue in UNIX while generating .dat file from large text file

Hello Gurus, We are facing some performance issue in UNIX. If someone had faced such kind of issue in past please provide your suggestions on this . Problem Definition: /Few of load processes of our Finance Application are facing issue in UNIX when they uses a shell script having below... (19 Replies)
Discussion started by: KRAMA
19 Replies
Login or Register to Ask a Question