Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Performance issue with 'grep' command for huge file size
Operating Systems HP-UX Performance issue with 'grep' command for huge file size Post 302574529 by arb_1984 on Thursday 17th of November 2011 12:24:54 PM
Old 11-17-2011
Performance issue with 'grep' command for huge file size

I have 2 files; one file (say, details.txt) contains the details of employees and another file (say, emp.txt) has some selected employee names. I am extracting employee details from details.txt by using emp.txt and the corresponding code is:
Code:
while read line
do
emp_name=`echo $line`
grep -e $emp_name details.txt >> output.txt
done < emp.txt

Above code is working fine and I am getting expected result. But, this code is taking too much time (I don't have exact time, more than 6 hrs, later on cancelled the script) while the file size is huge. As an example, I have details.txt of around 2.5GB and record count is around 7.5lacs and the emp.txt has 55K employee name. Can you please suggest any other option/ command which will be better to handle such huge file. Thanks.
Last edited by vbe; 11-18-2011 at 09:33 AM.. Reason: attempt to use code tags ( fighting with fonts...)
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Grep matched records from huge file

111111111100000000001111111111 123232323200000010001114545454 232435424200000000001232131212 342354234301000000002323423443 232435424200000000001232131212 2390898994200000000001238908092 This is the record format. From 11th position to 20th position in a record there are 0's occuring,and... (6 Replies)
Discussion started by: mjkreddy
6 Replies

2. Shell Programming and Scripting

performance of shell script ( grep command)

Hi, I have to find out the run time for 40-45 different componets. These components writes in to a genreric log file in a single directory. eg. directory is LOG and the log file name format is generic_log_<process_id>_<date YY_MM_DD_HH_MM_SS>.log i am taking the run time using the time... (3 Replies)
Discussion started by: vikash_k
3 Replies

3. Shell Programming and Scripting

Implement in one line sed or awk having no delimiter and file size is huge

I have file which contains around 5000 lines. The lines are fixed legth but having no delimiter.Each line line contains nearly 3000 characters. I want to delete the lines a> if it starts with 1 and if 576th postion is a digit i,e 0-9 or b> if it starts with 0 or 9(i,e header and footer) ... (4 Replies)
Discussion started by: millan
4 Replies

4. Shell Programming and Scripting

Severe performance issue while 'grep'ing on large volume of data

Background ------------- The Unix flavor can be any amongst Solaris, AIX, HP-UX and Linux. I have below 2 flat files. File-1 ------ Contains 50,000 rows with 2 fields in each row, separated by pipe. Row structure is like Object_Id|Object_Name, as following: 111|XXX 222|YYY 333|ZZZ ... (6 Replies)
Discussion started by: Souvik
6 Replies

5. Shell Programming and Scripting

FTP a huge Size file

Dear All, Good Evening!! I have a requirement to ftp a 220GB backup file to a remote backup server. I wrote a script for this purpose. But it takes more than 8 hours to transfer this file. Is there any other method to do it in less time??? Thanks in Advance!!! ---------- Post updated... (5 Replies)
Discussion started by: Naga06
5 Replies

6. Shell Programming and Scripting

Optimised way for search & replace a value on one line in a very huge file (File Size is 24 GB).

Hi Experts, I had to edit (a particular value) in header line of a very huge file so for that i wanted to search & replace a particular value on a file which was of 24 GB in Size. I managed to do it but it took long time to complete. Can anyone please tell me how can we do it in a optimised... (7 Replies)
Discussion started by: manishkomar007
7 Replies

7. Shell Programming and Scripting

Performance issue while using find command

Hi, I have created a shell script for Server Log Automation Process. I have used find xargs grep command to search the string. for Example, find -name | xargs grep "816995225" > test.txt . Here my problem is, We have lot of records and we want to grep the string... (4 Replies)
Discussion started by: nanthagopal
4 Replies

8. UNIX for Advanced & Expert Users

Performance problem with removing duplicates in a huge file (50+ GB)

I'm trying to remove duplicate data from an input file with unsorted data which is of size >50GB and write the unique records to a new file. I'm trying and already tried out a variety of options posted in similar threads/forums. But no luck so far.. Any suggestions please ? Thanks !! (9 Replies)
Discussion started by: Kannan K
9 Replies

9. UNIX for Dummies Questions & Answers

What is the faster way to grep from huge file?

Hi All, I am new to this forum and this is my first post. My requirement is like to optimize the time taken to grep the file with 40000 lines. There are two files FILEA(40000 lines) FILEB(40000 lines). The requirement is like this, both the file will be in the format below... (11 Replies)
Discussion started by: mad man
11 Replies

10. Shell Programming and Scripting

Performance Issue for a file search command

Hi All, This query is regarding performance improvement of a command. I have a list of IDs in a file (say file1 with single ID column) and file2 has the data rows. I need to get the IDs from file1 and search in file2, matching rows from file2 should be written to a file3. For this... (4 Replies)
Discussion started by: Tanu
4 Replies

LEARN ABOUT DEBIAN

mdbfontset

mdbFontset(5)							 The m17n Library						     mdbFontset(5)

NAME

       mdbFontset - Fontset

DESCRIPTION

       The m17n library loads a fontset definition from the m17n database by the tags <fontset, FONTSET-NAME>. The plist format of the data is as
       follows:

       FONTSET ::= PER-SCRIPT * PER-CHARSET * FALLBACK *

       PER-SCRIPT ::= '(' SCRIPT PER-LANGUAGE + ')'

       PER-LANGUAGE ::= '(' LANGUAGE FONT-SPEC-ELEMENT + ')'

       PER-CHARSET ::= '(' CHARSET FONT-SPEC-ELEMENT + ')'

       FALLBACK ::= FONT-SPEC-ELEMENT

       FONT-SPEC-ELEMENT ::= '(' FONT-SPEC [ FLT-NAME ] ')'

       FONT-SPEC ::=
	    '(' [ FOUNDRY FAMILY
		  [ WEIGHT [ STYLE [ STRETCH [ ADSTYLE ]]]]]
		REGISTRY
	     [ OTF-SPEC ] [ LANG-SPEC ] ')'

       SCRIPT is a symbol of script name (e.g. latin, han) or nil. LANGUAGE is a two-letter symbol of language name code defined by ISO 639 (e.g.
       ja, zh) or nil.

       FONT-SPEC is to specify properties of a font. FOUNDRY to REGISTRY are symbols corresponding to Mfoundry to Mregistry property of a font.
       See m17nFont for the meaning of each property.

       OTF-SPEC is a symbol specifyng the required OTF features. The symbol name has the following syntax.

	 OTF-SPEC-NAME ::= ':otf=' SCRIPT LANGSYS ? GSUB-FEATURES ? GPOS-FEATURES ?

	 SCRIPT ::= SYMBOL
	 LANGSYS ::= '/' SYMBOL

	 GSUB-FEATURES ::= '=' FEATURE-LIST ?

	 GPOS-FEATURES ::= '+' FEATURE-LIST ?

	 FEATURE-LIST ::= '~' ? FEATURE ( ',' '~' ? FEATURE ',' )

       Here, FEATURE is a four-letter Open Type feature.

       LANG-SPEC is a symbol specifying the required language support. The symbol name has the following syntax.

	 LANG-SPEC-NAME ::= ':lang=' LANG

       Here, LANG is a two or three-letter ISO-639 language code.

       FLT-NAME is a name of Font Layout Table (Font Layout Table).

EXAMPLE

       This is an example of PER_SCRIPT.

       (han
	 (ja
	   ((jisx0208.1983-0)))
	 (zh
	   ((gb2312.1980-0)))
	 (nil
	   ((big5-0))))

       It instructs the font selector to use a font of registry 'jisx0208.1983-0' for a 'han' character (i.e. a character whose Mscript property
       is 'han') if the character has Mlanguage text property 'ja' in an M-text and the character is in the repertories of such fonts. Otherwise,
       try a font of registry 'gb2312.1980-0' or 'big5-0'. If that 'han' character does not have Mlanguage text property, try all three fonts.

       See the function mdraw_text() for the detail of how a font is selected.

COPYRIGHT

       Copyright (C) 2001 Information-technology Promotion Agency (IPA)
       Copyright (C) 2001-2011 National Institute of Advanced Industrial Science and Technology (AIST)
       Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License
       <http://www.gnu.org/licenses/fdl.html>.

Version 1.6.2							    12 Jan 2011 						     mdbFontset(5)
All times are GMT -4. The time now is 11:31 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy