Sponsored Content
Operating Systems HP-UX Performance issue with 'grep' command for huge file size Post 302574570 by arb_1984 on Thursday 17th of November 2011 03:07:31 PM
Old 11-17-2011
Thank you all for your quick response !! Thanks a lot rwuertn; '-F' option is working and I am able to extract the required data within less time period.

However, the files are like:
Code:
emp.txt
------------
John
Kevin
Prakash
Susan
Ken

details.txt
-------------
HDR|Prakash D
DTL|Prakash|EMP0000010|Sr Associate|FL
HDR|Kevin T
DTL|Kevin|EMP0000004|Analyst|IL
HDR|John M
DTL|John|EMP0000184|Manager|CA

Thanks again Smilie

Last edited by Scott; 11-17-2011 at 05:52 PM.. Reason: Code tags...
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Grep matched records from huge file

111111111100000000001111111111 123232323200000010001114545454 232435424200000000001232131212 342354234301000000002323423443 232435424200000000001232131212 2390898994200000000001238908092 This is the record format. From 11th position to 20th position in a record there are 0's occuring,and... (6 Replies)
Discussion started by: mjkreddy
6 Replies

2. Shell Programming and Scripting

performance of shell script ( grep command)

Hi, I have to find out the run time for 40-45 different componets. These components writes in to a genreric log file in a single directory. eg. directory is LOG and the log file name format is generic_log_<process_id>_<date YY_MM_DD_HH_MM_SS>.log i am taking the run time using the time... (3 Replies)
Discussion started by: vikash_k
3 Replies

3. Shell Programming and Scripting

Implement in one line sed or awk having no delimiter and file size is huge

I have file which contains around 5000 lines. The lines are fixed legth but having no delimiter.Each line line contains nearly 3000 characters. I want to delete the lines a> if it starts with 1 and if 576th postion is a digit i,e 0-9 or b> if it starts with 0 or 9(i,e header and footer) ... (4 Replies)
Discussion started by: millan
4 Replies

4. Shell Programming and Scripting

Severe performance issue while 'grep'ing on large volume of data

Background ------------- The Unix flavor can be any amongst Solaris, AIX, HP-UX and Linux. I have below 2 flat files. File-1 ------ Contains 50,000 rows with 2 fields in each row, separated by pipe. Row structure is like Object_Id|Object_Name, as following: 111|XXX 222|YYY 333|ZZZ ... (6 Replies)
Discussion started by: Souvik
6 Replies

5. Shell Programming and Scripting

FTP a huge Size file

Dear All, Good Evening!! I have a requirement to ftp a 220GB backup file to a remote backup server. I wrote a script for this purpose. But it takes more than 8 hours to transfer this file. Is there any other method to do it in less time??? Thanks in Advance!!! ---------- Post updated... (5 Replies)
Discussion started by: Naga06
5 Replies

6. Shell Programming and Scripting

Optimised way for search & replace a value on one line in a very huge file (File Size is 24 GB).

Hi Experts, I had to edit (a particular value) in header line of a very huge file so for that i wanted to search & replace a particular value on a file which was of 24 GB in Size. I managed to do it but it took long time to complete. Can anyone please tell me how can we do it in a optimised... (7 Replies)
Discussion started by: manishkomar007
7 Replies

7. Shell Programming and Scripting

Performance issue while using find command

Hi, I have created a shell script for Server Log Automation Process. I have used find xargs grep command to search the string. for Example, find -name | xargs grep "816995225" > test.txt . Here my problem is, We have lot of records and we want to grep the string... (4 Replies)
Discussion started by: nanthagopal
4 Replies

8. UNIX for Advanced & Expert Users

Performance problem with removing duplicates in a huge file (50+ GB)

I'm trying to remove duplicate data from an input file with unsorted data which is of size >50GB and write the unique records to a new file. I'm trying and already tried out a variety of options posted in similar threads/forums. But no luck so far.. Any suggestions please ? Thanks !! (9 Replies)
Discussion started by: Kannan K
9 Replies

9. UNIX for Dummies Questions & Answers

What is the faster way to grep from huge file?

Hi All, I am new to this forum and this is my first post. My requirement is like to optimize the time taken to grep the file with 40000 lines. There are two files FILEA(40000 lines) FILEB(40000 lines). The requirement is like this, both the file will be in the format below... (11 Replies)
Discussion started by: mad man
11 Replies

10. Shell Programming and Scripting

Performance Issue for a file search command

Hi All, This query is regarding performance improvement of a command. I have a list of IDs in a file (say file1 with single ID column) and file2 has the data rows. I need to get the IDs from file1 and search in file2, matching rows from file2 should be written to a file3. For this... (4 Replies)
Discussion started by: Tanu
4 Replies
pfshdrcalibrate(1)					      General Commands Manual						pfshdrcalibrate(1)

NAME
pfshdrcalibrate - Create an HDR image or calibrate a response curve from a set of differently exposed images supplied in PFS stream. SYNOPSIS
pfshdrcalibrate [--response <type>] [--calibration <type>] [--gauss <val>] [--response-file <filename.m>] [--save-response <filename.m>] [--multiplier <val>] [--bpp <val>] [--luminance] [--samples <val>] [--help] [--verbose] DESCRIPTION
Create an HDR image or calibrate a response curve from a set of differently exposed images supplied in PFS stream. When used with 8bit images, luminance in the output HDR image corresponds to real world values in [cd/m^2] provided that hdrgen script con- tained correct information on exposure time, aperture and iso speed. Note that sometimes ISO speed indicated by camera does not correspond to standard (ISO-100 is in fact ISO-125). The accuracy of absolute calibration has not been thoroughly tested with different camera models, however one can expect the relative mea- surement error below 8%. Use pfsabsolute in case of systematic error. OPTIONS
--response <type>, -r <type> Allows to choose from predefined response curves. This can be used either to apply this response or use it as an initialization for automatic self-calibration. Predefined response curves are: "linear", "gamma", "log". Default is "linear". --calibration <type>, -c <type> Type of automatic self-calibration method used for recovery of the response curve. Accepted types include: "none", "robertson", "mitsunaga". Default is "robertson". In case "none" is chosen, a predefined response will be used, without self-calibration. More infomation on the algorithms can be found in: M.A. Robertson, S. Borman and R.L. Stevenson Dynamic range improvement through multiple exposures In: Proc. of International Conference on Image Processing 1999 (ICIP 99), pp 159-163 vol.3 and T. Mitsunaga and S. K. Nayar Radiometric Self Calibration In: Proc on IEEE Conf. on Computer Vision and Pattern Recognition (CVPR'99). Volume 1, p. 1374 --gauss <val>, -g <val> Sigma value for the Gaussian used as a weighting function. Applies to Robertson02 algorithm. Default value: 16.0f --response-file <filename.m>, -f <filename.m> Use response curve saved in the matlab format file. Turns off automatic self-calibration. Uses Robertson02 model to apply the response curve. --save-response <filename.m>, -s <filename.m> Saves the response curve calculated during automatic self-calibration stage in a matlab format file. Can be later reused for set of images captured with given camera. Also works fine for plotting with gnuplot. --multiplier <val>, -m <val> Input multiplier value. Can be used to manipulate the range of source exposures. Default value is 256 since LDR images are by default scaled to 0..1. --bpp <val>, -b <val> Number of bits per pixel in input data from the camera. Default value is 8. --samples <val>, -p <val> Number of sample pixels used in inverse response computations in Mitsunaga algorithm. Default is 50000. --fix-saturated, -x Use this option if you see black pixels in overexposed / saturated areas. The black pixels are visible if all exposures contain pixel values that are outside reliable range (are under- or over-exposed). This flag gives non-zero weight for the brightest and the darkest pixels, thus avoiding zero-weighted pixels. Note that the calculated luminance values for these pixels are not reliable. --luminance, -Y Recovery of response curve will be performed for luminance channel only. --verbose Print additional information during program execution. --help Print list of commandline options. EXAMPLES
pfsinhdrgen sample.hdrgen | pfshdrcalibrate -v -s response.m Recover the response curve from set of exposures defined in sample.hdrgen and save it to response.m file. To view the response curve, run gnuplot and write 'plot "response.m"'. pfsinhdrgen sample.hdrgen | pfshdrcalibrate -x -f response.m | pfsview Create an HDR image from exposures defined in sample.hdrgen using the response curve "response.m" and view it. Fix the problem with black values given to overexposed pixels. pfsinhdrgen sample.hdrgen | pfshdrcalibrate | pfsview Create an HDR image from exposures defined in sample.hdrgen using the default self-calibration method and view it. pfsinhdrgen sample_dcraw.hdrgen | pfshdrcalibrate -b 16 -r linear -c none | pfsview Given that the script sample_dcraw.hdrgen refers to camera RAW files (see pfsindcraw), this example will generate an HDR image assuming a linear response. pfsinhdrgen sample.hdrgen | pfshdrcalibrate | pfsview Create an HDR image from exposures defined in sample.hdrgen using the default self-calibration method and view it. pfsinhdrgen sample.hdrgen | pfshdrcalibrate -c mitsunaga -samples 100000 | pfsglview Create an HDR image from exposures defined in sample.hdrgen using the mitsunaga self-calibration method with 100000 samples and view it in pfsglview. SEE ALSO
pfsinhdrgen(1) jpeg2hdrgen(1) pfsview(1) pfsindcraw(1) pfsabsolute(1) pfsglview(1) BUGS
Please report bugs and comments on implementation to Grzegorz Krawczyk <gkrawczyk@users.sourceforge.net>. pfshdrcalibrate(1)
All times are GMT -4. The time now is 02:12 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy