Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Most efficient method to extract values from text files
Top Forums Shell Programming and Scripting Most efficient method to extract values from text files Post 302915629 by h0x0r21 on Thursday 4th of September 2014 04:19:39 AM
Old 09-04-2014
Most efficient method to extract values from text files
I have a list of files defined in a single file , one on each line.(No.of files may wary each time)

eg. content of ETL_LOOKUP.dat

Code:
/data/project/randomname
/data/project/ramname
/data/project/raname
/data/project/radomname
/data/project/raame
/data/project/andomname

size of these files is from 5-20 mb. I want to scan each of these files one by one and extract multiple values from each of them. The format for extraction would be same for each of the files.

The high level code that i am thinking of is :

Code:
while read lookupfilecontent
      wf_dir = `grep lookupfilecontent |cut -d ',' -f7` 
      sess_dir =` grep  lookupfilecontent |cut -d ',' -f10| cut -d ',' -f1`
      #this goes on for 7-8 values
done<data/SrcFiles/ETL_LOOKUP.dat

Now i want to ask if there a more efficient way for this ? Wouldn't using grep mutiple times be a performance concern as I am reading single file everytime in each line ?
Last edited by Scrutinizer; 09-04-2014 at 05:24 AM.. Reason: add'l code tags
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

File transformation - what is most efficient method

I've done quite a bit of searching on this but cannot seem to find exactly what I'm looking for. Say I have a | delimited input file with 6 columns and I need to change the value of a few columns and create an output file. With my limited knowledge I can do this with many lines of code but want... (5 Replies)
Discussion started by: 1superdork
5 Replies

2. Shell Programming and Scripting

multiplying values from two text files

Im very new to programming. But I would like to write a script which extracts and multiply values from 2 txt and output as a new file. Can someone please teach me how to write it? Thank you so much for example File A File B 1 34 1 2 2 13 2 2 3 8 3 3 File C output 1 68 2... (2 Replies)
Discussion started by: crunchichichi
2 Replies

3. Shell Programming and Scripting

Extract text and store in separate files

Hi, I have a file which looks like this: .I 1 some text .A this is the first line .I 2 some text again .B this is the second line .I 3 again some text .C this is the third line I want to have my output like this in separate files: (7 Replies)
Discussion started by: shoaibjameel123
7 Replies

4. Programming

modifying 2 or more values using method in Java

i wanna make modification in two or more identifiers,using a method in JAVA , but the only way to do this is by returning a value ,then also only one variable value can be modified .. one way 2 do do this is by use of array and passing its name in method ,, please suggest a different way of doing... (2 Replies)
Discussion started by: upvan111
2 Replies

5. Homework & Coursework Questions

Efficient Text File Writing

Use and complete the template provided. The entire template must be completed. If you don't, your post may be deleted! 1. The problem statement, all variables and given/known data: Write a template main.c file via shell script to make it easier for yourself later. The issue here isn't writing... (2 Replies)
Discussion started by: george3isme
2 Replies

6. Shell Programming and Scripting

Efficient population of array from text file

Hi, I am trying to populate an array with data from a text file. I have a working method using awk but it is too slow and inefficent. See below. The text file has 70,000 lines. As awk is a line editor it reads each line of the file until it gets to the required line and then processes it.... (3 Replies)
Discussion started by: carlr
3 Replies

7. Shell Programming and Scripting

Efficient method of determining if a string is in a file.

Hi, I was hoping someone could suggest an alternative to code I currently have as mine takes up far too much processor time and it to slow. The situation: I have a programme that runs on some files just before they are zipped up and archived, the program appends a one line summary of the... (4 Replies)
Discussion started by: RECrerar
4 Replies

8. Shell Programming and Scripting

Extract lines from text files

I have some files containing the following data # RESIDUE AA STRUCTURE BP1 BP2 ACC N-H-->O O-->H-N N-H-->O O-->H-N TCO KAPPA ALPHA PHI PSI X-CA Y-CA Z-CA 1 196 A M 0 0 230 0, 0.0 2,-0.2 0, 0.0 0, 0.0 0.000 360.0 360.0 360.0 76.4 21.7 -6.8 11.3 2 197 A D + 0 0 175 1,-0.1 2,-0.1 0, 0.0 0, 0.0... (10 Replies)
Discussion started by: edweena
10 Replies

9. UNIX for Dummies Questions & Answers

More efficient way to print variable values?

Hello, Through the process of a executing a shell script, I extract the values for a number of variables (arbitrarily declared as a through i) and towards the end I print them out like shown below: #!bin/sh # bits of code to get values for the variables ...... ...... # print the values... (4 Replies)
Discussion started by: Gussifinknottle
4 Replies

10. Shell Programming and Scripting

Portable and efficient way to add text after pattern

Shell: sh/bash OS: Linux (all unix flavors) Suppose i have a variable with this content: ArgZ=' import os import sys MySpecialpath = os.path.abspath(sys.argv) # ' ArgZB='#REGEN #REGEN #REGEN ' I want to add this text to a file/script, only under the following conditions: 1. ... (1 Reply)
Discussion started by: SkySmart
1 Replies

LEARN ABOUT DEBIAN

innoextract

innoextract(1)						      General Commands Manual						    innoextract(1)

NAME

       innoextract - tool to extract installers created by Inno Setup

SYNOPSIS

       innoextract [-behlLqstv] [-ccolor] [-pprogress] installers ...

DESCRIPTION

       innoextract is a tool that can extract installer executables created by Inno Setup.

       innoextract will extract files from a installers specified on the command line.

       To  extract  a  multi-part  installer with external data files, only the executable (.exe) file needs to be given as an argument to innoex-
       tract.

OPTIONS

       -c --color [enable]
	      By default innoextract will try to detect if the terminal supports shell escape codes and enable or  disable  color  output  accord-
	      ingly. Pass 1 or true to --color to force color output. Pass 0 or false to never output color codes.

       --dump Don't convert Windows paths to UNIX paths and don't substitute variables in paths.

       -e --extract
	      Extract all files to the current directory. This is the default action. You may only specify one of --extract , --list and --test

       -h --help
	      Show a list of the supported options.

       --language [lang]
	      Extract only language-independent files and files for the given language. By default all files are extracted.

       --license
	      Show license information.

       -l --list
	      List files contained in the installer but don't extract anything. You may only specify one of --extract , --list and --test

       -L --lowercase
	      Convert filenames stored in the installer to lower-case before extracting.

       -p --progress [enable]
	      By  default  innoextract	will  try  to detect if the terminal supports shell escape codes and enable or disable progress bar output
	      accordingly. Pass 1 or true to --progress to force progress bar output. Pass 0 or false to never show a progress bar.

       -q --quiet
	      Less verbose output.

       -s --silent
	      Don't output anything except errors and warnings.

       -t --test
	      Test archive integrity but don't write any output files. You may only specify one of --extract , --list and --test

       -v --version
	      Show the innoextract version number and supported Inno Setup versions.

LIMITATIONS

       innoextract currently only supports extracting all the data. There is no support for extracting individual files, components or languages.

       Included scripts and checks are not executed.

       Data is always extracted to the current directory and the mapping from Inno Setup variables like the application directory  to  subdirecto-
       ries is hard-coded.

       innoextract  does  not check if an installer includes multiple files with the same name and will continually overwrite the destination file
       when extracting.

       Names for data files in multi-file installers must follow the standard naming scheme.

       Encrypted installers are not supported.

SEE ALSO

       cabextract(1), unshield(1)

BUGS

       No known bugs.

AUTHOR

       Daniel Scharrer (daniel@constexpr.org)

1.2								    2012-04-01							    innoextract(1)
All times are GMT -4. The time now is 03:28 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy