Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Most efficient method to extract values from text files
Top Forums Shell Programming and Scripting Most efficient method to extract values from text files Post 302915629 by h0x0r21 on Thursday 4th of September 2014 04:19:39 AM
Old 09-04-2014
Most efficient method to extract values from text files
I have a list of files defined in a single file , one on each line.(No.of files may wary each time)

eg. content of ETL_LOOKUP.dat

Code:
/data/project/randomname
/data/project/ramname
/data/project/raname
/data/project/radomname
/data/project/raame
/data/project/andomname

size of these files is from 5-20 mb. I want to scan each of these files one by one and extract multiple values from each of them. The format for extraction would be same for each of the files.

The high level code that i am thinking of is :

Code:
while read lookupfilecontent
      wf_dir = `grep lookupfilecontent |cut -d ',' -f7` 
      sess_dir =` grep  lookupfilecontent |cut -d ',' -f10| cut -d ',' -f1`
      #this goes on for 7-8 values
done<data/SrcFiles/ETL_LOOKUP.dat

Now i want to ask if there a more efficient way for this ? Wouldn't using grep mutiple times be a performance concern as I am reading single file everytime in each line ?
Last edited by Scrutinizer; 09-04-2014 at 05:24 AM.. Reason: add'l code tags
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

File transformation - what is most efficient method

I've done quite a bit of searching on this but cannot seem to find exactly what I'm looking for. Say I have a | delimited input file with 6 columns and I need to change the value of a few columns and create an output file. With my limited knowledge I can do this with many lines of code but want... (5 Replies)
Discussion started by: 1superdork
5 Replies

2. Shell Programming and Scripting

multiplying values from two text files

Im very new to programming. But I would like to write a script which extracts and multiply values from 2 txt and output as a new file. Can someone please teach me how to write it? Thank you so much for example File A File B 1 34 1 2 2 13 2 2 3 8 3 3 File C output 1 68 2... (2 Replies)
Discussion started by: crunchichichi
2 Replies

3. Shell Programming and Scripting

Extract text and store in separate files

Hi, I have a file which looks like this: .I 1 some text .A this is the first line .I 2 some text again .B this is the second line .I 3 again some text .C this is the third line I want to have my output like this in separate files: (7 Replies)
Discussion started by: shoaibjameel123
7 Replies

4. Programming

modifying 2 or more values using method in Java

i wanna make modification in two or more identifiers,using a method in JAVA , but the only way to do this is by returning a value ,then also only one variable value can be modified .. one way 2 do do this is by use of array and passing its name in method ,, please suggest a different way of doing... (2 Replies)
Discussion started by: upvan111
2 Replies

5. Homework & Coursework Questions

Efficient Text File Writing

Use and complete the template provided. The entire template must be completed. If you don't, your post may be deleted! 1. The problem statement, all variables and given/known data: Write a template main.c file via shell script to make it easier for yourself later. The issue here isn't writing... (2 Replies)
Discussion started by: george3isme
2 Replies

6. Shell Programming and Scripting

Efficient population of array from text file

Hi, I am trying to populate an array with data from a text file. I have a working method using awk but it is too slow and inefficent. See below. The text file has 70,000 lines. As awk is a line editor it reads each line of the file until it gets to the required line and then processes it.... (3 Replies)
Discussion started by: carlr
3 Replies

7. Shell Programming and Scripting

Efficient method of determining if a string is in a file.

Hi, I was hoping someone could suggest an alternative to code I currently have as mine takes up far too much processor time and it to slow. The situation: I have a programme that runs on some files just before they are zipped up and archived, the program appends a one line summary of the... (4 Replies)
Discussion started by: RECrerar
4 Replies

8. Shell Programming and Scripting

Extract lines from text files

I have some files containing the following data # RESIDUE AA STRUCTURE BP1 BP2 ACC N-H-->O O-->H-N N-H-->O O-->H-N TCO KAPPA ALPHA PHI PSI X-CA Y-CA Z-CA 1 196 A M 0 0 230 0, 0.0 2,-0.2 0, 0.0 0, 0.0 0.000 360.0 360.0 360.0 76.4 21.7 -6.8 11.3 2 197 A D + 0 0 175 1,-0.1 2,-0.1 0, 0.0 0, 0.0... (10 Replies)
Discussion started by: edweena
10 Replies

9. UNIX for Dummies Questions & Answers

More efficient way to print variable values?

Hello, Through the process of a executing a shell script, I extract the values for a number of variables (arbitrarily declared as a through i) and towards the end I print them out like shown below: #!bin/sh # bits of code to get values for the variables ...... ...... # print the values... (4 Replies)
Discussion started by: Gussifinknottle
4 Replies

10. Shell Programming and Scripting

Portable and efficient way to add text after pattern

Shell: sh/bash OS: Linux (all unix flavors) Suppose i have a variable with this content: ArgZ=' import os import sys MySpecialpath = os.path.abspath(sys.argv) # ' ArgZB='#REGEN #REGEN #REGEN ' I want to add this text to a file/script, only under the following conditions: 1. ... (1 Reply)
Discussion started by: SkySmart
1 Replies

LEARN ABOUT X11R4

zgrep

ZGREP(1)                                                      General Commands Manual                                                     ZGREP(1)

NAME

       zgrep - search possibly compressed files for a regular expression

SYNOPSIS

       zgrep [ grep_options ] [ -e ] pattern filename...

DESCRIPTION

       Zgrep  invokes  grep  on  compressed  or  gzipped  files.   These  grep  options  will  cause  zgrep  to  terminate  with  an  error  code:
       (-[drRzZ]|--di*|--exc*|--inc*|--rec*|--nu*).  All other options specified are passed directly to grep.  If no file is specified,  then  the
       standard input is decompressed if necessary and fed to grep.  Otherwise the given files are uncompressed if necessary and fed to grep.

       If the GREP environment variable is set, zgrep uses it as the grep program to be invoked.

EXIT CODE

       2 - An option that is not supported was specified.

AUTHOR

       Charles Levert (charles@comm.polymtl.ca)

SEE ALSO

       grep(1), gzexe(1), gzip(1), zdiff(1), zforce(1), zmore(1), znew(1)

                                                                                                                                          ZGREP(1)
All times are GMT -4. The time now is 11:16 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy