Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Speeding up substitutions
Top Forums Shell Programming and Scripting Speeding up substitutions Post 302921029 by senhia83 on Tuesday 14th of October 2014 11:20:08 AM
Old 10-14-2014
Speeding up substitutions
Hi all,

I have a lookup table from which I am looking up values (from col1) and replacing them by corresponding values (from col2) in another file.

Code:
 
lookup file
 
a,b
c,d

So just replace a by b, and replace c by d.

Code:
 
mainfile
 
a,fvvgeggsegg,dvs
a,fgeggefddddddddddg
b,dfqefeff,dfs
b,fdfsfgggddgg
c,sddddd

Code:
 
output
 
b,fvvgeggsegg,dvs
b,fgeggefddddddddddg
b,dfqefeff,dfs
b,fdfsfgggddgg
d,sddddd

My working slow code

Code:
 
 
awk -F',' 'NR==FNR{a[$1]=$2;next}{ for (i in a) gsub(i,a[i])}1'  lookup mainfile

I need to speed this up , as the mainfile is 5GB and it is taking forever.
Right now the code is replacing all occurrences of the lookup strings in the mainfile, but actually I need to do that for only the first column,,is there a way to speed this up?

thanks a lot , I`m using the latest cygwin with 24 gigs of memory.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Speeding up processing a file

Hi guys, I'm hoping you can help me here. I've knocked up a script that looks at a (huge) log file, and pulls from each line the hour of each transaction and how long each transaction took. The data is stored sequentially as: 07:01 blah blah blah 12456 blah 07:03 blah blah blah 234 blah... (4 Replies)
Discussion started by: dlam
4 Replies

2. Shell Programming and Scripting

Perl - nested substitutions

How can I nest substitutions ? My solution just seems cheap ... sample data Cisco Catalyst Operating System Software, Version 235.5(18) Cisco Catalyst Operating System Software, Version 17.6(7) Cisco Catalyst Operating System Software, Version 19.6(7) Cisco Catalyst Operating System... (1 Reply)
Discussion started by: popeye
1 Replies

3. Shell Programming and Scripting

Multiple variable substitutions

Is there anyway to accomplish this? (ksh) FILES_TO_PROCESS='NAME1 NAME2' SOURCE_NAME1=/tmp/myfile TARGET_NAME1=/somewhere/else # other file names for i in $FILES_TO_PROCESS do file1=SOURCE_$i file2=TARGET_$i echo cp ${$file1} ${$file2} <-- how do get this to work. done (2 Replies)
Discussion started by: koondog
2 Replies

4. Shell Programming and Scripting

arrays and substitutions

I am working on a bash script and ran around this issue. here's the code : #!/bin/bash string="\"bin\" \"barn\" \"bin, barn /\"" array=($string) echo -e "\nMethod 1\narray is ---> ${array}" echo -e "array=($string)" array=("bin" "barn" "bin, barn /") echo -e "\nMethod 2\narray is... (4 Replies)
Discussion started by: titou_dude
4 Replies

5. Shell Programming and Scripting

Two substitutions in one echo

PHOST1=temp i=1 I want to display the value of PHOST1 by making use of variable i inplace of 1 something like this echo "$PHOST$i" # -> This doesn't seem to work. Please provide me the correct syntax. I tried many different ways echo ${PHOST${i}} echo ${PHOST Nothing seems... (6 Replies)
Discussion started by: blazer789
6 Replies

6. Shell Programming and Scripting

Speeding up search and replace in a for loop

Hello, I am using sed in a for loop to replace text in a 100MB file. I have about 55,000 entries to convert in a csv file with two entries per line. The following script works to search file.txt for the first field from conversion.csv and then replace it with the second field. While it works fine,... (15 Replies)
Discussion started by: pbluescript
15 Replies

7. Shell Programming and Scripting

How can I write nested command substitutions?

Hello How can write the nested command substitutions? echo `expr substr $x 1 expr ${#x} - 1` the above code is not working! Thanks in advance Regards Chetanz (5 Replies)
Discussion started by: Chetanz
5 Replies

8. UNIX for Dummies Questions & Answers

Multiple substitutions in one expression using sed

Hi, I'm trying to get multiple substitutions in one expression using sed: echo "-foo-_-bar--foo-_bar_-_foo_bar_-foo_-_bar_-" | sed -e "s//-/g" So, as you can see I'm trying to replace all instances of _-, -_, -- with - (dash) I have provided bad example. The question is how to use multiple... (6 Replies)
Discussion started by: useretail
6 Replies

9. Shell Programming and Scripting

Help speeding up script

This is my first experience writing unix script. I've created the following script. It does what I want it to do, but I need it to be a lot faster. Is there any way to speed it up? cat 'Tax_Provision_Sample.dat' | sort | while read p; do fn=`echo $p|cut -d~ -f2,4,3,8,9`; echo $p >> "$fn.txt";... (20 Replies)
Discussion started by: JohnN6
20 Replies

10. Shell Programming and Scripting

Speeding up shell script with grep

HI Guys hoping some one can help I have two files on both containing uk phone numbers master is a file which has been collated over a few years ad currently contains around 4 million numbers new is a file which also contains 4 million number i need to split new nto two separate files... (4 Replies)
Discussion started by: dunryc
4 Replies

LEARN ABOUT DEBIAN

locale::currency

Locale::Currency(3perl) 				 Perl Programmers Reference Guide				   Locale::Currency(3perl)

NAME

       Locale::Currency - standard codes for currency identification

SYNOPSIS

	   use Locale::Currency;

	   $curr = code2currency('usd');     # $curr gets 'US Dollar'
	   $code = currency2code('Euro');    # $code gets 'eur'

	   @codes   = all_currency_codes();
	   @names   = all_currency_names();

DESCRIPTION

       The "Locale::Currency" module provides access to standard codes used for identifying currencies and funds, such as those defined in ISO
       4217.

       Most of the routines take an optional additional argument which specifies the code set to use. If not specified, the default ISO 4217
       three-letter codes will be used.

SUPPORTED CODE SETS

       There are several different code sets you can use for identifying currencies. The ones currently supported are:

       alpha
	   This is a set of three-letter (uppercase) codes from ISO 4217 such as EUR for Euro.

	   Two of the codes specified by the standard (XTS which is reserved for testing purposes and XXX which is for transactions where no
	   currency is involved) are omitted.

	   This code set is identified with the symbol "LOCALE_CURR_ALPHA".

	   This is the default code set.

       num This is the set of three-digit numeric codes from ISO 4217.

	   This code set is identified with the symbol "LOCALE_CURR_NUMERIC".

ROUTINES

       code2currency ( CODE [,CODESET] )
       currency2code ( NAME [,CODESET] )
       currency_code2code ( CODE ,CODESET ,CODESET2 )
       all_currency_codes ( [CODESET] )
       all_currency_names ( [CODESET] )
       Locale::Currency::rename_currency  ( CODE ,NEW_NAME [,CODESET] )
       Locale::Currency::add_currency  ( CODE ,NAME [,CODESET] )
       Locale::Currency::delete_currency  ( CODE [,CODESET] )
       Locale::Currency::add_currency_alias  ( NAME ,NEW_NAME )
       Locale::Currency::delete_currency_alias	( NAME )
       Locale::Currency::rename_currency_code  ( CODE ,NEW_CODE [,CODESET] )
       Locale::Currency::add_currency_code_alias  ( CODE ,NEW_CODE [,CODESET] )
       Locale::Currency::delete_currency_code_alias  ( CODE [,CODESET] )
	   These routines are all documented in the Locale::Codes man page.

SEE ALSO

       Locale::Codes
       Locale::Constants
       http://www.iso.org/iso/support/currency_codes_list-1.htm
	   The ISO 4217 data.

AUTHOR

       See Locale::Codes for full author history.

       Currently maintained by Sullivan Beck (sbeck@cpan.org).

COPYRIGHT

	  Copyright (c) 1997-2001 Canon Research Centre Europe (CRE).
	  Copyright (c) 2001	  Michael Hennecke
	  Copyright (c) 2001-2010 Neil Bowers
	  Copyright (c) 2010-2011 Sullivan Beck

       This module is free software; you can redistribute it and/or modify it under the same terms as Perl itself.

perl v5.14.2							    2011-09-26						   Locale::Currency(3perl)
All times are GMT -4. The time now is 12:59 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy