Visit Our UNIX and Linux User Community


Enhance existing script: Extract Multiple variables & Input in an echo string


 
Thread Tools Search this Thread
Top Forums Shell Programming and Scripting Enhance existing script: Extract Multiple variables & Input in an echo string
# 1  
Old 12-18-2019
Enhance existing script: Extract Multiple variables & Input in an echo string

Hi Experts

I need your help to optimize my script to execute better as I have nearly 1M records & the script is taking close to 40 minutes to execute, so would need support on a faster alternative.

Input: file
Code:
{"house":"1024","zip":"2345","city":"asd","country":"zzv"}
{"city":"asd","house":"1024","zip":"2845","country":"zzv"}
{"house":"1028","zip":"2645","city":"asd","country":"zzv"}
{"zip":"2545","house":"1021","city":"asd","country":"zzv"}
{"city":"asd","house":"1020","zip":"2345","country":"zzv"}

Script:
Code:
for i in `cat file`
do
	HNO=`echo $i | awk -F"\"house\":" '{print $2}'| cut -d"," -f1 | sed 's/\"//g'`
	ZIP=`echo $i | awk -F"\"zip\":" '{print $2}'| cut -d"," -f1 | sed 's/\"//g'`
	CIT=`echo $i | awk -F"\"city\":" '{print $2}'| cut -d"," -f1 | sed 's/\"//g'`
	echo 'House number is '$HNO', Zip Code is '$ZIP', City is '$CIT' and Country is zzv'
done

Output:
Code:
House number is 1024, Zip Code is 2345, City is asd and Country is zzv
House number is 1024, Zip Code is 2845, City is asd and Country is zzv
House number is 1028, Zip Code is 2645, City is asd and Country is zzv
House number is 1021, Zip Code is 2545, City is asd and Country is zzv
House number is 1020, Zip Code is 2345, City is asd and Country is zzv

Thanks for your help in advance
Regards
Nk
# 2  
Old 12-19-2019
Hi, the reason that the script is so slow, is that it is firing off 12 different sub shells per line in the input file, so that is 12 million sub shells in total, which is quite costly.
Examples of sub shells in the script are:
  • awk -F"\"house\":" '{print $2}'
  • cut -d"," -f1
  • sed 's/\"//g'
  • `echo $i | awk -F"\"house\":" '{print $2}'| cut -d"," -f1 | sed 's/\"//g'
  • awk -F"\"zip\":" '{print $2}'
  • etcetera




Try eliminating those sub shells, by using a JSON parser:
Code:
jq -r '"House number is " + .house + ", Zip Code is " + .zip + ", City is " + .city + " and Country is " + .country' file

Code:
House number is 1024, Zip Code is 2345, City is asd and Country is zzv
House number is 1024, Zip Code is 2845, City is asd and Country is zzv
House number is 1028, Zip Code is 2645, City is asd and Country is zzv
House number is 1021, Zip Code is 2545, City is asd and Country is zzv
House number is 1020, Zip Code is 2345, City is asd and Country is zzv


--
If you do not have a json parser you could try using a single script without sub shells for this particular sample:
Code:
awk -F\" '
  {
    split(x,F)
    for(i=2; i<=NF; i+=4)
      F[$i]=$(i+2)
    printf "House number is %s, Zip Code is %s, City is %s and Country is %s\n", F["house"], F["zip"], F["city"], F["country"]
  }
' file


Last edited by Scrutinizer; 12-19-2019 at 07:43 PM..
These 3 Users Gave Thanks to Scrutinizer For This Post:
# 3  
Old 12-19-2019
Thanks for your quick help.

It works absolutely fine & the execution Time is improved significantly.

Regards
Nk

Previous Thread | Next Thread
Test Your Knowledge in Computers #298
Difficulty: Easy
Linux Mint actively develops software for its operating system. Most of the development is done in Javascript and the source code is available on GitHub.
True or False?

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Extract multiple values into corresponding variables

Here is my input # MANIFEST.MF Manifest-Version: 1.0 Build-Jdk: 1.6.0 Built-By: CM_TEAM Build_SvnRev: 662789 Build_Number: 13.0.0.0-JDK8 Build_Date: Wed 04/05/2017-20:48:19.17 Archiver-Version: Plexus Archiver Created-By: Apache Maven 3.1.0 Here is the expected output:... (4 Replies)
Discussion started by: kchinnam
4 Replies

2. Shell Programming and Scripting

How to read each line from input file, assign variables, and echo to output file?

I've got a file that looks like this (spaces before first entries intentional): 12345650-000005000GL140227 ANNUAL HELC FEE EN 22345650-000005000GL140227 ANNUAL HELC FEE EN 32345650-000005000GL140227 ANNUAL HELC FEE EN I want to read through the file line by line,... (6 Replies)
Discussion started by: Scottie1954
6 Replies

3. Shell Programming and Scripting

Pattern Match & Extract from a string

Hi, I have long string in 2nd field, as shown below: REF1 | CLESCLJSCSHSCSMSCSNSCSRSCUDSCUFSCU7SCV1SCWPSCXGPDBACAPA0DHDPDMESED6 REF2 | SBR4PCBFPCDRSCSCG3SCHEBSCKNSCKPSCLLSCMCZXTNPCVFPCV6P4KL0DMDSDSASEWG I have a group of fixed patterns which can occur in these long strings & only... (11 Replies)
Discussion started by: karumudi7
11 Replies

4. Shell Programming and Scripting

Script to find & replace a multiple lines string across multiple php files and subdirectories

Hey guys. I know pratically 0 about Linux, so could anyone please give me instructions on how to accomplish this ? The distro is RedHat 4.1.2 and i need to find and replace a multiple lines string in several php files across subdirectories. So lets say im at root/dir1/dir2/ , when i execute... (12 Replies)
Discussion started by: spfc_dmt
12 Replies

5. Shell Programming and Scripting

awk + gsub to search multiple input values & replace with located string + extra text

Hi all. I have the following command that is successfully searching for any one of the strings on all lines of a file and replacing it with the instructed value. cat inputFile | awk '{gsub(/aaa|bbb|ccc|ddd/,"1234")}1' > outputFile This does in fact replace any occurrence of aaa, bbb,... (2 Replies)
Discussion started by: dazhoop
2 Replies

6. Shell Programming and Scripting

Multiple Variables in Array from Existing Directories

I would like to extract directories from a specific place and keep them into an array of variables to run functions into it. Example, bash-3.00$ls adrian bryan caren derrick I want to keep each directory names into a variable adrian --> document bryan --> document caren --> document... (3 Replies)
Discussion started by: lynxlee
3 Replies

7. Shell Programming and Scripting

Tailing new log file & echo the string on console

Guys, I do have a script that runs to take the server out from network, after running the script it is writing the new log file{outFile} in to directory . Now what i need is my script should tail the last modified file{outFile} & search the string {Server Status} ans should echo the same at the... (0 Replies)
Discussion started by: raghunsi
0 Replies

8. Shell Programming and Scripting

Find & Replace string in multiple files & folders using perl

find . -type f -name "*.sql" -print|xargs perl -i -pe 's/pattern/replaced/g' this is simple logic to find and replace in multiple files & folders Hope this helps. Thanks Zaheer (0 Replies)
Discussion started by: Zaheer.mic
0 Replies

9. Shell Programming and Scripting

Extract numbers from a string and store in variables

Hi All, Is it possible in Unix shell script to extract numbers from a string containing ".", such as; 5.2.314 And store in variables so; var1 = 5 var2 = 2 var3 = 314 Thanks in advance for any help anyone can provide dave (6 Replies)
Discussion started by: davewg
6 Replies

10. UNIX for Dummies Questions & Answers

extract from string variable into new variables

I have a variable which consists of a string like this: 001 aaabc 44 a bbb12 How do I extract each substring, delimited by the spaces, into new variables - one for each substring? eg var1 will be 001, var2 will be aaabc, var3 will be 44, var4 will be a, etc? I've come up with this:... (2 Replies)
Discussion started by: Sniper Pixie
2 Replies

Featured Tech Videos