Sponsored Content
Top Forums Shell Programming and Scripting Search and replace multiple patterns in a particular column only - efficient script Post 302926759 by ongoto on Wednesday 26th of November 2014 10:15:54 PM
Old 11-26-2014
@ Chubler_XL
You are abolutely right. I was only using the provided data.

This might work for the situation you presented...
sed -i "$lines s/\b$kee2/$repl/g" trydata1
I'll do the edit.

But that still doesn't cure the 'other columns' bit, does it?

---------- Post updated at 07:15 PM ---------- Previous update was at 04:22 PM ----------

The theory here is to reduce disk reads by a margin.
Disk writes can't be helped unless one builds a file
in memory and writes it out all at once.
Bash speed just is what it is...and so is my skill set. Smilie

Your AWK example is 9 times faster on my machine! That's BIG!

Code:
#!/bin/bash

# Load both data files into memory
< trydata1 mapfile data1
< trydata2 mapfile data2
for f1 in ${data1[*]};
do
    lines=$(( $lines + 1 ))
    kee2=$(echo $f1 | cut -d, -f4)
    for val2 in ${data2[*]};
    do
        if [[ $val2 =~ ^$kee2 ]]; then
            repl=${val2##*,}
            sed -i "$lines s/\b$kee2/$repl/g" trydata1
        fi
    done
done
cat trydata1

# Using this data set with no column issues...
# ----------------------
# xyz,122,913,opq,876
# rst,956,921,xyz,012
# 456,890,903,rst,467
# 000,5xyz1,000,xyz,000
# 4567,5rst1,opq,xyz,000
# rst,5opq1,rst,02opq,000
# 000,5xyz1,xyz,opq,000

# real    0m0.028s
# user    0m0.008s
# sys    0m0.009s

# On a machine running @ 2GHz
# If my math is right...
# 0.028 seconds to process 7 lines
# time to process 30,000 lines ~ 2 minutes


Last edited by ongoto; 11-26-2014 at 11:37 PM..
 

10 More Discussions You Might Find Interesting

1. UNIX for Dummies Questions & Answers

multiple input search and replace script

hi, i want to create a script that will search and replace the values inside a particular file. i have 5 files that i need to change some values inside and i don't want to use vi to edit these files. All the inputted values on the script below will be passed into the files. cho "" echo... (3 Replies)
Discussion started by: tungaw2004
3 Replies

2. Shell Programming and Scripting

Complex Search/Replace Multiple Files Script Needed

I have a rather complicated search and replace I need to do among several dozen files and over a hundred occurrences. My site is written in PHP and throughout the old code, you will find things like die("Operation Aborted due to....."); For my new design skins for the site, I need to get... (2 Replies)
Discussion started by: UCCCC
2 Replies

3. UNIX for Dummies Questions & Answers

replace multiple patterns in a string/filename

This should be somewhat simple, but I need some help with this one. I have a bunch of files with tags on the end like so... Filename {tag1}.ext Filename2 {tag1} {tag2}.ext I want to hold in a variable just the filename with all the " {tag}" removed. The tag can be anything so I'm looking... (4 Replies)
Discussion started by: kerppz
4 Replies

4. Shell Programming and Scripting

Search multiple patterns in multiple files

Hi, I have to write one script that has to search a list of numbers in certain zipped files. For eg. one file file1.txt contains the numbers. File1.txt contains 5,00,000 numbers and I have to search each number in zipped files(The number of zipped files are around 1000 each file is 5 MB) I have... (10 Replies)
Discussion started by: vsachan
10 Replies

5. Shell Programming and Scripting

search multiple patterns

I have two lists in a file that look like a b b a e f c d f e d c I would like a final list a b c d e f I've tried multiple grep and awk but can't get it to work (8 Replies)
Discussion started by: godzilla07
8 Replies

6. Shell Programming and Scripting

How to search Multiple patterns in unix

Hi, I tried to search multiple pattern using awk trans=1234 reason=LN MISMATCH rec=`awk '/$trans/ && /'"$reason"'/' file` whenevr i tried to run on command promt it is executing but when i tried to implment same logic in shell script,it is failing i.e $rec is empty ... (6 Replies)
Discussion started by: ns64110
6 Replies

7. Shell Programming and Scripting

a column containing multiple patterns perl

If U have a question if a file is 33 ABC 276 LRR pir UJU 45 BCD 777 HIGH pred IJJ 67 BGH 66 LRR_1 prcc KIK 77 GYH 88 LOW pol KKK perl -lne '$a++ if /LRR/,/LOW/, /HIGH/; END {print $a+0}' (2 Replies)
Discussion started by: cdfd123
2 Replies

8. Shell Programming and Scripting

Search patterns in multiple logs parallelly.

Hi All, I am starting a service which will redirect its out put into 2 logs say A and B. Now for succesful startup of the service i need to search pattern1 in log A and pattern2 in log B which are writen continuosly. Now my requirement is to find the patterns in the increasing logs A and B... (19 Replies)
Discussion started by: Girish19
19 Replies

9. Shell Programming and Scripting

Replace multiple patterns together with retaining the text in between

Hi Team I have the following text in one of the file j1738-abc-system_id(in.value1)-2838 G566-deF-system_id(in.value2)-7489 I want to remove system_id(...) combination completely The output should look like this j1738-abc-in.value1-2838 G566-deF-in.value2-7489 Any help is appreciated... (4 Replies)
Discussion started by: Thierry Henry
4 Replies

10. Shell Programming and Scripting

Search Multiple patterns and display

Hi, I have scenario like below and need to search for multiple patterns Eg: Test Time Started= secs Time Ended = secc Green test Test Time Started= secs Time Ended = secc Green test Output: I need to display the text starting with Test and starting with Time... (2 Replies)
Discussion started by: weknowd
2 Replies
DB2_FIELD_TYPE(3)							 1							 DB2_FIELD_TYPE(3)

db2_field_type - Returns the data type of the indicated column in a result set

SYNOPSIS
string db2_field_type (resource $stmt, mixed $column) DESCRIPTION
Returns the data type of the indicated column in a result set. PARAMETERS
o $stmt - Specifies a statement resource containing a result set. o $column - Specifies the column in the result set. This can either be an integer representing the 0-indexed position of the column, or a string containing the name of the column. RETURN VALUES
Returns a string containing the defined data type of the specified column. If the specified column does not exist in the result set, db2_field_type(3) returns FALSE. SEE ALSO
db2_field_display_size(3), db2_field_name(3), db2_field_num(3), db2_field_precision(3), db2_field_scale(3), db2_field_width(3). PHP Documentation Group DB2_FIELD_TYPE(3)
All times are GMT -4. The time now is 07:58 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy