Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Speeding up shell script with grep
Top Forums Shell Programming and Scripting Speeding up shell script with grep Post 302994602 by dunryc on Saturday 25th of March 2017 04:37:32 AM
Old 03-25-2017
Speeding up shell script with grep
HI Guys hoping some one can help

I have two files on both containing uk phone numbers

master is a file which has been collated over a few years ad currently contains around 4 million numbers

new is a file which also contains 4 million number i need to split new nto two separate files one which contains number that already exist in master and one which contains numbers that don't already exist in master. I can do this but it takes around 80 hours to complete !! can any one offer any suggestions on how to speed this up ?

Code:
while read -r phone_number; do

        echo  "checking master for phone number $phone_number"

        if grep "$phone_number" master.csv; then
  
           echo "$phone_number already exists in master file "
          echo "$phone_number" >> unusable_numbers    
        else

           echo "$phone_number looks good we can use this saving to usable_numbers"
           echo "$phone_number" >> usable_numbers 

        fi

done < new

any help would be gratefully appreciated
Last edited by RudiC; 03-25-2017 at 06:58 AM..
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

grep in Shell script

Hello I do want to write a script which will check any errors say "-error" in the log file then have to send email to the concern person . And the concern person will correct the error . Next time if the script runs eventhough the error has been corrected it will ... (1 Reply)
Discussion started by: Krishnaramjis
1 Replies

2. UNIX for Dummies Questions & Answers

Speeding up a Shell Script (find, grep and a for loop)

Hi all, I'm having some trouble with a shell script that I have put together to search our web pages for links to PDFs. The first thing I did was: ls -R | grep .pdf > /tmp/dave_pdfs.outWhich generates a list of all of the PDFs on the server. For the sake of arguement, say it looks like... (8 Replies)
Discussion started by: Dave Stockdale
8 Replies

3. Shell Programming and Scripting

grep'ing and sed'ing chunks in bash... need help on speeding up a log parser.

I have a file that is 20 - 80+ MB in size that is a certain type of log file. It logs one of our processes and this process is multi-threaded. Therefore the log file is kind of a mess. Here's an example: The logfile looks like: "DATE TIME - THREAD ID - Details", and a new file is created... (4 Replies)
Discussion started by: elinenbe
4 Replies

4. Shell Programming and Scripting

How to grep sql error in shell script and exit the script?

I need help in the following script. I want to grep the sql errors insert into the error table and exit the shell script if there is any error, otherwise keep running the scripts. Here is my script #!/bin/csh -f source .orapass set user = $USER set pass = $PASS cd /opt/data/scripts echo... (2 Replies)
Discussion started by: allinshell99
2 Replies

5. Shell Programming and Scripting

Shell Script with Grep

Hi guys - below is my script that is checking for current file, size and timestamp. However I added a "grep" feature in it (line in red), but not getting the desired result. I am trying to acheive in output: 1. Show me the file name, timestamp, size and grep'ed words It would be a... (2 Replies)
Discussion started by: DallasT
2 Replies

6. UNIX for Dummies Questions & Answers

Speeding/Optimizing GREP search on CSV files

Hi all, I have problem with searching hundreds of CSV files, the problem is that search is lasting too long (over 5min). Csv files are "," delimited, and have 30 fields each line, but I always grep same 4 fields - so is there a way to grep just those 4 fields to speed-up search. Example:... (11 Replies)
Discussion started by: Whit3H0rse
11 Replies

7. Shell Programming and Scripting

speeding up bash script with "while read line"

Hello everybody, I'm still slowly treading my way into bash scripting (without any prior programming experience) and hence my code is mostly what some might call "creative" if they meant well :D I have created a script that serves its purpose but it does so very slowly, since it needs to work... (4 Replies)
Discussion started by: origamisven
4 Replies

8. Shell Programming and Scripting

Help speeding up script

This is my first experience writing unix script. I've created the following script. It does what I want it to do, but I need it to be a lot faster. Is there any way to speed it up? cat 'Tax_Provision_Sample.dat' | sort | while read p; do fn=`echo $p|cut -d~ -f2,4,3,8,9`; echo $p >> "$fn.txt";... (20 Replies)
Discussion started by: JohnN6
20 Replies

9. Shell Programming and Scripting

Help 'speeding' up this 'parsing' script - taking 24+ hours to run

Hi, I've written a ksh script that read a file and parse/filter/format each line. The script runs as expected but it runs for 24+ hours for a file that has 2million lines. And sometimes, the input file has 10million lines which means it can be running for more than 2 days and still not finish.... (9 Replies)
Discussion started by: newbie_01
9 Replies

10. Shell Programming and Scripting

Help with speeding up my working script to take less time - how to use more CPU usage for a script

Hello experts, we have input files with 700K lines each (one generated for every hour). and we need to convert them as below and move them to another directory once. Sample INPUT:- # cat test1 1559205600000,8474,NormalizedPortInfo,PctDiscards,0.0,Interface,BG-CTA-AX1.test.com,Vl111... (7 Replies)
Discussion started by: prvnrk
7 Replies

LEARN ABOUT ULTRIX

kdb_init

kdb_init(8krb)															    kdb_init(8krb)

Name
       kdb_init - initialize the Kerberos master database

Syntax
       /var/dss/kerberos/bin/kdb_init [ realm-name ] [ database-name ]

Arguments
       realm-name   The realm of the Kerberos database.

       database-name
		    A database specified so that the current and files are not overwritten.

Description
       The  utility  creates  and  initializes	the Kerberos master database.  The utility creates the database files: and It also initializes the
       database by adding three database entries: the master database principal, a Kerberos default principal, the ticket-granting service princi-
       pal ( and the password changing principal,

       The master database principal is the entry to the database itself.  You cannot use or modify the database without the master database pass-
       word.  The Kerberos default principal provides a template for service principals.

       The ticket-granting service, is used by Kerberos principals to obtain tickets to communicate with other Kerberos principals.  The password-
       changing principal is not used.

       If  realm-name  is  omitted  when  you enter the command, prompts for it. The program also prompts for the master database key.	You cannot
       manipulate the database without this key.

       By using database-name, you can create another database to prevent the current and files from being overwritten.

       After using to set up the master database, you may want to use the utility to hide the  master  database  password  on  the  database  host
       machine.  This  enables	Kerberos  administration programs to access and manipulate the master database, without needing the password to be
       entered manually.

Files
       See Also
	      kdb_util(8krb), kstash(8krb), kdb_edit(8krb), kdb_destroy(8krb)

																    kdb_init(8krb)
All times are GMT -4. The time now is 11:04 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy