Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Speeding up shell script with grep
Top Forums Shell Programming and Scripting Speeding up shell script with grep Post 302994609 by Don Cragun on Saturday 25th of March 2017 06:07:42 AM
Old 03-25-2017
In addition to what bakunin suggested, you might also consider the following to more closely match the output produced by your current script...

Making the wild assumptions that:
  1. master.csv is a character separated values file with comma as the character separating fields, and
  2. the field containing the phone number in master.csv is the 1st field
how long does the following script take:
Code:
awk -v fn=1 -F',' '
FNR == NR {
	p[$fn]
	next
}
{	print "checking master for phone number", $1
	if($1 in p) {
		print $1, "already exists in master file "
		print > "unusable_numbers"
	} else {
		print $1, "looks good we can use this saving to usable_numbers"
		print > "useable_numbers"
	}
}' master.csv new

to do the same job?

If the field number in master.csv is not the 1st field, change the value assigned to the fn variable from 1 to the field number of the field containing the phone number.

If the field separators in master.csv are not commas, change the character in the -F option-argument to the desired character.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

grep in Shell script

Hello I do want to write a script which will check any errors say "-error" in the log file then have to send email to the concern person . And the concern person will correct the error . Next time if the script runs eventhough the error has been corrected it will ... (1 Reply)
Discussion started by: Krishnaramjis
1 Replies

2. UNIX for Dummies Questions & Answers

Speeding up a Shell Script (find, grep and a for loop)

Hi all, I'm having some trouble with a shell script that I have put together to search our web pages for links to PDFs. The first thing I did was: ls -R | grep .pdf > /tmp/dave_pdfs.outWhich generates a list of all of the PDFs on the server. For the sake of arguement, say it looks like... (8 Replies)
Discussion started by: Dave Stockdale
8 Replies

3. Shell Programming and Scripting

grep'ing and sed'ing chunks in bash... need help on speeding up a log parser.

I have a file that is 20 - 80+ MB in size that is a certain type of log file. It logs one of our processes and this process is multi-threaded. Therefore the log file is kind of a mess. Here's an example: The logfile looks like: "DATE TIME - THREAD ID - Details", and a new file is created... (4 Replies)
Discussion started by: elinenbe
4 Replies

4. Shell Programming and Scripting

How to grep sql error in shell script and exit the script?

I need help in the following script. I want to grep the sql errors insert into the error table and exit the shell script if there is any error, otherwise keep running the scripts. Here is my script #!/bin/csh -f source .orapass set user = $USER set pass = $PASS cd /opt/data/scripts echo... (2 Replies)
Discussion started by: allinshell99
2 Replies

5. Shell Programming and Scripting

Shell Script with Grep

Hi guys - below is my script that is checking for current file, size and timestamp. However I added a "grep" feature in it (line in red), but not getting the desired result. I am trying to acheive in output: 1. Show me the file name, timestamp, size and grep'ed words It would be a... (2 Replies)
Discussion started by: DallasT
2 Replies

6. UNIX for Dummies Questions & Answers

Speeding/Optimizing GREP search on CSV files

Hi all, I have problem with searching hundreds of CSV files, the problem is that search is lasting too long (over 5min). Csv files are "," delimited, and have 30 fields each line, but I always grep same 4 fields - so is there a way to grep just those 4 fields to speed-up search. Example:... (11 Replies)
Discussion started by: Whit3H0rse
11 Replies

7. Shell Programming and Scripting

speeding up bash script with "while read line"

Hello everybody, I'm still slowly treading my way into bash scripting (without any prior programming experience) and hence my code is mostly what some might call "creative" if they meant well :D I have created a script that serves its purpose but it does so very slowly, since it needs to work... (4 Replies)
Discussion started by: origamisven
4 Replies

8. Shell Programming and Scripting

Help speeding up script

This is my first experience writing unix script. I've created the following script. It does what I want it to do, but I need it to be a lot faster. Is there any way to speed it up? cat 'Tax_Provision_Sample.dat' | sort | while read p; do fn=`echo $p|cut -d~ -f2,4,3,8,9`; echo $p >> "$fn.txt";... (20 Replies)
Discussion started by: JohnN6
20 Replies

9. Shell Programming and Scripting

Help 'speeding' up this 'parsing' script - taking 24+ hours to run

Hi, I've written a ksh script that read a file and parse/filter/format each line. The script runs as expected but it runs for 24+ hours for a file that has 2million lines. And sometimes, the input file has 10million lines which means it can be running for more than 2 days and still not finish.... (9 Replies)
Discussion started by: newbie_01
9 Replies

10. Shell Programming and Scripting

Help with speeding up my working script to take less time - how to use more CPU usage for a script

Hello experts, we have input files with 700K lines each (one generated for every hour). and we need to convert them as below and move them to another directory once. Sample INPUT:- # cat test1 1559205600000,8474,NormalizedPortInfo,PctDiscards,0.0,Interface,BG-CTA-AX1.test.com,Vl111... (7 Replies)
Discussion started by: prvnrk
7 Replies

LEARN ABOUT REDHAT

pilot-addresses

pilot-addresses(1)					      General Commands Manual						pilot-addresses(1)

NAME

       pilot-addresses	- read and write address book databases to and from a Palm handheld device, such as those made from Palm, Handspring, Han-
       dera, TRGPro, Sony or other Palm Compatible Handheld PDA device

SYNOPSIS

       pilot-addresses -p <port> [-c category ] [-d category ] [-r file | -w file ]

       (Note that some options are not shown above)

DESCRIPTION

       pilot-addresses allows the user to read all entries in the Palm address book database, write new entries into the database,  and  delete  a
       category or delete all entries in the database.

TARGET DEVICE

       The  default  serial  device  used to communicate with a Palm is /dev/pilot.  If the environment variable $PILOTPORT is set, its value will
       override the default. A serial device specified on the command-line will be used regardless of any $PILOTPORT setting.

OPTIONS

       Several options exist, including...

       -p --port <port>,
	      Use device file port to communicate with the Palm handheld device. If this is not specified, will look for the  $PILOTPORT  environ-
	      ment variable. If both are not found, will fall back to /dev/pilot.

       -h --help
	      Display help synopsis for pilot-addresses

       -v --version
	      Display version of pilot-addresses

       -a     Augments	fields in address book records with additional information.  The augmented information is placed before and separated from
	      the field with a semi-colon, (;).

	      Augmented information includes:
	      category_name - placed in front of each record or

	      ["Work" | "Home" | "Fax" | "Other" | "E-mail" | "Main" | "Pager" | "Mobile" ] - placed in front of each phone number field.

	      Empty fields are not augmented.

       -c category
	      Install records to category category by default. Normally pilot-addresses uses Unfiled as the default category. This option is over-
	      ridden by the category specified in the record with the -a option.

       -d category
	      Delete all records in the specified category before installing new records.

       -D     Delete all address book records in all categories. Obviously, be very careful with this one.

       -e     Escape  all  special  characters	with a backslash. This enables you to read and write entries with newline characters in a field or
	      note.

       -q     Causes pilot-addresses to be quiet and not prompt you to press the HotSync button.

       -r file
	      Reads records from file and install them to the Palm address book database. (Use the -w file  to	get  a	template  file	for  input
	      records.)

       -t delim
	      Include  category  in  each record, use the delimiter specified to separate all fields of a record. Delimiters are specified as fol-
	      lows: 3=tab, 2=;, 1=,.  This overrides the default delimiter of comma between fields and	semi-colon  between  a	field's  augmented
	      information. (Please note that this may generate confusing results when used with the -a option.)

       -T     Write a header line with field titles as the first line of the data file.

       -w file
	      Get all address book records from the Palm address book database and writes them into file

USAGE

       The  program will connect to a target device and port, prompt the user to HotSync, and perform the requested read or write operation speci-
       fied by the user.

EXAMPLES

       To write all address records in a Palm to the file addrbook.csv:
	      pilot-addresses -w addrbook.csv
       or
	      pilot-addresses -p /dev/irnine -w addrbook.csv

       To read the address book records in the file addrbook.csv and install them on a Palm:
	      pilot-addresses -r addrbook.csv

       To read the address book records in the file addrbook.csv and place them into the Palm address book database category Special  after  first
       deleting all current records in the Special category on the palm:
	      pilot-addresses -c Special -d Special -r addrbook.csv

SEE ALSO

       pilot-link(7)

KNOWN BUGS

       pilot-addresses has no known bugs.

REPORTING BUGS

       Report bugs at http://bugs.pilot-link.org/

AUTHOR

       pilot-addresses originally written by Kenneth Albanowski, manual page was written by Robert Wittig <bob.wittig@gt.org>.

Free Software Foundation				    Palm Computing Device Tools 					pilot-addresses(1)
All times are GMT -4. The time now is 09:35 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy