Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: Incredibly inefficient cat | grep script
Top Forums Shell Programming and Scripting Incredibly inefficient cat | grep script Post 302913160 by MadeInGermany on Thursday 14th of August 2014 07:31:24 AM
Old 08-14-2014
With your new information, you can speed up Corona's perl solutions with awk:
Code:
pipesplit.pl -l 5000 awk 'FILENAME=="-" {A[$1]; next} ($1 in A)' - FS="," production_list.csv < accurate_list.csv

The hash lookups are faster than an RE that can match anywhere in the lines.
Also, increasing the 5000 by a factor 10 will increase speed (and memory consumption) by the same factor.
BTW I find the following order more intuitive:
Code:
< accurate_list.csv pipesplit.pl -l 5000 awk 'FILENAME=="-" {A[$1]; next} ($1 in A)' - FS="," production_list.csv

Last edited by MadeInGermany; 08-14-2014 at 08:40 AM.. Reason: added the missing FS=","
This User Gave Thanks to MadeInGermany For This Post:
Corona688
 

10 More Discussions You Might Find Interesting

1. UNIX for Advanced & Expert Users

cat and grep not working

I am trying to cat a file and then grep that file for a number. I can do it fine on other files but this particular file will not do anything. I tried running it on an older file from the same device but it is just not working. The file is nothing more than a flat file on a unix box. Here is just a... (3 Replies)
Discussion started by: jphess
3 Replies

2. Shell Programming and Scripting

cat in the command line doesn't match cat in the script

Hello, So I sorted my file as I was supposed to: sort -n -r -k 2 -k 1 file1 | uniq > file2 and when I wrote > cat file2 in the command line, I got what I was expecting, but in the script itself ... sort -n -r -k 2 -k 1 averages | uniq > temp cat file2 It wrote a whole... (21 Replies)
Discussion started by: shira
21 Replies

3. Shell Programming and Scripting

Perl sum really inefficient!!

Hi all, I have a file like the following: ID, 2,Andrew,0,1,2,3,4,2,5,6,7,7,9,3,4,5,34,3,2,1,5,6,78,89,8,7,6...................... 4,James,0,6,7,0,5,6,4,7,8,9,6,46,6,3,2,5,6,87,0,341,0,5,2,5,6.................... END, (there are more entires on each line but to keep it simple I've left... (10 Replies)
Discussion started by: Donkey25
10 Replies

4. Shell Programming and Scripting

Problem with IF - CAT - GREP in simple shell script

Hi all, Here is my requirement I have to search 'ORA' word in out.log file,if it is present then i need to send that file (out.log) content to some mail id.If 'ORA' word is not in that file then i need to send 'load succesful' message to some mail id. The below the shell script is not... (5 Replies)
Discussion started by: mak_boop
5 Replies

5. Shell Programming and Scripting

cat /etc/passwd and grep -v on /etc/shells

Hi All, I'd like to do this cat /etc/passwd and grep -v on the /etc/shells list I'd like to find all shell that doesn't exist on the /etc/passwd. Is there an easy way without doing a egrep -v "/bin/sh|/bin/bash................"? How do I use a file /etc/shells as my list for... (4 Replies)
Discussion started by: itik
4 Replies

6. Shell Programming and Scripting

cat -n and grep

I am not sure if using cat -n is the most efficient way to split a file into multiple files, one file per line in the source file. I thought using cat -n would make it easy to process the file because it produces an output that numbers each line that I could then grep for with the regex "^ *$i".... (3 Replies)
Discussion started by: kapu
3 Replies

7. Shell Programming and Scripting

grep or cat using sed

Is there a way using grep or cat a file to create a new file based on whether the first 9 positions of each record is less than 399999999? This is a fixed file format. (3 Replies)
Discussion started by: ski
3 Replies

8. UNIX for Dummies Questions & Answers

Grep and cat combined

Hello, i need to search one word (snp1) from many files and copy the content of the columns of this word in new file. example: file 1: SNP BP CHR P snp1 1 3 0.01 snp2 2 2 0.05 . . file 2: SNP BP CHR P snp1 1 3 0.06 snp2 2 2 0.3 output... (6 Replies)
Discussion started by: biopsy
6 Replies

9. Shell Programming and Scripting

Replace cat and grep with <

Hello someone told me to use OS=`awk '{print int($3)}' < /etc/redhat-release` instead of OS=cat /etc/redhat-release | `awk '{print int($3)}'` any idea for the reason ? (5 Replies)
Discussion started by: nimafire
5 Replies

10. UNIX for Dummies Questions & Answers

Grep or cat The Whole Directory PROBLEMS :(

Hi Guys This is my first post so I am not sure how things go here. I'm sorry if I'm breaking the rule or something. Feel free to correct me about that :) So as I was saying... I'd been trying to grep this folder containing 900,000 txt files but seems no luck. I get either "No such file... (6 Replies)
Discussion started by: Nexeu
6 Replies

LEARN ABOUT CENTOS

powertop

POWERTOP(8)							  powertop manual						       POWERTOP(8)

NAME

       powertop - a power consumption and power management diagnosis tool.

SYNOPSIS

       powertop [ options ]

DESCRIPTION

       powertop  is  a	program that helps to diagnose various issues with power consumption and power management. It also has an interactive mode
       allowing one to experiment with various power management settings. When invoking powertop without arguments powertop starts in  interactive
       mode.

OPTIONS

       --calibrate
	      runs  powertop  in  calibration mode. When running on battery, powertop can track power consumption as well as system activity. When
	      there are enough measurements, powertop can start to report power estimates. One can get	more  accurate	estimates  by  using  this
	      option to enable a calibration cycle. This will cycle through various display levels and USB device activities and workloads.

       --csv[=FILENAME]
	      generate	a  CSV report. If a filename is not specified then the default name "powertop.csv" is used. The CSV report can be used for
	      reporting and data analysis.

       --debug
	       run in "debug" mode.

       --extech=devnode
	      use the Extech Power Analyzer for measurements. This allows one to specify the serial device node of the serial to USB adaptor  con-
	      necting to the Extech Power Analyzer, for example /dev/ttyUSB0.

       --help
	       show the help message.

       --html[=FILENAME]
	      generate	an  HTML report. If a filename is not specified then the default name "powertop.html" is used. The HTML report can be sent
	      to others to help diagnose power issues.

       --iteration[=iterations]
	      number of times to run each test.

       --workload[=workload]
	      file to execute for workload.

       --quiet
	       supress stderr output.

       --time[=seconds]
	      generate a report for a specified number of seconds.

       --version
	      print version information and exit.

BUGS

       Send bug reports to <powertop@lists.01.org>

SEE ALSO

       The program is more fully described at https://01.org/powertop

AUTHOR

       powertop was written by	Arjan  van  de	Ven  <arjan@linux.intel.com>,  and  is	currently  maintained  by  Chris  E  Ferron  <chris.e.fer-
       ron@linux.intel.com>.

Linux								   June 1, 2012 						       POWERTOP(8)
All times are GMT -4. The time now is 01:06 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy