Using grep -f works, but with two files as large as indicated will take its (serious) time, and may eventually run out of memory. Try
Code:
sort master.csv new | uniq -d
, then use the resultant file in similar way (uniq -u) to extract unique values from either original file.
Comparison of both approaches on ~20k files:
Code:
time grep -ffile2 file1
real 0m0.352s
user 0m0.280s
sys 0m0.052s
time sort file[12] | uniq -d
real 0m0.037s
user 0m0.032s
sys 0m0.004s
EDIT: Times spent for two files with roughly 4E6 entries each, and about 1E6 lines overlap (on a two processor linux host):
Code:
time sort file[12] | uniq -d > f1
real 0m14.975s
user 0m27.048s
sys 0m0.792s
time sort file1 f1 | uniq -u > f2
real 0m9.331s
user 0m16.488s
sys 0m0.572s
Hello
I do want to write a script which will check any errors say "-error" in the log file then have to send email to the concern person . And the concern person will correct the error .
Next time if the script runs eventhough the error has been corrected it will ... (1 Reply)
Hi all,
I'm having some trouble with a shell script that I have put together to search our web pages for links to PDFs.
The first thing I did was:
ls -R | grep .pdf > /tmp/dave_pdfs.outWhich generates a list of all of the PDFs on the server. For the sake of arguement, say it looks like... (8 Replies)
I have a file that is 20 - 80+ MB in size that is a certain type of log file.
It logs one of our processes and this process is multi-threaded. Therefore the log file is kind of a mess. Here's an example:
The logfile looks like: "DATE TIME - THREAD ID - Details", and a new file is created... (4 Replies)
I need help in the following script. I want to grep the sql errors insert into the error table and exit the shell script if there is any error, otherwise keep running the scripts.
Here is my script
#!/bin/csh -f
source .orapass
set user = $USER
set pass = $PASS
cd /opt/data/scripts
echo... (2 Replies)
Hi guys - below is my script that is checking for current file, size and timestamp.
However I added a "grep" feature in it (line in red), but not getting the desired result.
I am trying to acheive in output:
1. Show me the file name, timestamp, size and grep'ed words
It would be a... (2 Replies)
Hi all,
I have problem with searching hundreds of CSV files, the problem is that search is lasting too long (over 5min).
Csv files are "," delimited, and have 30 fields each line, but I always grep same 4 fields - so is there a way to grep just those 4 fields to speed-up search.
Example:... (11 Replies)
Hello everybody,
I'm still slowly treading my way into bash scripting (without any prior programming experience) and hence my code is mostly what some might call "creative" if they meant well :D
I have created a script that serves its purpose but it does so very slowly, since it needs to work... (4 Replies)
This is my first experience writing unix script. I've created the following script. It does what I want it to do, but I need it to be a lot faster. Is there any way to speed it up?
cat 'Tax_Provision_Sample.dat' | sort | while read p; do fn=`echo $p|cut -d~ -f2,4,3,8,9`; echo $p >> "$fn.txt";... (20 Replies)
Hi,
I've written a ksh script that read a file and parse/filter/format each line. The script runs as expected but it runs for 24+ hours for a file that has 2million lines. And sometimes, the input file has 10million lines which means it can be running for more than 2 days and still not finish.... (9 Replies)
Hello experts,
we have input files with 700K lines each (one generated for every hour). and we need to convert them as below and move them to another directory once.
Sample INPUT:-
# cat test1
1559205600000,8474,NormalizedPortInfo,PctDiscards,0.0,Interface,BG-CTA-AX1.test.com,Vl111... (7 Replies)
Discussion started by: prvnrk
7 Replies
LEARN ABOUT LINUX
npm-run-script
NPM-RUN-SCRIPT(1)NPM-RUN-SCRIPT(1)NAME
npm-run-script - Run arbitrary package scripts
SYNOPSIS
npm run-script <command> [--silent] [-- <args>...]
alias: npm run
DESCRIPTION
This runs an arbitrary command from a package's "scripts" object. If no "command" is provided, it will list the available scripts.
run[-script] is used by the test, start, restart, and stop commands, but can be called directly, as well. When the scripts in the package
are printed out, they're separated into lifecycle (test, start, restart) and directly-run scripts.
As of ` https://blog.npmjs.org/post/98131109725/npm-2-0-0, you can use custom arguments when executing scripts. The special option -- is
used by getopt https://goo.gl/KxMmtG to delimit the end of the options. npm will pass all the arguments after the -- directly to your
script:
npm run test -- --grep="pattern"
The arguments will only be passed to the script specified after npm run and not to any pre or post script.
The env script is a special built-in command that can be used to list environment variables that will be available to the script at run-
time. If an "env" command is defined in your package, it will take precedence over the built-in.
In addition to the shell's pre-existing PATH, npm run adds node_modules/.bin to the PATH provided to scripts. Any binaries provided by
locally-installed dependencies can be used without the node_modules/.bin prefix. For example, if there is a devDependency on tap in your
package, you should write:
"scripts": {"test": "tap test/*.js"}
instead of
"scripts": {"test": "node_modules/.bin/tap test/*.js"}
to run your tests.
The actual shell your script is run within is platform dependent. By default, on Unix-like systems it is the /bin/sh command, on Windows it
is the cmd.exe. The actual shell referred to by /bin/sh also depends on the system. As of `
https://github.com/npm/npm/releases/tag/v5.1.0 you can customize the shell with the script-shell configuration.
Scripts are run from the root of the module, regardless of what your current working directory is when you call npm run. If you want your
script to use different behavior based on what subdirectory you're in, you can use the INIT_CWD environment variable, which holds the full
path you were in when you ran npm run.
npm run sets the NODE environment variable to the node executable with which npm is executed. Also, if the --scripts-prepend-node-path is
passed, the directory within which node resides is added to the PATH. If --scripts-prepend-node-path=auto is passed (which has been the
default in npm v3), this is only performed when that node executable is not found in the PATH.
If you try to run a script without having a node_modules directory and it fails, you will be given a warning to run npm install, just in
case you've forgotten.
You can use the --silent flag to prevent showing npm ERR! output on error.
You can use the --if-present flag to avoid exiting with a non-zero exit code when the script is undefined. This lets you run potentially
undefined scripts without breaking the execution chain.
SEE ALSO
o npm help 7 scripts
o npm help test
o npm help start
o npm help restart
o npm help stop
o npm help 7 config
January 2019 NPM-RUN-SCRIPT(1)