Sponsored Content
Top Forums Shell Programming and Scripting Problem using grep in bash script Post 302565241 by errcricket on Monday 17th of October 2011 10:59:35 AM
Old 10-17-2011
searching files using grep command in bash script

When it comes to programing and UNIX, I know just enough to be really really dangerous.

I have written a python script to parse through a file that contains ~1 million lines. Depending on whether a certain string is matched, the line is copied into a particular file. For the sake of brevity, the lines are something like this:

Quote:
ABC-1
ABC-1
CCC-33
CCC-33
CCC-33
...
I tried the python code out on a small file, and everything seems to work. However, since the actual file is a massive, I want to double check it with grep to make sure that the total number of ABC-1's in file x is the same number of ABC-1's in file y.

On the command line, I wrote a simple script that will check this for me.

Code:
grep "ABC-1" fileName.fna | wc -l

This seems to work just fine.

Problem: The contents of the original file are copied into 10 other files. I want check each file AND since there are ~50 unique strings (i.e. ABC-1), I would like to check for each string. Writing the simple script ~500 times is tedious.

I wrote a bash script but when I execute the file from the command line (
Quote:
bash counts.sh
), I get an error saying wc is an illegal option and that it cannot be found.

Code:
#!/bin/bash
echo "ABC-1 in fileName.fna"
grep "ABC-1" fileName.fna | wc -l

echo "CCC-33 in fileName3.fna"
grep "CCC-33" fileName3.fna | wc -l

Ideally, I think I should make a vector/list/array of file names and a vector/list/array of searchable strings and use a loop that will print out the string, the filename, and the number of times the string occurs in the file...but I don't know how to do that.

So if anyone knows how to re-arrange my 1-liner script - thank you. If anyone can help me with writing a loop script - thank you. Either option would be awesome.

Last edited by errcricket; 10-17-2011 at 12:18 PM.. Reason: better title
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Bash script (using find and grep)

I'm trying to make a simple search script but cannot get it right. The script should search for keywords inside files. Then return the file paths in a variable. (Each file path separated with \n). #!/bin/bash SEARCHQUERY="searchword1 searchword2 searchword3"; for WORD in $SEARCHQUERY do ... (6 Replies)
Discussion started by: limmer
6 Replies

2. Shell Programming and Scripting

Problem in bash script

I have written a script and I get error and I don't understand why. neededParameters=2 numOfParameters=0 correctNum=0 while getopts "s:l:" opt do case "$opt" in s) serviceName= $OPTARG #errorline 1 numOfParameters= $numOfParameters + 1 ;; l) ... (12 Replies)
Discussion started by: programAngel
12 Replies

3. Shell Programming and Scripting

Bash script - Grep

Hi, I am very new to bash scripting and I need to write a bash script that takes two arguments, a string and a file. The output should be each line which matches the string *from the beginning of the line*. For example, given a string "ANA" the line starting with "ANABEL" will be printed, but... (9 Replies)
Discussion started by: jboy
9 Replies

4. Shell Programming and Scripting

Help with grep and read function in a BASH Script

I'm putting together a script that will search my mail archives for emails that meet certain criteria and output the files to a text file. I can manually cat that text file and pipe it into sendmail and it will work (i.e. cat /pathtofile/foo.txt | sendmail -t me@company.com) My script sends... (7 Replies)
Discussion started by: binary-ninja
7 Replies

5. Shell Programming and Scripting

pipe to grep doesn't work in bash script

Hi, I'm trying to write a script that checks gvfs to see if a mount exists so I can run it from network-manager's status hooks. I thought I'd pipe the output of gvfs-mount -l to grep for the particular mounts I care about. When I do this in a bash script: cmnd="gvfs-mount -l | grep -i... (4 Replies)
Discussion started by: kcstrom
4 Replies

6. Shell Programming and Scripting

Problem with Bash Script.

Hi guys! I'm new to the forum and to the Bash coding scene. I have the following code paths=/test/a paths=/test/b keywords=\"*car*\" keywords=\"*food*\" for file in `find paths -type f -ctime -1 -name keywords -print 2>/dev/null` do #.... do stuff here for every $file found... (5 Replies)
Discussion started by: RedSpyder
5 Replies

7. UNIX for Dummies Questions & Answers

Problem with multiple grep in bash loop

Hello, I am trying to create a matrix of 0's and 1's depending on whether a gene and sample name are found in the same line in a file called results.txt. An example of the results.txt file is (tab-delimited): Sample1 Gene1 ## Gene2 ## Sample2 Gene2 ## Gene 4 ## Sample3 Gene3 ... (2 Replies)
Discussion started by: InfoSeeker2
2 Replies

8. UNIX for Dummies Questions & Answers

[Solved] Grep in bash script

Hi Experts, I'm writing script to find out last files and its modified date - unfortunately am having problem with the below script. Error message: "grep: sales.txt: No such file or directory" #!/bin/bash var=1 var1=`awk '{n++} END {print n}' sales.txt` while ] do prod=$var... (6 Replies)
Discussion started by: parpaa
6 Replies

9. Shell Programming and Scripting

How to use grep in a loop using a bash script?

Dear all, Please help with the following. I have a file, let's call it data.txt, that has 3 columns and approx 700,000 lines, and looks like this: rs1234 A C rs1236 T G rs2345 G T Please use code tags as required by forum rules! I have a second file, called reference.txt,... (1 Reply)
Discussion started by: aberg
1 Replies

10. Shell Programming and Scripting

Is there a BASH script allowing me to grep specifics from /var/log/messages?

I am wondering if there is a script (if one exists, not confident in my own scripting ability) that is able to bring up specified information from the /var/log/messages. I need to show logged traffic on specific dates and times and protocols (ie. Show all insecure FTP traffic (most likely via... (13 Replies)
Discussion started by: vgplayer54
13 Replies
PMDABASH(1)						      General Commands Manual						       PMDABASH(1)

NAME
pmdabash - Bourne-Again SHell trace performance metrics domain agent SYNOPSIS
$PCP_PMDAS_DIR/bash/pmdabash [-C] [-d domain] [-l logfile] [-I interval] [-t timeout] [-U username] configfile DESCRIPTION
pmdabash is an experimental Performance Metrics Domain Agent (PMDA) which exports "xtrace" events from a traced bash(1) process. This includes the command execution information that would usually be sent to standard error with the set -x option to the shell. Event metrics are exported showing each command executed, the function name and line number in the script, and a timestamp. Additionally, the process identifier for the shell and its parent process are exported. This requires bash version 4 or later. A brief description of the pmdabash command line options follows: -d It is absolutely crucial that the performance metrics domain number specified here is unique and consistent. That is, domain should be different for every PMDA on the one host, and the same domain number should be used for the same PMDA on all hosts. -l Location of the log file. By default, a log file named bash.log is written in the current directory of pmcd(1) when pmdabash is started, i.e. $PCP_LOG_DIR/pmcd. If the log file cannot be created or is not writable, output is written to the standard error instead. -s Amount of time (in seconds) between subsequent evaluations of the shell trace file descriptor(s). The default is 2 seconds. -m Maximum amount of memory to be allowed for each event queue (one per traced process). The default is 2 megabytes. -U User account under which to run the agent. The default is the unprivileged "pcp" account in current versions of PCP, but in older versions the superuser account ("root") was used by default. INSTALLATION
In order for a host to export the names, help text and values for the bash performance metrics, do the following as root: # cd $PCP_PMDAS_DIR/bash # ./Install As soon as an instrumented shell script (see INSTRUMENTATION selection below) is run, with tracing enabled, new metric values will appear - no further setup of the agent is required. If you want to undo the installation, do the following as root: # cd $PCP_PMDAS_DIR/bash # ./Remove pmdabash is launched by pmcd(1) and should never be executed directly. The Install and Remove scripts notify pmcd(1) when the agent is installed or removed. INSTRUMENTATION
In order to allow the flow of event data between a bash(1) script and pmdabash, the script should take the following actions: #!/bin/sh source $PCP_DIR/etc/pcp.sh pcp_trace on $@ # enable tracing echo "awoke, $count" pcp_trace off # disable tracing The tracing can be enabled and disabled any number of times by the script. On successful installation of the agent, several metrics will be available: $ pminfo bash bash.xtrace.numclients bash.xtrace.maxmem bash.xtrace.queuemem bash.xtrace.count bash.xtrace.records bash.xtrace.parameters.pid bash.xtrace.parameters.parent bash.xtrace.parameters.lineno bash.xtrace.parameters.function bash.xtrace.parameters.command When an instrumented script is running, the generation of event records can be verified using the pmevent(1) command, as follows: $ pmevent -t 1 -x '' bash.xtrace.records host: localhost samples: all bash.xtrace.records["4538 ./test-trace.sh 1 2 3"]: 5 event records 10:00:05.000 --- event record [0] flags 0x19 (point,id,parent) --- bash.xtrace.parameters.pid 4538 bash.xtrace.parameters.parent 4432 bash.xtrace.parameters.lineno 43 bash.xtrace.parameters.command "true" 10:00:05.000 --- event record [1] flags 0x19 (point,id,parent) --- bash.xtrace.parameters.pid 4538 bash.xtrace.parameters.parent 4432 bash.xtrace.parameters.lineno 45 bash.xtrace.parameters.command "(( count++ ))" 10:00:05.000 --- event record [2] flags 0x19 (point,id,parent) --- bash.xtrace.parameters.pid 4538 bash.xtrace.parameters.parent 4432 bash.xtrace.parameters.lineno 46 bash.xtrace.parameters.command "echo 'awoke, 3'" 10:00:05.000 --- event record [3] flags 0x19 (point,id,parent) --- bash.xtrace.parameters.pid 4538 bash.xtrace.parameters.parent 4432 bash.xtrace.parameters.lineno 47 bash.xtrace.parameters.command "tired 2" 10:00:05.000 --- event record [4] flags 0x19 (point,id,parent) --- bash.xtrace.parameters.pid 4538 bash.xtrace.parameters.parent 4432 bash.xtrace.parameters.lineno 38 bash.xtrace.parameters.function "tired" bash.xtrace.parameters.command "sleep 2" FILES
$PCP_PMCDCONF_PATH command line options used to launch pmdabash $PCP_PMDAS_DIR/bash/help default help text file for the bash metrics $PCP_PMDAS_DIR/bash/Install installation script for the pmdabash agent $PCP_PMDAS_DIR/bash/Remove undo installation script for pmdabash $PCP_LOG_DIR/pmcd/bash.log default log file for error messages and other information from pmdabash PCP ENVIRONMENT
Environment variables with the prefix PCP_ are used to parameterize the file and directory names used by PCP. On each installation, the file /etc/pcp.conf contains the local values for these variables. The $PCP_CONF variable may be used to specify an alternative configura- tion file, as described in pcp.conf(5). SEE ALSO
bash(1), pmevent(1) and pmcd(1). Performance Co-Pilot PCP PMDABASH(1)
All times are GMT -4. The time now is 06:24 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy