Login or Register to Ask a Question and Join Our Community

Sponsored Content
Full Discussion: How to control grep output intact for each matching line?
Top Forums Shell Programming and Scripting How to control grep output intact for each matching line? Post 303024540 by yifangt on Wednesday 10th of October 2018 01:18:52 PM
Old 10-10-2018
How to control grep output intact for each matching line?
I have multiple (~80) files (some can be as big as 30GB of >1 billion of lines!) to grep on a pattern, and piped the match to a single file. I have a 96-core machine so that each grep job was sent to the background to speed up the search:
Code:
file1.tab
chr1A_part1    123241847    123241848
chr1A_part1    123241848    123241849
chr1A_part1    123241849    123241850
chr1A_part1    123241850    123241851
......

The input files have uniformly 3 fields each row, so should the output file,
Code:
for file in $(cat files.list); do 
grep -F chr1A ${file} >> subset_chr1A.tab &
done

but I found some of the matching lines are broken and the output file became a mess!
Code:
subset_chr1A.tab
chr1A_part1    123241847    123241848
chr1A_part1    123241848    123241849
chr1A_part1    1232
41849    123241850
ch1
chr1A_part1    12
3241850    
chr1A_part1    123441848    123441849
123541851
...

It seems to me the problem is from the writing of the pipe, as 80 grep jobs for 80 files are writing to the same output file. By default grep prints matching lines so that I assume each row should be printed as a whole, but it did not in my case.

What is wrong here?
Last edited by yifangt; 10-10-2018 at 02:20 PM.. Reason: typos
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

need grep to output basename and line#

I have a script that sorta works the way I want but I would rather just get the base name and line number from the grep output. My current script is this one liner: grep -n "$1" $SCCSPATH/*/s.*.k | cut -c1-80 which if I was searching for 121197 I would get something like this: ... (18 Replies)
Discussion started by: zoo591
18 Replies

2. UNIX for Advanced & Expert Users

Grep Line with Matching Fields

Below is the scenario. Help is appreciated. File1: ( 500,000 lines ) : Three fields comma delimited : Not sorted 1234FAA,435612,88975 1224FAB,12345,212356 File2: ( 4,000,000 lines ) : Six fields comma delimited (Last 3 field should match the 3 fields of File1) : Not Sorted : ... (13 Replies)
Discussion started by: hemangjani
13 Replies

3. UNIX for Dummies Questions & Answers

How to grep / zgrep to output ONLY the matching filename and line number?

Hi all, I am trying to zgrep / grep list of files so that it displays only the matching filename:line number and does not display the whole line, like: (echo "1.txt";echo "2.txt") | xargs zgrep -no STRING If I use -o option, it displays the matching STRING and if not used, displays the... (3 Replies)
Discussion started by: vvaidyan
3 Replies

4. UNIX for Dummies Questions & Answers

Grep or other ways to output line above and/or below searched line

Hi all, Would like to know how I could search for a string 'xyz' but have the output show the line plus the line above and/or below all lines found. eg. search for xyz from file containing: abc 12345 asdf xyz asdfds wwwww kjkjkj ppppp kkkxyz eeee zzzzz and the output to... (2 Replies)
Discussion started by: sammac
2 Replies

5. Shell Programming and Scripting

Identify matching data in a file and output to original line, in perl

Hi, I haven't done this for awhile, and further, I've never done it in perl so I appreciate any help you can give me. I have a file of lines, each with 5 data points that look like this: AB,N,ALLIANCEBERNSTEIN HLDNG L.P,AB,N ALD,N,ALLIED CAPITAL CORPORATION,ALD,N AFC,N,ALLIED CAPITAL... (4 Replies)
Discussion started by: Pcushing
4 Replies

6. Shell Programming and Scripting

find out line number of matching string using grep

Hi all, I want to display line number for matching string in a file. can anyone please help me. I used grep -n "ABC" file so it displays 6 ABC. But i only want to have line number,i don't want that it should prefix matching context with line number. Actually my original... (10 Replies)
Discussion started by: sarbjit
10 Replies

7. Shell Programming and Scripting

Help in removing control M and Line feed in output file.

Hi All, In my output file i am getting control m character and also the line feeds at different places and with different combinations, the content of the file is supposed to be in a single line but if there is a line feed in between then from there onwards it's going into new line. I tried... (7 Replies)
Discussion started by: Bipin Kumar
7 Replies

8. Shell Programming and Scripting

Grep log file to get line above matching pattern

Hi, I have a log file that looks like this "delete" : { "_type" : "cl", "_id" : "1000600000000562636", "_version" : 1, "status" : 200, "found" : false } }, { "delete" : { "_type" : "cl", "_id" : "1000600000000562643", ... (4 Replies)
Discussion started by: wahi80
4 Replies

9. Shell Programming and Scripting

Printing the output of a gzip command intact

how can i get the printf command or the echo command to print the data that are inbetween the first and the last quotes? #!/bin/sh printf '%s\n' "^_<8b>^H^@U<8c>MX^@^Cí=ÙzÚH<97>×ð^Teìn<8c>Ób_<9d><9f>dXd<9b>^N^F7<82>8qâÎ'^K^Y^T<83>D<90>°M^Lý^Hó^Fs5w3ß|s5/ÐýbS%©<84>^DBH... (4 Replies)
Discussion started by: SkySmart
4 Replies

10. UNIX for Beginners Questions & Answers

Grep file starting from pattern matching line

I have a file with a list of references towards the end and want to apply a grep for some string. text .... @unnumbered References @sp 1 @paragraphindent 0 2017. @strong{Chalenski, D.A.}; Wang, K.; Tatanova, Maria; Lopez, Jorge L.; Hatchell, P.; Dutta, P.; @strong{Small airgun... (1 Reply)
Discussion started by: kristinu
1 Replies

LEARN ABOUT REDHAT

set_color

set_color(1)							       fish							      set_color(1)

NAME

       set_color - set_color - set the terminal color

set_color - set the terminal color
   Synopsis
       set_color [-v --version] [-h --help] [-b --background COLOR] [COLOR]

   Description
       Change the foreground and/or background color of the terminal. COLOR is one of black, red, green, brown, yellow, blue, magenta, purple,
       cyan, white and normal.

       o -b, --background Set the background color

       o -c, --print-colors Prints a list of all valid color names

       o -h, --help Display help message and exit

       o -o, --bold Set bold or extra bright mode

       o -u, --underline Set underlined mode

       o -v, --version Display version and exit

       Calling set_color normal will set the terminal color to whatever is the default color of the terminal.

       Some terminals use the --bold escape sequence to switch to a brighter color set. On such terminals, set_color white will result in a grey
       font color, while set_color --bold white will result in a white font color.

       Not all terminal emulators support all these features. This is not a bug in set_color but a missing feature in the terminal emulator.

       set_color uses the terminfo database to look up how to change terminal colors on whatever terminal is in use. Some systems have old and
       incomplete terminfo databases, and may lack color information for terminals that support it. Download and install the latest version of
       ncurses and recompile fish against it in order to fix this issue.

Version 1.23.1							  Sun Jan 8 2012						      set_color(1)
All times are GMT -4. The time now is 01:13 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy