Sponsored Content
Top Forums Shell Programming and Scripting Removing file lines that each match to a different patterns Post 302412980 by Jo_puzzled on Wednesday 14th of April 2010 07:01:10 AM
Old 04-14-2010
Removing file lines that each match to a different patterns

I have a very large file (10,000,000 lines), that contains a sample id and a property of that sample. I have another file that contains around 1,000,000 lines with sample ids that I want to remove from the original file (create a new file without these lines).
I know how to do this in Perl, but it is too time consuming to run. I am aware of sed and awk as commands that should be able to complete this task in a much faster time. I have tried to implement codes that I thought would work, even after consulting previous posts, none seem to quite cover it. I also find it hard to debug as the server I'm working on is French so I don't understand the error messages of my command.

Please could anyone suggest a quick way of achieving this ?

Here are examples of the files I'm dealing with.

Here is a tab delineated sample id and property.
Code:
HELIUM:1:2:3      ABCDEF
HELIUM:1:2:4      ADEFBC
HELIUM:1:2:5      BDFACE
HELIUM:1:2:6      BEBACG
HELIUM:1:2:7      ABCDEF
HELIUM:1:2:8      ADEFBC
HELIUM:1:2:9      BDFACE
HELIUM:1:3:0      BEBACG

Here is a list of ids (The common prefix is missing) I wish to remove:
Code:
:1:2:3
:1:2:5
:1:2:6
:1:2:9

Many thanks in advance for any help you can provide.

Last edited by Franklin52; 04-15-2010 at 08:47 AM.. Reason: Please use code tags!
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

sed/awk help to match list of patterns and remove from org file

Hi, From the pattern mentioned below remove lines based on pattern range. Conditions 1 Look For all lines starting with ALTER TABLE and Ending with ; and contains the word MOVE.I wanto to remove these lines from the file sample below. Note : The above pattern list could be found in... (1 Reply)
Discussion started by: rajan_san
1 Replies

2. Shell Programming and Scripting

Searching patterns in 1 file and deleting all lines with those patterns in 2nd file

Hi Gurus, I have a file say for ex. file1 which has 3500 lines in it which are different account numbers and another file (file2) which has 230000 lines in it. I want to read all the lines in file1 and delete all those lines from file2 which has that same pattern as in file1. I am not quite... (4 Replies)
Discussion started by: toms
4 Replies

3. Shell Programming and Scripting

print lines which match multiple patterns

Hi, I have a text file as follows: 11:38:11.054 run1_rdseq avg_2-5 999988.0000 1024.0000 11:50:52.053 run3_rdrand 999988.0000 1135.0 128.0417 11:53:18.050 run4_wrrand avg_2-5 999988.0000 8180.5833 11:55:42.051 run4_wrrand avg_2-5 999988.0000 213.8333 11:55:06.053... (2 Replies)
Discussion started by: annazpereira
2 Replies

4. Shell Programming and Scripting

Match multiple patterns in a file and then print their respective next line

Dear all, I need to search multiple patterns and then I need to print their respective next lines. For an example, in the below table, I will look for 3 different patterns : 1) # ATC_Codes: 2) # Generic_Name: 3) # Drug_Target_1_Gene_Name: #BEGIN_DRUGCARD DB00001 # AHFS_Codes:... (3 Replies)
Discussion started by: AshwaniSharma09
3 Replies

5. Shell Programming and Scripting

Retrieve lines that match any occurence in a list of patterns

I have two files. The first containing a header and six columns of data. Example file 1: Number SNP ID dbSNP RS ID Chromosome Result_Call Physical Position 787066 SNP_A-8575395 RS6650104 1 NOCALL 564477 786872 SNP_A-8575125 RS10458597 1 AA ... (13 Replies)
Discussion started by: Selftaught
13 Replies

6. Shell Programming and Scripting

Match 2 different patterns and print the lines

Hi, i have been trying to extract multiple lines based on two different patterns as below:- file1 @jkm|kdo|aas012|192.2.3.1 blablbalablablkabblablabla sjfdsakfjladfjefhaghfagfkafagkjsghfalhfk fhajkhfadjkhfalhflaffajkgfajkghfajkhgfkf jahfjkhflkhalfdhfwearhahfl @jkm|sdf|wud08q|168.2.1.3... (8 Replies)
Discussion started by: redse171
8 Replies

7. UNIX for Dummies Questions & Answers

Match patterns from another file and tag

Hi all, I have a file , which has 6 tab delimited fields, with $3 and $4 subfielded with spaces. I wamt to match cols $2,$3,$4 of tmp1 with tmp2, ..and then flag the 5th col if found. tmp1 1756 Xerm XermA XermB XermC XermD AA TT AA GG A 1 1763 Xerm XermA XermB XermC... (3 Replies)
Discussion started by: senhia83
3 Replies

8. Shell Programming and Scripting

Removing multiple lines from input file, if multiple lines match a pattern.

GM, I have an issue at work, which requires a simple solution. But, after multiple attempts, I have not been able to hit on the code needed. I am assuming that sed, awk or even perl could do what I need. I have an application that adds extra blank page feeds, for multiple reports, when... (7 Replies)
Discussion started by: jxfish2
7 Replies

9. Shell Programming and Scripting

awk to print match or non-match and select fields/patterns for non-matches

In the awk below I am trying to output those lines that Match between file1 and file2, those Missing in file1, and those missing in file2. Using each $1,$2,$4,$5 value as a key to match on, that is if those 4 fields are found in both files the match, but if those 4 fields are not found then missing... (0 Replies)
Discussion started by: cmccabe
0 Replies

10. Shell Programming and Scripting

Selecting text on multiple lines, then removing a beginning and end patterns

I have a file similar to the below. I am selecting only the paragraphs with @inlineifset. I am using the following command sed '/@inlineifset/,/^ *$/!d; s/@inlineifset{mrg, @btpar{@//' $flnm >> $ofln This produces @section Correlations between seismograms,,,,}} ... (5 Replies)
Discussion started by: Danette
5 Replies
IRSEND(1)								FSF								 IRSEND(1)

NAME
irsend - basic LIRC program to send infra-red commands SYNOPSIS
irsend [options] DIRECTIVE REMOTE CODE [CODE...] DESCRIPTION
Asks the lircd daemon to send one or more CIR (Consumer Infra-Red) commands. This is intended for remote control of electronic devices such as TV boxes, HiFi sets, etc. DIRECTIVE can be: SEND_ONCE - send CODE [CODE ...] once SEND_START - start repeating CODE SEND_STOP - stop repeating CODE LIST - list configured remote items SET_TRANSMITTERS - set transmitters NUM [NUM ...] SIMULATE - simulate IR event REMOTE is the name of a remote, as described in the lircd configuration file. CODE is the name of a remote control key of REMOTE, as it appears in the lircd configuration file. NUM is the transmitter number of the hardware device. For the LIST DIRECTIVE, REMOTE and/or CODE can be empty: LIST "" "" - list all configured remote names LIST REMOTE "" - list all codes of REMOTE LIST REMOTE CODE - list only CODE of REMOTE The SIMULATE command only works if it has been explicitly enabled in lircd. -h --help display usage summary -v --version display version -d --device use given lircd socket [/var/run/lirc/lircd] -a --address=host[:port] connect to lircd at this address -# --count=n send command n times EXAMPLES
irsend LIST DenonTuner "" irsend SEND_ONCE DenonTuner PROG-SCAN irsend SEND_ONCE OnkyoAmpli VOL-UP VOL-UP VOL-UP VOL-UP irsend SEND_START OnkyoAmpli VOL-DOWN ; sleep 3 irsend SEND_STOP OnkyoAmpli VOL-DOWN irsend SET_TRANSMITTERS 1 irsend SET_TRANSMITTERS 1 3 4 irsend SIMULATE "0000000000000476 00 OK TECHNISAT_ST3004S" FILES
/etc/lirc/lircd.conf Default lircd configuration file. It should contain all the remotes, their infra-red codes and the corresponding timing and wave- form details. DIAGNOSTICS
If lircd is not running (or /var/run/lirc/lircd lacks write permissions) irsend aborts with the following diagnostics: "irsend: could not connect to socket" "irsend: Connection refused" (or "Permission denied"). SEE ALSO
The documentation for lirc is maintained as html pages. They are located under html/ in the documentation directory. lircd(8), mode2(1), smode2(1), xmode2(1), irrecord(1), irw(1), http://www.lirc.org. irsend 0.8.7pre1 May 2010 IRSEND(1)
All times are GMT -4. The time now is 03:07 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy