Sponsored Content
Top Forums Shell Programming and Scripting Remove matching lines with list of strings Post 302252202 by Nanu_Manju on Wednesday 29th of October 2008 02:58:29 AM
Old 10-29-2008
MySQL Thank you very much

Hi all,

Firstly, I am extremely sorry that I was late to reply and say thanks. Everytime I log out from this site happily. Thanks guys, keep up the good work. You people are wonderful.

Regards,
Manjax
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

different take on common ?: search for two strings and remove lines between them

Thank you for assisting, I've got a partial solution just needs a tweak. Hulk-BASH$ cat somefile.txt oh there is some stuff here some more stuff here START_LABEL stuff I want more stuff I want END_LABEL other stuff here too and even more stuff here too Hulk-BASH$ Hulk-BASH$ sed... (8 Replies)
Discussion started by: laser
8 Replies

2. Shell Programming and Scripting

compare two files and to remove the matching lines on both the files

I have two files and need to compare the two files and to remove the matching lines from both the files (4 Replies)
Discussion started by: shellscripter
4 Replies

3. Shell Programming and Scripting

Remove duplicate lines (the first matching line by field criteria)

Hello to all, I have this file 2002 1 23 0 0 2435.60 131.70 5.60 20.99 0.89 0.00 285.80 2303.90 2002 1 23 15 0 2436.60 132.90 6.45 21.19 1.03 0.00 285.80 2303.70 2002 1 23 ... (6 Replies)
Discussion started by: joggdial3000
6 Replies

4. Shell Programming and Scripting

How to remove lines containing strings

Hi Guys, I need some script in removing lines containing strings like ",, ," and "rows". Retain only numbers as the output. See below for the input and output file. INPUT FILE: 9817 9832 6285 6312 6313 6318 ,, , 6329 7078 7098 7130 7959 7983 (7 Replies)
Discussion started by: pinpe
7 Replies

5. Shell Programming and Scripting

Compare one files with strings from another + remove lines

Have two files and want to compare the content of file1 with file2. When matched remove the line. awk 'NR==FNR {b; next} !(b in $0)' file1 file2file1 1. if match 2. removefile2 1. this line has to be removed if match 2. this line has a match, remove 3. this line has no match, no removingThe... (3 Replies)
Discussion started by: sdf
3 Replies

6. Shell Programming and Scripting

Concatenating 2 lines from 2 files having matching strings

Hello All Unix Users, I am still new to Unix, however I am eager to learn it.. I have 2 files, some lines have some matching substrings, I would like to concatenate these lines into one lines, leaving other untouched. Here below is an example for that.. File 1 (fasta file): >292183... (6 Replies)
Discussion started by: Mohamed EL Hadi
6 Replies

7. Linux

Remove matching files from a list

hi, i have a cache file with below file list more gtda_11.cache GTDA_Dly_Pmix_GB_6_20130624.20130624070610.psv GTDA_Dly_Pmix_CH_006_20130624.20130624140018.psv GTDA_Dly_Pmix_GB_6_20130624.20130624070620.psv GTDA_Dly_Pmix_BE_6_20130624.20130624070620.psv... (2 Replies)
Discussion started by: renuk
2 Replies

8. Shell Programming and Scripting

Remove lines containing 2 or more duplicate strings

Within my text file i have several thousand lines of text with some lines containing duplicate strings/words. I would like to entirely remove those lines which contain the duplicate strings. Eg; One and a Two Unix.com is the Best This as a Line Line Example duplicate sentence with the word... (22 Replies)
Discussion started by: martinsmith
22 Replies

9. Shell Programming and Scripting

Remove lines matching a substring in a specific column

Dear group, I have following input text file: Brit 2016 11 18 12 00 10 1.485,00 EUR Brit 2016 11 18 12 00 10 142,64 EUR Brit 2016 11 18 12 00 10 19,80 EUR Brit 2016 11 18 12 00 10 545,00 EUR Brit 2016 11 18 12 00 10 6.450,00 EUR... (3 Replies)
Discussion started by: gfhsd
3 Replies

10. Shell Programming and Scripting

Remove lines with Pattren Matching

Hi , I need to remove the lines that matches the pattern TABLEEXCLUDE *.AQ$_*_F ; * is wildcard, it can be any word. For example, I have following file: TABLEEXCLUDE THOT.AQ$_PT_ADDR_CLEANUP_QTAB2_F ; TABLEEXCLUDE THOT.AQ$_MICRO_SERVICE_QT_F ; TEST TABLEEXCLUDE... (1 Reply)
Discussion started by: rcc50886
1 Replies
CTM_MAIL(1)						    BSD General Commands Manual 					       CTM_MAIL(1)

NAME
ctm_smail, ctm_dequeue, ctm_rmail -- send and receive ctm(1) deltas via mail SYNOPSIS
ctm_smail [-l log] [-m maxmsgsize] [-c maxctmsize] [-q queue-dir] ctm-delta mail-alias ctm_dequeue [-l log] [-n numchunks] queue-dir ctm_rmail [-Dfuv] [-l log] [-p piecedir] [-d deltadir] [-b basedir] [file ...] DESCRIPTION
In conjunction with the ctm(1) command, ctm_smail, ctm_dequeue and ctm_rmail are used to distribute changes to a source tree via email. The ctm_smail utility is given a compressed ctm delta, and a mailing list to send it to. It splits the delta into manageable pieces, encodes them as mail messages and sends them to the mailing list (optionally queued to spread the mail load). Each recipient uses ctm_rmail (either manually or automatically) to decode and reassemble the delta, and optionally call ctm to apply it to the source tree. At the moment, sev- eral source trees are distributed, and by several sites. These include the FreeBSD-current source and CVS trees, distributed by freefall.FreeBSD.org. Command line arguments for ctm_smail: -l log Instead of appearing on stderr, error diagnostics and informational messages (other than command line errors) are time stamped and written to the file log. -m maxmsgsize Limit the maximum size mail message that ctm_smail is allowed to send. It is approximate since mail headers and other niceties are not counted in this limit. If not specified, it will default to 64000 bytes, leaving room for 1535 bytes of headers before the rumoured 64k mail limit. -c maxctmsize Limit the maximum size delta that will be sent. Deltas bigger that this limit will cause an apology mail message to be sent to the mailing list. This is to prevent massive changes overwhelming users' mail boxes. Note that this is the size before encoding. Encoding causes a 4/3 size increase before mail headers are added. If not specified, there is no limit. -q queue-dir Instead of mailing the delta pieces now, store them in the given directory to be mailed later using ctm_dequeue. This feature allows the mailing of large deltas to be spread out over hours or even days to limit the impact on recipients with limited network bandwidth or small mail spool areas. ctm-delta is the delta to be sent, and mail-alias is the mailing list to send the delta to. The mail messages are sent using sendmail(8). Command line arguments for ctm_dequeue: -l log Instead of appearing on stderr, error diagnostics and informational messages (other than command line errors) are time stamped and written to the file log. -n numchunks Limit the number of mail messages that ctm_dequeue will send per run. By default, ctm_dequeue will send one mail message per run. queuedir is the directory containing the mail messages stored by ctm_smail. Up to numchunks mail messages will be sent in each run. The recipient mailing list is already encoded in the queued files. It is safe to run ctm_dequeue while ctm_smail is adding entries to the queue, or even to run ctm_smail multiple times concurrently, but a separate queue directory should be used for each tree being distributed. This is because entries are served in alphabetical order, and one tree will be unfairly serviced before any others, based on the delta names, not delta creation times. Command line arguments for ctm_rmail: -l log Instead of appearing on stderr, error diagnostics and informational messages (other than command line errors) are time stamped and written to the file log. -p piecedir Collect pieces of deltas in this directory. Each piece corresponds to a single mail message. Pieces are removed when complete deltas are built. If this flag is not given, no input files will be read, but completed deltas may still be applied with ctm if the -b flag is given. -d deltadir Collect completed deltas in this directory. Deltas are built from one or more pieces when all pieces are present. -b basedir Apply any completed deltas to this source tree. If this flag is not given, deltas will be stored, but not applied. The user may then apply the deltas manually, or by using ctm_rmail without the -p flag. Deltas will not be applied if they do not match the .ctm_status file in basedir (or if .ctm_status does not exist). -D Delete deltas after successful application by ctm. It is probably a good idea to avoid this flag (and keep all the deltas) as ctm has the ability to recover small groups of files from a full set of deltas. -f Fork and execute in the background while applying deltas with ctm. This is useful when automatically invoking ctm_rmail from sendmail because ctm can take a very long time to complete, causing other people's mail to be delayed, and can in theory cause spuri- ous mail retransmission due to the remote sendmail timing out, or even termination of ctm_rmail by mail filters such as MH's slocal. Do not worry about zillions of background ctm processes loading your machine, since locking is used to prevent more than one ctm invocation at a time. -u Pass the -u flag to the ctm command when applying the complete deltas, causing it to set the modification time of created and modi- fied files to the CTM delta creation time. -v Pass the -v flag to the ctm command when applying the complete deltas, causing a more informative output. All ctm output appears in the ctm_rmail log file. The file arguments (or stdin, if there are none) are scanned for delta pieces. Multiple delta pieces can be read from a single file, so an entire maildrop can be scanned and processed with a single command. It is safe to invoke ctm_rmail multiple times concurrently (with different input files), as might happen when sendmail is delivering mail asynchronously. This is because locking is used to keep things orderly. FILE FORMAT
Following are the important parts of an actual (very small) delta piece: From: owner-src-cur To: src-cur Subject: ctm-mail src-cur.0003.gz 1/4 CTM_MAIL BEGIN src-cur.0003.gz 1 4 H4sIAAAAAAACA3VU72/bNhD9bP0VByQoEiyRSZEUSQP9kKTeYCR2gDTdsGFAwB/HRogtG5K8NCj6 v4+UZSdtUQh6Rz0eee/xaF/dzx8up3/MFlDkBNrGnbttAwyo1pxoRgoiBNX/QJ5d3c9/X8DcPGGo lggkPiXngE4W1gUjKPJCYyk5MZRbIqmNW/ASglIFcdwIzTUxaAqhnCPcBqloKEkJVNDMF0Azk+Bo dDzzk0Ods/+A5gXv9YyJHjMCtJwQNeESNma7hOmXDRxn CTM_MAIL END 61065 The subject of the message always begins with ``ctm-mail'' followed by the name of the delta, which piece this is, and how many total pieces there are. The data are bracketed by ``CTM_MAIL BEGIN'' and ``CTM_MAIL END'' lines, duplicating the information in the subject line, plus a simple checksum. If the delta exceeds maxctmsize, then a message like this will be received instead: From: owner-src-cur To: src-cur Subject: ctm-notice src-cur.0999.gz src-cur.0999.gz is 792843 bytes. The limit is 300000 bytes. You can retrieve this delta via ftp. You are then on your own! ENVIRONMENT
If deltas are to be applied then ctm(1) and gunzip(1) must be in your PATH. FILES
QUEUEDIR/* Pieces of deltas encoded as mail messages waiting to be sent to the mailing list. PIECEDIR/* Pieces of deltas waiting for the rest to arrive. DELTADIR/* Completed deltas. BASEDIR/.ctm_status File containing the name and number of the next delta to be applied to this source tree. EXIT STATUS
The ctm_smail, ctm_dequeue and ctm_rmail utilities return exit status 0 for success, and 1 for various failures. The ctm_rmail utility is expected to be called from a mail transfer program, and thus signals failure only when the input mail message should be bounced (preferably into your regular maildrop, not back to the sender). In short, failure to apply a completed delta with ctm is not considered an error impor- tant enough to bounce the mail, and ctm_rmail returns an exit status of 0. EXAMPLES
To send delta 32 of src-cur to a group of wonderful code hackers known to sendmail as src-guys, limiting the mail size to roughly 60000 bytes, you could use: ctm_smail -m 60000 /wherever/it/is/src-cur.0032.gz src-guys To decode every ctm-mail message in your mailbox, assemble them into complete deltas, then apply any deltas built or lying around, you could use: ctm_rmail -p ~/pieces -d ~/deltas -b /usr/ctm-src-cur $MAIL (Note that no messages are deleted by ctm_rmail. Any mail reader could be used for that purpose.) To create a mail alias called receiver-dude that will automatically decode and assemble deltas, but not apply them, you could put the follow- ing lines in your /etc/mail/aliases file (assuming the /ctm/tmp and /ctm/deltas directories and /ctm/log file are writable by user daemon or group wheel): receiver-dude: "|ctm_rmail -p /ctm/tmp -d /ctm/deltas -l /ctm/log" owner-receiver-dude: real_dude@wherever.you.like The second line will catch failures and drop them into your regular mailbox, or wherever else you like. To apply all the deltas collected, and delete those applied, you could use: ctm_rmail -D -d /ctm/deltas -b /ctm/src-cur -l /ctm/apply.log For maximum flexibility, consider this excerpt from a procmail script: PATH=$HOME/bin:$PATH :0 w * ^Subject: ctm-mail cvs-cur | ctm_incoming together with the shell script ~/bin/ctm_incoming: #! /bin/sh PATH="$HOME/bin:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin" export PATH cd $HOME/ctm && ctm_rmail -f -p pieces -d deltas -l log -b /ctm which will deposit all ctm deltas in ~/ctm/deltas, apply them to the tree in /ctm, and drop any failures into your regular mail box. Note the PATH manipulation in ctm_incoming which allows ctm_rmail to execute ctm(1) on the (non-FreeBSD) machine that this example was taken from. SECURITY
On its own, CTM is an insecure protocol - there is no authentication performed that the changes applied to the source code were sent by a trusted party, and so care should be taken if the CTM deltas are obtained via an unauthenticated medium such as regular email. It is a rela- tively simple matter for an attacker to forge a CTM delta to replace or precede the legitimate one and insert malicious code into your source tree. If the legitimate delta is somehow prevented from arriving, this will go unnoticed until a later delta attempts to touch the same file, at which point the MD5 checksum will fail. To remedy this insecurity, CTM delta pieces generated by FreeBSD.org are cryptographically signed in a format compatible with the GNU Privacy Guard utility, available in /usr/ports/security/gpg, and the Pretty Good Privacy v5 utility, /usr/ports/security/pgp5. The relevant public key can be obtained by fingering ctm@FreeBSD.org. CTM deltas which are thus signed cannot be undetectably altered by an attacker. Therefore it is recommended that you make use of GPG or PGP5 to verify the signatures if you receive your CTM deltas via email. DIAGNOSTICS
In normal operation, ctm_smail will report messages like: ctm_smail: src-cur.0250.gz 1/2 sent to src-guys or, if queueing, ctm_smail: src-cur.0250.gz 1/2 queued for src-guys The ctm_dequeue utility will report messages like: ctm_dequeue: src-cur.0250.gz 1/2 sent The ctm_rmail utility will report messages like: ctm_rmail: src-cur.0250.gz 1/2 stored ctm_rmail: src-cur.0250.gz 2/2 stored ctm_rmail: src-cur.0250.gz complete If any of the input files do not contain a valid delta piece, ctm_rmail will report: ctm_rmail: message contains no delta and return an exit status of 1. You can use this to redirect wayward messages back into your real mailbox if your mail filter goes wonky. These messages go to stderr or to the log file. Messages from ctm(1) turn up here too. Error messages should be self explanatory. SEE ALSO
ctm(1), ctm(5) AUTHORS
Stephen McKay <mckay@FreeBSD.org> BSD
January 24, 1995 BSD
All times are GMT -4. The time now is 08:49 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy