Sponsored Content
Top Forums Shell Programming and Scripting Deleting Repeating lines from a txt file via script Post 302903937 by Gautham on Saturday 31st of May 2014 01:10:20 AM
Old 05-31-2014
Ya that was my concern..

Your code will remove all duplicate lines, but what i need is to remove only duplicate in headers like this:
Quote:
20140523121432,0,12,roger
Loc=redmont
From=roger@xxx.com, To=david@yyy.com,,
I thought of using any pattern matching kinda logic but not able to move with it..

Last edited by Gautham; 05-31-2014 at 02:17 AM.. Reason: updated
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

print all even lines of a txt file

In other news, I have a colors text file with hundreds of lines, and I want to print only the even numbered lines. for example I have this file looks something like this: ALLCOLORS.TXT red red green red blue red red red green red red blue green green green blue blue blue red blue blue blue... (1 Reply)
Discussion started by: ajp7701
1 Replies

2. Shell Programming and Scripting

Deleting lines that contain spaces in a txt file

I need some help deleting lines in a file that contain spaces. Im sure awk or sed will work but i dont know much about those commands. Any help is appreciated :D (7 Replies)
Discussion started by: r04dw4rri0r
7 Replies

3. UNIX for Dummies Questions & Answers

Deleting lines in .txt with nonspecific value

Hello, i am new to the forum and know nothing about programing, Linux or Unix :( hope somebody can help me out. I have a .txt file that i need to delete certain lines from. After searching the forum i noticed that using "sed" was the way to go, so i installed gnuwin32 (i use windows xp... (4 Replies)
Discussion started by: luis3141
4 Replies

4. Shell Programming and Scripting

script to delete lines from a txt file if pattern matches

File 6 dbnawldb010-b office Memphis_Corp_SQL_Diff Memphis-Corp-SQL-Inc-Application-Backup 03/09/11 03:24:04 42 luigi-b IPNRemitDB Memphis_Corp_SQL_Diff Memphis-Corp-SQL-Inc-Application-Backup 03/10/11 00:41:36 6 ebs-sqldev1-b IPNTracking Memphis_Corp_SQL_Diff... (4 Replies)
Discussion started by: ajiwww
4 Replies

5. Shell Programming and Scripting

sed to cp lines x->y from 1.txt into lines a->b in file2.txt

I have one base file, and multiple target files-- each have uniform line structure so no need to use grep to find things-- can just define sections by line number. My question is quite simple-- can I use sed to copy a defined block of lines (say lines 5-10) from filename1.txt to overwrite an... (3 Replies)
Discussion started by: czar21
3 Replies

6. Shell Programming and Scripting

merging two .txt files by alternating x lines from file 1 and y lines from file2

Hi everyone, I have two files (A and B) and want to combine them to one by always taking 10 rows from file A and subsequently 6 lines from file B. This process shall be repeated 40 times (file A = 400 lines; file B = 240 lines). Does anybody have an idea how to do that using perl, awk or sed?... (6 Replies)
Discussion started by: ink_LE
6 Replies

7. UNIX for Dummies Questions & Answers

find lines in file1.txt not found in file2.txt memory problem

I have a diff command that does what I want but when comparing large text/log files, it uses up all the memory I have (sometimes over 8gig of memory) diff file1.txt file2.txt | grep '^<'| awk '{$1="";print $0}' | sed 's/^ *//' Is there a better more efficient way to find the lines in one file... (5 Replies)
Discussion started by: raptor25
5 Replies

8. UNIX for Dummies Questions & Answers

redirecting arguments in a script to multiple lines in a .txt file

Ok hope my vocab is right here, i'm trying to write multiple sets of arguments to another file for example: I have a script that accepts four arguments and sends them to a another file $write.sh it then out in so the file receiver.txt would contain this: it then out in what... (2 Replies)
Discussion started by: austing5
2 Replies

9. Shell Programming and Scripting

Getting lines from .txt file

Hi I have a file with contents: NAMES John carrey williams How can I get all the names and store them in seperate variables(or arrays) please keep in mind that the no. of such names is not known.Three here is a bogus value ~thanks (4 Replies)
Discussion started by: leghorn
4 Replies

10. Shell Programming and Scripting

Creating repeating record in between file through script

apprecieate your help to resove this. My source file looke like 1001 000 HEADER 1001 001 RAJESH 1001 002 100 1001 002 200 1001 002 500 1001 006 FOOTER 1002 000 HEADER 1002 001 RAMESH 1002 002 100 1002 002 200 1002 002 500 1002 006 FOOTER my... (8 Replies)
Discussion started by: Ganesh L
8 Replies
REFORMAIL(1)						      Double Precision, Inc.						      REFORMAIL(1)

NAME
reformail - E-mail reformatting tool SYNOPSIS
reformail -s command [option...] reformail -D len filename reformail -x header:... reformail -X header:... reformail [options...] See below for additional options DESCRIPTION
The reformail program reads a message on standard input, reformats it in some way, and writes the message to standard output: Splitting mailboxes into individual messages The -s option splits the mbox-formatted mailbox file on standard input into individual messages. An external program is executed for each message. The contents of each individual message will be provided to the external program on standard input. If the FILENO environment variable is set to a number, reformail will consecutively increment FILENO each time the program is executed for each individual message. If FILENO is not set, it's initial value will be "000". If FILENO is set to a non-numeric value, FILENO will remain unchanged. Detecting duplicate messages The -D option implements a simple way to delete duplicate messages in incoming mail. filename is a file that will be approximately 'len' bytes long. This file will be used by reformail to save message IDs seen in recent mail. reformail reads the message on standard input. If the message has a Message-ID: header that's already in the cache file, reformail terminates with the exit code set to 0. Otherwise, reformail terminates with the exit code set to 1. Extracting headers The -x and -X options extract the indicated headers from the message, and print them to standard output. Multiple -x and -X options can be specified at the same time, and may be intermixed. The -x option extracts and prints the contents of the header. The -X option prints the name of the header as well. In all other situations, reformail copies the message on its standard input to its standard output, reformatting the message as follows: OPTIONS
-a'header: value' Append a custom header to the message if this header does not already exist. If the header is either Message-ID: or Resent-Message-ID: and the value is empty, reformail generates a (hopefully) unique message ID for you. -A'header: value' Append a custom header to the message even if this header already exists. If the header is either Message-ID: or Resent-Message-ID: and the value is empty, reformail generates a (hopefully) unique message ID for you. -c Concatenate multi-line headers. Headers split on multiple lines are combined into a single line. -dn If n is 1, each line will be terminated with CRLF. If n is 0 (default), each line will be terminated with LF. reformail reads a message with either line terminator, and will force the message to have the specified line termination. -f0 Remove the "From_" line from the message, if it's there. -f1 Add the "From_" line to the message, if it's not there. reformail will attempt to generate the "From_" line from any Errors-To:, Return-Path:, or From: headers in the message. "root" will be used if reformail is unable to determine the return address. -i'header: value' Appends a custom header to the message. If this header already exists it is renamed by prepending "Old-" to the name of the header. -I'header: value' Append a custom header to the message. If this header already exists in the message, the old header is completely removed. If the value is empty, any existing header is completely removed, and nothing gets appended. -R oldheader: newheader: Rename the indicated header. -u'header:' If this header occurs multiple times in the message, remove all occurrences except the first one. -U'header:' If this header occurs multiple times in the message, remove all occurrences except the last one. AUTOREPLIES
The autoreply options from earlier versions of mailbot have been moved into mailbot(1)[1]. BUGS
For the -a, -A, and -I options, a space after the header name and the colon is considered to be a non-empty field. Do not provide the same header to more than one family of header-modifying options, such as -u/-U and -a/-A. Doing so yields unpredictable results. It's better to run reformail several times (use a pipe, perhaps). SEE ALSO
courier(8)[2], sendmail(8), mailbot(1)[1], maildrop(1)[3]. AUTHOR
Sam Varshavchik Author NOTES
1. mailbot(1) [set $man.base.url.for.relative.links]/mailbot.html 2. courier(8) [set $man.base.url.for.relative.links]/courier.html 3. maildrop(1) [set $man.base.url.for.relative.links]/maildrop.html Courier Mail Server 08/30/2011 REFORMAIL(1)
All times are GMT -4. The time now is 05:38 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy