Sponsored Content
Operating Systems Linux Need awk script for removing duplicate records Post 302525407 by DGPickett on Thursday 26th of May 2011 03:09:47 PM
Old 05-26-2011
I see length changing but no pattern, dot becomes space(s) or tab, record counting, time field saving/overwriting but I do not have your requirements.
 

10 More Discussions You Might Find Interesting

1. Linux

Need awk script for removing duplicate records

I have huge txt file having millions of trade data. For e.g Trade.txt (first 8 lines in the file is header info) COB_DATE,TRADE_ID,SOURCE_SYSTEM_TRADE_ID,TRADE_GROUP_ID, TRADE_TYPE,DEALER_NAME,EXTERNAL_COUNTERPARTY_ID, EXTERNAL_COUNTERPARTY_NAME,DB_COUNTERPARTY_ID,... (6 Replies)
Discussion started by: nmumbarkar
6 Replies

2. Shell Programming and Scripting

Issues with filtering duplicate records using gawk script

Hi All, I have huge trade file with milions of trades.I need to remove duplicate records (e.g I have following records) 30/10/2009,trdeId1,..,.. 26/10/2009.tradeId1,..,..,, 30/10/2009,tradeId2,.. In the above case i need to filter duplicate recods and I should get following output.... (2 Replies)
Discussion started by: nmumbarkar
2 Replies

3. Shell Programming and Scripting

Removing duplicate records from 2 files

Can anyone help me to removing duplicate records from 2 separate files in UNIX? Please find the sample records for both the files cat Monday.dat 3FAHP0JA1AR319226MOHMED ATEK 966504453742 SAU2010DE 3LNHL2GC6AR636361HEA DEUK CHOI 821057314531 KOR2010LE 3MEHM0JG7AR652083MUTLAB NAL-NAFISAH... (4 Replies)
Discussion started by: zooby
4 Replies

4. Shell Programming and Scripting

Removing duplicate records in a file based on single column

Hi, I want to remove duplicate records including the first line based on column1. For example inputfile(filer.txt): ------------- 1,3000,5000 1,4000,6000 2,4000,600 2,5000,700 3,60000,4000 4,7000,7777 5,999,8888 expected output: ---------------- 3,60000,4000 4,7000,7777... (5 Replies)
Discussion started by: G.K.K
5 Replies

5. Shell Programming and Scripting

removing duplicate records comparing 2 csv files

Hi All, I want to remove the rows from File1.csv by comparing a column/field in the File2.csv. If both columns matches then I want that row to be deleted from File1 using shell script(awk). Here is an example on what I need. File1.csv: RAJAK,ACTIVE,1 VIJAY,ACTIVE,2 TAHA,ACTIVE,3... (6 Replies)
Discussion started by: rajak.net
6 Replies

6. Shell Programming and Scripting

Removing duplicate records in a file based on single column explanation

I was reading this thread. It looks like a simpler way to say this is to only keep uniq lines based on field or column 1. https://www.unix.com/shell-programming-scripting/165717-removing-duplicate-records-file-based-single-column.html Can someone explain this command please? How are there no... (5 Replies)
Discussion started by: cokedude
5 Replies

7. Shell Programming and Scripting

Help with removing duplicate entries with awk or Perl

Hi, I have a file which looks like:ke this : chr1 11127067 11132181 89 chr1 11128023 11128311 chr1 11130990 11131025 chr1 11127067 11132181 89 chr1 11128023 11128311 chr1 11131583... (22 Replies)
Discussion started by: Amit Pande
22 Replies

8. Homework & Coursework Questions

Script: Removing HTML tags and duplicate lines

Use and complete the template provided. The entire template must be completed. If you don't, your post may be deleted! 1. The problem statement, all variables and given/known data: You will write a script that will remove all HTML tags from an HTML document and remove any consecutive... (3 Replies)
Discussion started by: tburns517
3 Replies

9. Shell Programming and Scripting

To select non-duplicate records using awk

Friends, I have data sorted on id like this id addressl 1 abc 2 abc 2 abc 2 abc 3 aabc 4 abc 4 abc I want to pick all ids with addressesses leaving out duplicate records. Desired output would be id address 1 abc 2 abc 3 abc 4 abc (5 Replies)
Discussion started by: paresh n doshi
5 Replies

10. Shell Programming and Scripting

Removing specific records from files when duplicate key

Hello I have been trying to remove a row from a file which has the same first three columns as another row - I have tried lots of different combinations of suggestion on this forum but can't get it exactly right. what I have is 900 - 1000 = 0 900 - 1000 = 2562 1000 - 1100 = 0 1000 - 1100... (7 Replies)
Discussion started by: tinytimmay
7 Replies
FBB::mlm(3bobcat)					      OFoldStream manipulator						 FBB::mlm(3bobcat)

NAME
FBB::mlm - Manipulator modifying left margins of OFoldStream objects SYNOPSIS
#include <bobcat/ofoldstream> or #include <bobcat/ofoldstreambuf> Linking option: -lbobcat DESCRIPTION
The mlm class implements a manipulator that can be inserted into OFoldStream objects to modify the stream's left margin by a requested amount. The request cannot result in a negative left margin value. If a negative left margin would be the arithmetic result of the request then left margin 0 will silently be used. Depending on the tab-setting of the OFoldStream the inserted value represents the number of blank space characters or the number of tab-characters that will be added to the left margin. The request will be processed at the next newline character or std::flush or std::endl manipulator that is inserted into the stream. If a line is still empty once an mlm object and a flush manipulator are inserted into the stream then the new left margin will be effective at the next word inserted into that line (cf., the example section below) A bad_cast exception is thrown when the manipulator is inserted into an ostream that is not using a OFoldStreambuf buffer. NAMESPACE
FBB All constructors, members, operators and manipulators, mentioned in this man-page, are defined in the namespace FBB. INHERITS FROM
- CONSTRUCTOR
o mlm(int addValue): The standard copy constructor is available. MEMBER FUNCTIONS
There are no public or protected member functions in this class. EXAMPLE
#include <iostream> #include <bobcat/ofoldstream> using namespace std; using namespace FBB; int main() { OFoldStream out(cout, 0, 80); out << "hello world (left margin is 0)" << mlm(4) << " " "this uses a 4 character wide left margin " << mlm(-10) << flush << "left margin -6 changed to 0, active on this line "; return 0; } FILES
bobcat/mlm - defines the class interface SEE ALSO
bobcat(7), manipulators(3bobcat), lm(3bobcat), ofoldstream(3bobcat) BUGS
None Reported. DISTRIBUTION FILES
o bobcat_3.01.00-x.dsc: detached signature; o bobcat_3.01.00-x.tar.gz: source archive; o bobcat_3.01.00-x_i386.changes: change log; o libbobcat1_3.01.00-x_*.deb: debian package holding the libraries; o libbobcat1-dev_3.01.00-x_*.deb: debian package holding the libraries, headers and manual pages; o http://sourceforge.net/projects/bobcat: public archive location; BOBCAT
Bobcat is an acronym of `Brokken's Own Base Classes And Templates'. COPYRIGHT
This is free software, distributed under the terms of the GNU General Public License (GPL). AUTHOR
Frank B. Brokken (f.b.brokken@rug.nl). libbobcat1-dev_3.01.00-x.tar.gz 2005-2012 FBB::mlm(3bobcat)
All times are GMT -4. The time now is 08:12 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy