Sponsored Content
Top Forums Shell Programming and Scripting Formatting a file - Remove Duplicate Post 302528314 by ni2 on Tuesday 7th of June 2011 01:59:09 AM
Old 06-07-2011
Java

Try sort -u

Code:
shell>less file
TABLE1
TABLE1 A
TABLE2
TABLE2 B
TABLE3
TABLE4
TABLE4 C
TABLE4

shell> sort -u file

TABLE1
TABLE1 A
TABLE2
TABLE2 B
TABLE3
TABLE4
TABLE4 C

I noticed that in your output you dropped TABLE1 but kept TABLE4. Was that intentional?
 

9 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

Remove Duplicate Lines in File

I am doing KSH script to remove duplicate lines in a file. Let say the file has format below. FileA 1253-6856 3101-4011 1827-1356 1822-1157 1822-1157 1000-1410 1000-1410 1822-1231 1822-1231 3101-4011 1822-1157 1822-1231 and I want to simply it with no duplicate line as file... (5 Replies)
Discussion started by: Teh Tiack Ein
5 Replies

2. UNIX for Dummies Questions & Answers

Remove Duplicate lines from File

I have a log file "logreport" that contains several lines as seen below: 04:20:00 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 06:38:08 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but responded to ping 07:11:05 /usr/lib/snmp/snmpdx: Agent snmpd appeared dead but... (18 Replies)
Discussion started by: Nysif Steve
18 Replies

3. Shell Programming and Scripting

Remove all instances of duplicate records from the file

Hi experts, I am new to scripting. I have a requirement as below. File1: A|123|NAME1 A|123|NAME2 B|123|NAME3 File2: C|123|NAME4 C|123|NAME5 D|123|NAME6 1) I have 2 merge both the files. 2) need to do a sort ( key fields are first and second field) 3) remove all the instances... (3 Replies)
Discussion started by: vukkusila
3 Replies

4. Shell Programming and Scripting

Sort and Remove Duplicate on file

How do we sort and remove duplicate on column 1,2 retaining the record with maximum date (in feild 3) for the file with following format. aaa|1234|2010-12-31 aaa|1234|2010-11-10 bbb|345|2011-01-01 ccc|346|2011-02-01 bbb|345|2011-03-10 aaa|1234|2010-01-01 Required Output ... (5 Replies)
Discussion started by: mabarif16
5 Replies

5. Shell Programming and Scripting

remove of duplicate line from a file

I have a file a.txt having content like deepak ram sham deepram sita kumar I Want to delete the first line containing "deep" ... I tried using... grep -i 'deep' a.txt It gives me 2 rows...I want to delete the first one.. + need to know the command to delete the line from... (5 Replies)
Discussion started by: saluja.deepak
5 Replies

6. Shell Programming and Scripting

How do I remove the duplicate lines in this file?

Hey guys, need some help to fix this script. I am trying to remove all the duplicate lines in this file. I wrote the following script, but does not work. What is the problem? The output file should only contain five lines: Later! (5 Replies)
Discussion started by: Ernst
5 Replies

7. Shell Programming and Scripting

How to Remove duplicate value from file?

if different branch code is available for same BIC code and one of the branch code is XXX.only one row will be stored and with branch code as XXX .rest of the rows for the BIC code will not be stored. for example if $7 is BIC code and $8 is branch code INPUT file are following... (9 Replies)
Discussion started by: mohan sharma
9 Replies

8. Shell Programming and Scripting

Remove the duplicate content in a file

Here is the contents of test.txt Dependencies Resolved Changes in packages about to be updated: ChangeLog for: 1:perl-Archive-Extract-0.38-131.el6_4.x86_64, - Resolves: #915692 - CVE-2013-1667 (DoS in rehashing code) Dependencies Resolved Changes in packages about to be updated: ... (5 Replies)
Discussion started by: ashokvpp
5 Replies

9. Shell Programming and Scripting

Remove duplicate lines from a file

Hi, I have a csv file which contains some millions of lines in it. The first line(Header) repeats at every 50000th line. I want to remove all the duplicate headers from the second occurance(should not remove the first line). I don't want to use any pattern from the Header as I have some... (7 Replies)
Discussion started by: sudhakar T
7 Replies
SQL::Translator::Parser::SQLite(3pm)			User Contributed Perl Documentation		      SQL::Translator::Parser::SQLite(3pm)

NAME
SQL::Translator::Parser::SQLite - parser for SQLite SYNOPSIS
use SQL::Translator; use SQL::Translator::Parser::SQLite; my $translator = SQL::Translator->new; $translator->parser("SQL::Translator::Parser::SQLite"); DESCRIPTION
This is a grammar for parsing CREATE statements for SQLite as described here: http://www.sqlite.org/lang.html CREATE INDEX sql-statement ::= CREATE [TEMP | TEMPORARY] [UNIQUE] INDEX index-name ON [database-name .] table-name ( column-name [, column-name]* ) [ ON CONFLICT conflict-algorithm ] column-name ::= name [ ASC | DESC ] CREATE TABLE sql-command ::= CREATE [TEMP | TEMPORARY] TABLE table-name ( column-def [, column-def]* [, constraint]* ) sql-command ::= CREATE [TEMP | TEMPORARY] TABLE table-name AS select-statement column-def ::= name [type] [[CONSTRAINT name] column-constraint]* type ::= typename | typename ( number ) | typename ( number , number ) column-constraint ::= NOT NULL [ conflict-clause ] | PRIMARY KEY [sort-order] [ conflict-clause ] | UNIQUE [ conflict-clause ] | CHECK ( expr ) [ conflict-clause ] | DEFAULT value constraint ::= PRIMARY KEY ( name [, name]* ) [ conflict-clause ]| UNIQUE ( name [, name]* ) [ conflict-clause ] | CHECK ( expr ) [ conflict-clause ] conflict-clause ::= ON CONFLICT conflict-algorithm CREATE TRIGGER sql-statement ::= CREATE [TEMP | TEMPORARY] TRIGGER trigger-name [ BEFORE | AFTER ] database-event ON [database-name .] table-name trigger-action sql-statement ::= CREATE [TEMP | TEMPORARY] TRIGGER trigger-name INSTEAD OF database-event ON [database-name .] view-name trigger-action database-event ::= DELETE | INSERT | UPDATE | UPDATE OF column-list trigger-action ::= [ FOR EACH ROW | FOR EACH STATEMENT ] [ WHEN expression ] BEGIN trigger-step ; [ trigger-step ; ]* END trigger-step ::= update-statement | insert-statement | delete-statement | select-statement CREATE VIEW sql-command ::= CREATE [TEMP | TEMPORARY] VIEW view-name AS select-statement ON CONFLICT clause conflict-clause ::= ON CONFLICT conflict-algorithm conflict-algorithm ::= ROLLBACK | ABORT | FAIL | IGNORE | REPLACE expression expr ::= expr binary-op expr | expr like-op expr | unary-op expr | ( expr ) | column-name | table-name . column-name | database-name . table-name . column-name | literal-value | function-name ( expr-list | * ) | expr (+) | expr ISNULL | expr NOTNULL | expr [NOT] BETWEEN expr AND expr | expr [NOT] IN ( value-list ) | expr [NOT] IN ( select-statement ) | ( select-statement ) | CASE [expr] ( WHEN expr THEN expr )+ [ELSE expr] END like-op::= LIKE | GLOB | NOT LIKE | NOT GLOB AUTHOR
Ken Youens-Clark <kclark@cpan.org>. SEE ALSO
perl(1), Parse::RecDescent, SQL::Translator::Schema. perl v5.14.2 2012-01-18 SQL::Translator::Parser::SQLite(3pm)
All times are GMT -4. The time now is 05:15 PM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy