Sponsored Content
Top Forums Shell Programming and Scripting Deleting all occurences of a duplicate row Post 302213418 by ragavhere on Thursday 10th of July 2008 04:13:03 AM
Old 07-10-2008
Deleting all occurences of a duplicate row

Hi,

I need to delete all occurences of the repeated lines from a file and retain only the lines that is not repeated elsewhere in the file. As seen below the first two lines are same except that for the string "From BaseLine" and "From SMS".I shouldn't consider the string "From SMS" and "From BaseLine" for checking the repeated lines. I want to retain only the third line.

From BaseLine - 0T001 000 999999999 00101 20080411000000T1023.27
From SMS - 0T001 000 999999999 00101 20080411000000T1023.27
From BaseLine - 0T001 000 999999999 00101 20080411000000T109.019

My output should be the third line alone.

These file size would range from 100 MB to 900MB. The performance factor should also be considered. Can you please help me out?

Regards,

Ragav.
 

10 More Discussions You Might Find Interesting

1. Shell Programming and Scripting

sort and semi-duplicate row - keep latest only

I have a pipe delimited file. Key is field 2, date is field 5 (as example, my real file is more complicated of course, but the KEY and DATE are accurate) There can be duplicate rows for a key with different dates. I need to keep only rows with latest date in this case. Example data: ... (4 Replies)
Discussion started by: LisaS
4 Replies

2. Shell Programming and Scripting

Delete a row that has a duplicate column

I'm trying to remove lines of data that contain duplicate data in a specific column. For example. apple 12345 apple 54321 apple 14234 orange 55656 orange 88989 orange 99898 I only want to see apple 12345 orange 55656 How would i go about doing this? (5 Replies)
Discussion started by: spartan22
5 Replies

3. Shell Programming and Scripting

how to identify duplicate columns in a row

Hi, How to identify duplicate columns in a row? Input data: may have 30 columns 9211480750 LK 120070417 920091030 9211480893 AZ 120070607 9205323621 O7 120090914 120090914 1420090914 2020090914 2020090914 9211479568 AZ 120070327 320090730 9211479571 MM 120070326 9211480892 MM 120070324... (3 Replies)
Discussion started by: suresh3566
3 Replies

4. Shell Programming and Scripting

Deleting Duplicate Records

Hello, I'm have a file of xy data with over 1000 records. I want to delete both x and y values for any record that has the same x value as any previous record thus removing the duplicates from my file. Can anyone help? Thanks, Dan (3 Replies)
Discussion started by: DFr0st
3 Replies

5. Shell Programming and Scripting

REMOVE DUPLICATE IN a ROW AFTER CHECKING THE FIRST SIMILAR NAME

Hi all I have a big file like this in rows and columns from 2 column onwards the next column is desciption of previous column means 3rd columns is description of 2 columns and 5 column is description of 4 column. All cloumns are separated by comma ... (1 Reply)
Discussion started by: manigrover
1 Replies

6. Shell Programming and Scripting

deleting dupes in a row

Hello, I have a large database in which name homonyms are arranged in a row. Since the database is large and generated by hand, very often dupes creep in. I want to remove the dupes either using an awk or perl script. An input is given below The expected output is given below: As can be... (2 Replies)
Discussion started by: gimley
2 Replies

7. Shell Programming and Scripting

Moving new row and deleting old row to another table

Hi, I want to move a new row to another table if the field from new row doesn't have the first word that I categorized (like: IRC blablabla, PTM blablabla, ADM blablabla, BS blablabla). I already use this script but doesn't work as I expected. CHECK_KEYWORD="$( mysql -uroot -p123456 smsd -N... (7 Replies)
Discussion started by: jazzyzha
7 Replies

8. Shell Programming and Scripting

In php, Moving a new row to another table and deleting old row

Hi, I already succeed moving a new row to another table if the field from new row doesn't have the first word that I categorized (like: IRC blablabla, PTM blablabla, ADM blablabla, BS blablabla). But it can't delete the old row. Please help me with the script. my php script: INSERT INTO... (2 Replies)
Discussion started by: jazzyzha
2 Replies

9. Shell Programming and Scripting

Delete duplicate row

Hi all, how can delete duplicate files in file form, e.g. $cat file1 aaa 123 234 345 456 bbb 345 345 657 568 ccc 345 768 897 456 aaa 123 234 345 456 ddd 786 784 234 263 ccc 345 768 897 456 aaa 123 234 345 456 ccc 345 768 897 456 then i need ouput file1 some, (4 Replies)
Discussion started by: aav1307
4 Replies

10. Shell Programming and Scripting

Delete duplicate row based on criteria

Hi, I have an input file as shown below: 20140102;13:30;FR-AUD-LIBOR-1W;2.495 20140103;13:30;FR-AUD-LIBOR-1W;2.475 20140106;13:30;FR-AUD-LIBOR-1W;2.495 20140107;13:30;FR-AUD-LIBOR-1W;2.475 20140108;13:30;FR-AUD-LIBOR-1W;2.475 20140109;13:30;FR-AUD-LIBOR-1W;2.475... (2 Replies)
Discussion started by: shash
2 Replies
GSMSMSSTORE(8)						      System Manager's Manual						    GSMSMSSTORE(8)

NAME
gsmsmsstore - SMS store manipulation program SYNOPSIS
gsmsmsstore [ -a ] [ --add ] [ -b baudrate ] [ --baudrate baudrate ] [ -c ] [ --copy ] [ -C service centre address ] [ --sca service centre address ] [ -d destination device or file ] [ --destination destination device or file ] [ -h ] [ --help ] [ -I init string ] [ --init init string ] [ -k ] [ --backup ] [ -l ] [ --list ] [ -s source device or file ] [ --source source device or file ] [ -t SMS store name ] [ --store SMS store name ] [ -v ] [ --version ] [ -V ] [ --verbose ] [ -x ] [ --delete ] [ -X ] [ --xonxoff ] { indices } [ phonenumber text ] DESCRIPTION
gsmsmsstore can store or retrieve SMS messages entries residing in a GSM mobile phone's SMS store to or from a file, add SMS messages to a store, or list the store's contents. Additionally, it is possible to add SMS submit messages to a store. gsmsmsstore reads entries from the source which can be a mobile phone (if a serial device file is given) or a file (if a file name is given). The source is never modified. gsmsmsstore writes SMS messages to a destination file or device in the case of --copy, --backup, and --add. The --list option does not change any file but just lists the contents to standard output. The --backup and --copy options require both source and destination files or devices. The --list option requires a source. The --add and --delete options require a destination file or device. If "-" is given as the parameter for the --source or --destination options, the SMS store is read from standard input and/or written to standard output, respectively. SMS message files are not human-readable. Error messages are printed to the standard error output. If the program terminates on error the error code 1 is returned. OPTIONS
-a, --add Adds an SMS submit message with recipient address phonenumber and text text to the destination. -b baudrate, --baudrate baudrate The baud rate to use. The default baudrate is 38400. -c, --copy This causes the contents of the source to be copied to the destination. After this operation the destination has exactly the same contents as the source. If indices are given on the command line only those SMS messages denoted by the indices are copied to the destination. -C service centre address, --sca service centre address Sets the service centre address to use for all SUBMIT SMSs (may not work with some phones). -d destination, --destination destination The destination device or file. -h, --help Prints an option summary. -I init string, --init init string Initialization string to send to the TA (default: "E0"). Note that the sequence "ATZ" is sent first. -k, --backup This causes those entries to be added from the source to the destination that are not already present in the destination. If indices are given on the command line only those SMS messages denoted by the indices are backed up (ie. added) to the destination. -l, --list Prints out the entire contents of the source in human-readable form. -s source, --source source The source device or file. -t SMS store name, --store SMS store name The name of the SMS store to read from or write to. This information is only used for device sources and destinations. A commonly available message store is "SM" (SIM card). -v, --version Prints the program version. -V, --verbose Prints out a detailed progress report. -x, --delete Delete the SMS messages as denoted by the indices from the destination. -X, --xonxoff Uses software handshaking (XON/XOFF) for accessing the device. EXAMPLES
The following command lists all entries in the mobile phone connected to /dev/mobilephone to the standard output: gsmsmsstore -b 19200 -s /dev/mobilephone -t SM -l The following adds entries 4, 7, and 10 from the device /dev/mobilephone to the file smsstore: gsmsmsstore -s /dev/mobilephone -d /home/fred/smsstore -t SM -b 4 7 10 AUTHOR
Peter Hofmann <software@pxh.de> BUGS
Report bugs to software@pxh.de. Include a complete, self-contained example that will allow the bug to be reproduced, and say which version of gsmsmsstore you are using. COPYRIGHT
Copyright (C) 1999 Peter Hofmann gsmsmsstore is free software; you can redistribute it and/or modify it under the terms of the GNU Library General Public License as pub- lished by the Free Software Foundation; either version 2, or (at your option) any later version. gsmsmsstore is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABIL- ITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Library General Public License for more details. You should have received a copy of the GNU Library General Public License along with gsmsmsstore; see the file COPYING. If not, write to the Free Software Foundation, 675 Mass Ave, Cambridge, MA 02139, USA. SEE ALSO
gsminfo(7), gsmctl(1), gsmpb(1), gsmsendsms(1), gsmsmsd(8). gsmsmsstore v1.10 Sat Jun 16 22:10:00 UTC 2012 GSMSMSSTORE(8)
All times are GMT -4. The time now is 10:57 AM.
Unix & Linux Forums Content Copyright 1993-2022. All Rights Reserved.
Privacy Policy