I need help in figuring out hoe to delete lines in a data file. The data file is huge. I am currently using "vi" to search and delete the lines - which is cumbersome since it takes lots of time to save that file (due to its huge size).
Here is the issue. I have a data file with the following data - seperated by "|"
I want to remove the line which is almost duplicate which is line#5. Line # 2 and line #5 are almost duplicate but the fifth field is different.I need to search only on the 1st field of the record (which in this case is "fld2") and then delete the 2nd occurence of the same 1st field.
Can this be done ? if yes how? The file contains around 500K rows.
Last edited by Franklin52; 04-03-2012 at 10:25 AM..
Reason: Please use code tags for data and code samples, thank you
Hi All,
I want to delete duplicate records from a tilde delimited file. Criteria is considering the first 2 fields, the combination of which has to be unique, below is a sample of records in the input file
1620000010338~2446694087~0~20061130220000~A00BCC1CT... (5 Replies)
Ok here's what I'm trying to do. I need to get a listing of all the mountpoints on a system into a file, which is easy enough, just using something like "mount | awk '{print $1}'"
However, on a couple of systems, they have some mount points looking like this:
/stage
/stand
/usr
/MFPIS... (2 Replies)
I have a file content like below.
"0000000","ABLNCYI","BOTH",1049,2058,"XYZ","5711002","","Y","","","","","","","",""
"0000000","ABLNCYI","BOTH",1049,2058,"XYZ","5711002","","Y","","","","","","","",""
"0000000","ABLNCYI","BOTH",1049,2058,"XYZ","5711002","","Y","","","","","","","",""... (5 Replies)
OK, I have read several things on how to do this, but can't make it work. I am writing this to a vi file then calling it as an awk script.
So I need to search a file for duplicate lines, delete duplicate lines, then write the result to another file, say /home/accountant/files/docs/nodup
... (2 Replies)
Hi please help me how to remove duplicate lines in any file.
I have a file having huge number of lines.
i want to remove selected lines in it.
And also if there exists duplicate lines, I want to delete the rest & just keep one of them.
Please help me with any unix commands or even fortran... (7 Replies)
I am completely new to shell scripting but have been assigned the task of creating several batch files to manipulate data. My final task requires me to find lines that have duplicates present then delete not only the duplicate but the original as well. The script will be used in a windows... (9 Replies)
Hi ALL
I need a help
I need to retain only the first line of 035 if I have two line before =040 , if only one then need to take that
Eg:
Input
=035 (ABC)12324141241
=035 (XYZPQR)704124
=040 AB$QS$WEWR
=035 (ABC)08080880809
=035 (XYZPQR)9809314
=040 ... (4 Replies)
I want to delete partical duplicate file
>gma-miR156d Gm01,PACID=26323927 150.00 -18.28 2 18 17 35 16 75.00% 81.25%
>>gma-miR156d Gm01,PACID=26323927 150.00 -18.28 150.00 -18.28 1 21 119 17
I want to order by the second column and delete the... (1 Reply)
Hi,
i need help to remove duplicates in my file. The problem is i need to delete one duplicate for each line only. the input file as follows and it is not tab delimited:-
The output need to remove 2nd word (in red) that duplicate with 1st word (in blue). Other duplicates should remained... (12 Replies)
Hello All !
I need your help on this case,
I have a csv file with this:
ITEM105;ARI FSR;2016-02-01 08:02;243
ITEM101;ARI FSR;2016-02-01 06:02;240
ITEM032;RNO TLE;2016-02-01 11:03;320
ITEM032;RNO TLE;2016-02-02 05:43;320
ITEM032;RNO TLE;2016-02-01 02:03;320
ITEM032;RNO... (2 Replies)
Discussion started by: vadim-bzh
2 Replies
LEARN ABOUT OSF1
curs_deleteln
curs_deleteln(3) Library Functions Manual curs_deleteln(3)NAME
curs_deleteln, deleteln, wdeleteln, insdelln, winsdelln, insertln, winsertln - Delete or insert lines in a Curses window
SYNOPSIS
#include <curses.h>
int deleteln(
void ); int wdeleteln(
WINDOW *win ); int insdelln(
int n ); int winsdelln(
WINDOW *win,
int n ); int insertln(
void ); int winsertln(
WINDOW *win );
LIBRARY
Curses Library (libcurses)
STANDARDS
Interfaces documented on this reference page conform to industry standards as follows:
deleteln, wdeleteln, insdelln, winsdelln, insertln, winsertln: XCURSES4.2
Refer to the standards(5) reference page for more information about industry standards and associated tags.
DESCRIPTION
The deleteln and wdeleteln routines do the following: Delete the line under the cursor in the current or specified window Move all lines
below the current line up one line Clear the bottom line of the window
These routines do not change the cursor position. (This does not imply use of a hardware delete line feature.)
The insdelln and winsdelln routines do the following: For positive n, insert n lines into the current or specified window above the current
line
The n bottom lines are lost. For negative n, delete n lines (starting with the one under the cursor), and move the remaining lines
up
The bottom n lines are cleared.
These routines do not change the current cursor position.
The insertln and insertln routines insert a blank line above the current line. The bottom line is lost. (This does not imply use of a hard-
ware insert line feature.)
NOTES
The header file <curses.h> automatically includes the header file <stdio.h>.
Note that all routines except winsdelln may be macros.
RETURN VALUES
All routines return ERR upon failure and OK upon successful completion.
SEE ALSO
Functions: curses(3)
Others: standards(5)curs_deleteln(3)