05-13-2013
Quote:
Originally Posted by
duplicate
Based on 5 fields the duplicates has to be removed .
What are the 5 fields and what's the field seperator?
10 More Discussions You Might Find Interesting
1. UNIX for Dummies Questions & Answers
I am using Unix as my OS on my server and would like to format my hard drive. How do I go about wiping my hard drive or is there a removal tool that I can use? (1 Reply)
Discussion started by: anaconda
1 Replies
2. UNIX for Dummies Questions & Answers
Hi All,
(And first up, a Happy New Year to you all! 363 days to go!)
I need to make sure that our old Sun Sparcstations and Servers are clear of any Confidential data before I return them...
Suggestions?
I'm formatting the discs to make sure that there's nothing obvious, but would... (5 Replies)
Discussion started by: geralex
5 Replies
3. UNIX for Dummies Questions & Answers
Hi All,
I have a problem with the sort and duplicate filter command I am using in one of my scripts. I have a '|' delimited file and want to sort and remove duplicates on the 1,2,15 fields. These fields constitute the primary key of the table I will be loading the data into. But I see that some... (4 Replies)
Discussion started by: novice1324
4 Replies
4. Shell Programming and Scripting
I have a file which consists of 1000 entries. Out of 1000 entries i have 500 Duplicate Entires. I want to remove the first Duplicate Entry (i,e entire Line) in the File.
The example of the File is shown below:
8244100010143276|MARISOL CARO||MORALES|HSD768|CARR 430 KM 1.7 ... (1 Reply)
Discussion started by: ravi_rn
1 Replies
5. Homework & Coursework Questions
Use and complete the template provided. The entire template must be completed. If you don't, your post may be deleted!
1. The problem statement, all variables and given/known data:
remove all files and only files that the first three characters start with numerals.
2. Relevant commands,... (6 Replies)
Discussion started by: linuxtraining
6 Replies
6. Shell Programming and Scripting
I have a multi-line string I'm trying to do some clean-up on.
Example:
1|575|67866|"fgnhdgj"|"afhgajh
agfajgf
ahfjhgfk
ahfkhf"|568
2|56|5435|"mayank"|"gupta
gdja
agdjagf"|677
3|5666|5435|"mayank"|"gupta
gdja
agdjagf"|677
I need a shell script that replace all \n under " ". (11 Replies)
Discussion started by: mayankgupta18
11 Replies
7. Shell Programming and Scripting
Hi,
In a file, I have to mark duplicate records as 'D' and the latest record alone as 'C'.
In the below file, I have to identify if duplicate records are there or not based on Man_ID, Man_DT, Ship_ID and I have to mark the record with latest Ship_DT as "C" and other as "D" (I have to create... (7 Replies)
Discussion started by: machomaddy
7 Replies
8. Shell Programming and Scripting
I'm looking to remove duplicate rows from a CSV file with a twist.
The first row is a header.
There are 31 columns. I want to remove duplicates when the first 29 rows are identical ignoring row 30 and 31 BUT the duplicate that is kept should have the shortest total character length in rows 30... (6 Replies)
Discussion started by: Michael Stora
6 Replies
9. Web Development
first off i want to thank you for such a great site, you helped me narrow down a long search on what was wrong with my forum, i have a large forum motorbicycling DOT com which i had someone remove vbseo before they went under because of all the security problems. anyway i have lots of 404 errors... (13 Replies)
Discussion started by: atcspaul
13 Replies
10. Shell Programming and Scripting
I have a script that builds a database ~30 million lines, ~3.7 GB .cvs file. After multiple optimzations It takes about 62 min to bring in and parse all the files and used to take 10 min to remove duplicates until I was requested to add another column. I am using the highly optimized awk code:
awk... (34 Replies)
Discussion started by: Michael Stora
34 Replies
LEARN ABOUT DEBIAN
recsel
RECSEL(1) User Commands RECSEL(1)
NAME
recsel - print records from a recfile
SYNOPSIS
recsel [OPTION]... [-t TYPE] [-n INDEXES | -e RECORD_EXPR | -q EXPR | -m NUM] [-c | (-p|-P) FIELD_EXPR] [FILE]...
DESCRIPTION
Select and print rec data.
-d, --include-descriptors
print record descriptors along with the matched records.
-C, --collapse
do not section the result in records with newlines.
-S, --sort=FIELD
sort the output by the specified field.
-U, --uniq
remove duplicated fields in the output records.
-s, --password=STR
decrypt confidential fields with the given password.
--help print a help message and exit.
--version
show version and exit.
Record selection options:
-i, --case-insensitive
make strings case-insensitive in selection expressions.
-t, --type=TYPE
operate on records of the specified type only.
-e, --expression=EXPR
selection expression.
-q, --quick=STR
select records with fields containing a string.
-n, --number=NUM,...
select specific records by position, with ranges.
-R, --random=NUM
select a given number of random records.
Output options:
-p, --print=FIELDS
comma-separated list of fields to print for each matching record.
-P, --print-values=FIELDS
as -p, but print only the values of the selected fields.
-R, --print-row=FIELDS
as -P, but separate the values with spaces instead of newlines.
-c, --count
print a count of the matching records instead of the records themselves.
Special options:
--print-sexps
print the data in sexps instead of rec format.
AUTHOR
Written by Jose E. Marchesi.
REPORTING BUGS
Report bugs to: bug-recutils@gnu.org
GNU recutils home page: <http://www.gnu.org/software/recutils/>
General help using GNU software: <http://www.gnu.org/gethelp/>
COPYRIGHT
Copyright (C) 2010, 2011, 2012 Jose E. Marchesi. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.
SEE ALSO
The full documentation for recsel is maintained as a Texinfo manual. If the info and recsel programs are properly installed at your site,
the command
info recsel
should give you access to the complete manual.
recsel 1.4.93 January 2012 RECSEL(1)