We once needed a code that would run on a number of different systems, yet produce consistent results. We ran into the situation that utility uniq was not consistent among the systems. We introducing an option:
By substituting this idea for the system version of uniq, we were able to produce consistent results.
I think this problem can approached with the sort idea of danmero, but with the stable option set, and a "final filter" that eliminates duplicates. Because the file is already sorted, no additional storage is needed: in the final filter, if the fields of the incoming record differ from that in storage, then write out the saved line, and save the new line. If the fields are the same, then save the new instance of the line. Our code was in perl, but awk could be as easily used.
:)Hi Friends,
I have certain log files extracted. I want it to be converted in pipe ( | ) delimited file. How do i do it?
E.g.
Account Balance : 123456789 Rs O/P (Account Balance: | 123456789 Rs)
Account Balance (Last) > 987654321 Rs O/P (Account Balance (Last) | 987654321 Rs)
Last... (5 Replies)
i have a file whose data is like this::
osr_pe_assign|-120|wg000d@att.com|4|
osr_evt|-21|wg000d@att.com|4|
pe_avail|-21|wg000d@att.com|4|
osr_svt|-11|wg000d@att.com|4|
pe_mop|-13|wg000d@att.com|4|
instar_ready|-35|wg000d@att.com|4|
nsdnet_ready|-90|wg000d@att.com|4|... (6 Replies)
This is my input file with extra information in the HEADER and leading & trailing SPACES between PIPE delimiter.
02/04/2010 Dynamic List Display 1
--------------------------------------------------------------------------------------... (6 Replies)
Hi All,
I have space delimited file similar to the one as shown below.. I need to convert it as a pipe delimited, the values inside the pipe delimited file should be as highlighted...
AA ATIU2345098809 009697 005374
BB ATIU2345097809 005445 006518
CC ATIU9685098809 003215 003571
DD... (7 Replies)
I have a file which was pipe delimited, I need to make it tab delimited. I tried with sed but no use
cat file | sed 's/|//t/g'
The above command substituted "/t" not tab in the place of pipe.
Sample file:
abc|123|2012-01-30|2012-04-28|xyz
have to convert to:
abc 123... (6 Replies)
I have file as below
column1|column2|column3|column4|column5|
fill1|fill2|fill3|fill4|fill5|
abc1|abc2|abc3|abc4|abc5|
.
.
.
.
i need to remove column2,3, from that file
column1|column4|column5|
fill1|fill4|fill5|
abc1|abc4|abc5|
.
.
. (3 Replies)
Hi guys,
I need to know how i can ignore Pipe '|' if Pipe is coming as a column in Pipe delimited file
for eg:
file 1:
xx|yy|"xyz|zzz"|zzz|12...
using below awk command
awk 'BEGIN {FS=OFS="|" } print $3
i would get xyz
But i want as :
xyz|zzz to consider as whole column... (13 Replies)
Hi guys,Got a bit of a bind I'm in. I'm looking to remove duplicates from a pipe delimited file, but do so based on 2 columns. Sounds easy enough, but here's the kicker...
Column #1 is a simple ID, which is used to identify the duplicate.
Once dups are identified, I need to only keep the one... (2 Replies)
Hi, I have a rquirement in unix as below .
I have a text file with me seperated by | symbol and i need to generate a excel file through unix commands/script so that each value will go to each column.
ex:
Input Text file:
1|A|apple
2|B|bottle
excel file to be generated as output as... (9 Replies)
I have a file as below
Emp1|FirstName|MiddleName|LastName|Address|Pincode|PhoneNumber
1234|FirstName1|MiddleName2|LastName3| Add1 || ADD2|123|000000000
2345|FirstName2|MiddleName3|LastName4|
Add1 || ADD2|
234|000000000
OUTPUT :
... (1 Reply)
Discussion started by: styris
1 Replies
LEARN ABOUT XFREE86
uniq
UNIQ(1) User Commands UNIQ(1)NAME
uniq - report or omit repeated lines
SYNOPSIS
uniq [OPTION]... [INPUT [OUTPUT]]
DESCRIPTION
Filter adjacent matching lines from INPUT (or standard input), writing to OUTPUT (or standard output).
With no options, matching lines are merged to the first occurrence.
Mandatory arguments to long options are mandatory for short options too.
-c, --count
prefix lines by the number of occurrences
-d, --repeated
only print duplicate lines, one for each group
-D print all duplicate lines
--all-repeated[=METHOD]
like -D, but allow separating groups with an empty line; METHOD={none(default),prepend,separate}
-f, --skip-fields=N
avoid comparing the first N fields
--group[=METHOD]
show all items, separating groups with an empty line; METHOD={separate(default),prepend,append,both}
-i, --ignore-case
ignore differences in case when comparing
-s, --skip-chars=N
avoid comparing the first N characters
-u, --unique
only print unique lines
-z, --zero-terminated
line delimiter is NUL, not newline
-w, --check-chars=N
compare no more than N characters in lines
--help display this help and exit
--version
output version information and exit
A field is a run of blanks (usually spaces and/or TABs), then non-blank characters. Fields are skipped before chars.
Note: 'uniq' does not detect repeated lines unless they are adjacent. You may want to sort the input first, or use 'sort -u' without
'uniq'. Also, comparisons honor the rules specified by 'LC_COLLATE'.
AUTHOR
Written by Richard M. Stallman and David MacKenzie.
REPORTING BUGS
GNU coreutils online help: <http://www.gnu.org/software/coreutils/>
Report uniq translation bugs to <http://translationproject.org/team/>
COPYRIGHT
Copyright (C) 2017 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law.
SEE ALSO comm(1), join(1), sort(1)
Full documentation at: <http://www.gnu.org/software/coreutils/uniq>
or available locally via: info '(coreutils) uniq invocation'
GNU coreutils 8.28 January 2018 UNIQ(1)