|
|
Sponsored Content
Top Forums
Shell Programming and Scripting
[uniq + awk?] How to remove duplicate blocks of lines in files?
Post 302557144 by vgersh99 on Tuesday 20th of September 2011 12:36:29 PM
|
|
10 More Discussions You Might Find Interesting
1. Shell Programming and Scripting
Hi,
is it possible to remove all duplicate lines from all txt files in a specific folder?
This is too hard for me maybe someone could help.
lets say we have an amount of textfiles 1 or 2 or 3 or... maximum 50
each textfile has lines with text.
I want all lines of all textfiles... (8 Replies)
Discussion started by: lowmaster
8 Replies
2. UNIX for Dummies Questions & Answers
I have a data set that has 4 columns, I want to know if I can delete duplicate lines while ignoring one of the columns, for example
10 chr1 ASF 30
15 chr1 ASF 20
5 chr1 ASF 30
6 chr2 EBC 15
4 chr2 EBC 30
...
I want to know if I can delete duplicate lines while ignoring column 1, so the... (5 Replies)
Discussion started by: japaneseguitars
5 Replies
3. Shell Programming and Scripting
Input:
hello hello
hello hello
monkey
donkey
hello hello
drink
dance
drink
Output should be:
hello hello
monkey
donkey
drink
dance (9 Replies)
Discussion started by: cola
9 Replies
4. Shell Programming and Scripting
Dear All,
I have multiple files having number of records, consist of more than 10 columns some column values are duplicate and i want to remove these duplicate values from these files.
Duplicate values may come in different files.... all files laying in single directory..
Need help to... (3 Replies)
Discussion started by: arvindng
3 Replies
5. Shell Programming and Scripting
Hi,
I came to know that using awk '!x++' removes the duplicate lines. Can anyone please explain the above syntax. I want to understand how the above awk syntax removes the duplicates.
Thanks in advance,
sudvishw :confused: (7 Replies)
Discussion started by: sudvishw
7 Replies
6. Shell Programming and Scripting
Cant get awk 1liner to remove duplicate lines from Delimited file, get "event not found" error..help
Hi,
I am on a Solaris8 machine
If someone can help me with adjusting this awk 1 liner (turning it into a real awkscript) to get by this "event not found error"
...or
Present Perl solution code that works for Perl5.8 in the csh shell ...that would be great.
******************
... (3 Replies)
Discussion started by: andy b
3 Replies
7. Windows & DOS: Issues & Discussions
So, I have text files,
one "fail.txt"
And one
"color.txt"
I now want to use a command line (DOS) to remove ANY line that is PRESENT IN BOTH from each text file.
Afterwards there shall be no duplicate lines. (1 Reply)
Discussion started by: pasc
1 Replies
8. Shell Programming and Scripting
Hi All
I have a list of files which will have duplicate list of blocks of text. Following is a sample of the file, I have removed the sensitive information from the file.
All the code samples starts from <TR BGCOLOR="white"> and Ends with IP address and two html tags like this.
10.14.22.22... (3 Replies)
Discussion started by: mahasona
3 Replies
9. Shell Programming and Scripting
I have two large files (~250GB) that I am trying to remove the where GT: 0/0 or 1/1 or 2/2 for both files. I was going to use a bash with the below awk, which I think will find each line but how do I remove that line is that condition is found? Thank you :).
Input
20 60055 . A ... (4 Replies)
Discussion started by: cmccabe
4 Replies
10. Shell Programming and Scripting
I create a CGI in bash/html.
My awk script looks like :
echo "<table>"
for fn in /var/www/cgi-bin/LPAR_MAP/*;
do
echo "<td>"
echo "<PRE>"
awk -F',|;' -v test="$test" '
NR==1 {
split(FILENAME ,a,"");
}
$0 ~ test {
if(!header++){
... (12 Replies)
Discussion started by: Tim2424
12 Replies
LEARN ABOUT SUSE
ucblinks
ucblinks(1B) SunOS/BSD Compatibility Package Commands ucblinks(1B) NAME
ucblinks - adds /dev entries to give SunOS 4.x compatible names to SunOS 5.x devices SYNOPSIS
/usr/ucb/ucblinks [-e rulebase] [-r rootdir] DESCRIPTION
ucblinks creates symbolic links under the /dev directory for devices whose SunOS 5.x names differ from their SunOS 4.x names. Where possi- ble, these symbolic links point to the device's SunOS 5.x name rather than to the actual /devices entry. ucblinks does not remove unneeded compatibility links; these must be removed by hand. ucblinks should be called each time the system is reconfiguration-booted, after any new SunOS 5.x links that are needed have been created, since the reconfiguration may have resulted in more compatibility names being needed. In releases prior to SunOS 5.4, ucblinks used a nawk rule-base to construct the SunOS 4.x compatible names. ucblinks no longer uses nawk for the default operation, although nawk rule-bases can still be specifed with the -e option. The nawk rule-base equivalent to the SunOS 5.4 default operation can be found in /usr/ucblib/ucblinks.awk. OPTIONS
-e rulebase Specify rulebase as the file containing nawk(1) pattern-action statements. -r rootdir Specify rootdir as the directory under which dev and devices will be found, rather than the standard root directory /. FILES
/usr/ucblib/ucblinks.awk sample rule-base for compatibility links ATTRIBUTES
See attributes(5) for descriptions of the following attributes: +-----------------------------+-----------------------------+ | ATTRIBUTE TYPE | ATTRIBUTE VALUE | +-----------------------------+-----------------------------+ |Availability |SUNWscpu | +-----------------------------+-----------------------------+ SEE ALSO
devlinks(1M), disks(1M), ports(1M), tapes(1M), attributes(5) SunOS 5.10 13 Apr 1994 ucblinks(1B)